2024-02-19T23:56:16.765 INFO:root:teuthology version: 0.0.1.dev260+gb1dac55 2024-02-19T23:56:16.766 DEBUG:teuthology.run:Teuthology command: teuthology --description orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} --name gabrioux-2024-02-19_22:54:55-orch:cephadm-wip-guits-testing-6-2024-02-19-2119-pacific-distro-default-smithi --verbose --owner scheduled_gabrioux@teuthology --archive /home/teuthworker/archive/gabrioux-2024-02-19_22:54:55-orch:cephadm-wip-guits-testing-6-2024-02-19-2119-pacific-distro-default-smithi/7568131 -- /home/teuthworker/archive/gabrioux-2024-02-19_22:54:55-orch:cephadm-wip-guits-testing-6-2024-02-19-2119-pacific-distro-default-smithi/7568131/orig.config.yaml 2024-02-19T23:56:16.814 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-02-19T23:56:16.909 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/gabrioux-2024-02-19_22:54:55-orch:cephadm-wip-guits-testing-6-2024-02-19-2119-pacific-distro-default-smithi/7568131 branch: wip-guits-testing-6-2024-02-19-2119-pacific description: orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} email: gabrioux@ibm.com first_in_suite: false job_id: '7568131' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi meta: - desc: 'setup ceph/pacific v16.2.4 ' name: gabrioux-2024-02-19_22:54:55-orch:cephadm-wip-guits-testing-6-2024-02-19-2119-pacific-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 8.stream overrides: admin_socket: branch: wip-guits-testing-6-2024-02-19-2119-pacific ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: mon pg warn min per osd: 0 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - \(PG_DEGRADED\) - Degraded data redundancy - \(PG_ - MDS_INSUFFICIENT_STANDBY - deprecated feature inline_data - compat changed unexpectedly - overall HEALTH_ - OSD_DOWN - OSD_ - but it is still running - is not responding sha1: 19854089e18d4f65dda2b6cd74e737367c2514bd ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 19854089e18d4f65dda2b6cd74e737367c2514bd selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-guits-testing-6-2024-02-19-2119-pacific sha1: 19854089e18d4f65dda2b6cd74e737367c2514bd owner: scheduled_gabrioux@teuthology priority: 70 repo: https://git.ceph.com/ceph-ci.git roles: - - host.a - client.0 - osd.0 - osd.1 - osd.2 - - host.b - client.1 - osd.3 - osd.4 - osd.5 seed: 5185 sha1: 19854089e18d4f65dda2b6cd74e737367c2514bd sleep_before_teardown: 0 subset: 1/999 suite: orch:cephadm suite_branch: wip-guits-testing-6-2024-02-19-2119-pacific suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_19854089e18d4f65dda2b6cd74e737367c2514bd/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 19854089e18d4f65dda2b6cd74e737367c2514bd targets: smithi084.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPNDTO6YWTW+/cXAw1cHDFMHP2Z6Mck+lVm3Zr/19y56KcjLXZcZTPFPbsbMmRH2/7ACg4UxKAjEljm75Wq2Tv4= smithi132.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKO7VWT3OFStyyg3pKrCB0HbjeNRE/yRBAu8QNQYTwU8KFhSKWl9zsWsMZ8QipkP2tfE8vtTV0BeqyBYoaWcnfY= tasks: - pexec: all: - sudo cp /etc/containers/registries.conf /etc/containers/registries.conf.backup - sudo dnf -y module reset container-tools - sudo dnf -y module install container-tools --allowerasing --nobest - sudo cp /etc/containers/registries.conf.backup /etc/containers/registries.conf - pexec: clients: - sudo modprobe -r ceph - sudo modprobe ceph disable_send_metrics=on - install: exclude_packages: - ceph-volume tag: v16.2.4 - print: '**** done install task...' - cephadm: avoid_pacific_features: true cephadm_branch: v16.2.4 cephadm_git_url: https://github.com/ceph/ceph image: docker.io/ceph/ceph:v16.2.4 roleless: true - print: '**** done starting v16.2.4' - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: host.a: - ceph fs volume create cephfs --placement=4 - ceph fs dump - cephadm.shell: host.a: - ceph fs set cephfs max_mds 1 - cephadm.shell: host.a: - ceph fs set cephfs allow_standby_replay true - cephadm.shell: host.a: - ceph fs set cephfs inline_data true --yes-i-really-really-mean-it - cephadm.shell: host.a: - ceph fs dump - ceph --format=json fs dump | jq -e ".filesystems | length == 1" - while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done - fs.pre_upgrade_save: null - kclient: null - print: '**** done client' - parallel: - upgrade-tasks - workload-tasks - cephadm.shell: host.a: - ceph fs dump - fs.post_upgrade_checks: null teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: b1dac5519c57c269ea98a22fb7729016a1d0e4d2 timestamp: 2024-02-19_22:54:55 tube: smithi upgrade-tasks: sequential: - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 host.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph fs dump; ceph orch upgrade status ; sleep 30 ; done - ceph orch ps - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 user: gabrioux verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 workload-tasks: sequential: - workunit: clients: all: - suites/fsstress.sh 2024-02-19T23:56:16.909 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_19854089e18d4f65dda2b6cd74e737367c2514bd/qa; will attempt to use it 2024-02-19T23:56:16.910 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_19854089e18d4f65dda2b6cd74e737367c2514bd/qa/tasks 2024-02-19T23:56:16.910 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-02-19T23:56:16.912 INFO:teuthology.task.internal:Checking packages... 2024-02-19T23:56:16.935 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '19854089e18d4f65dda2b6cd74e737367c2514bd' 2024-02-19T23:56:16.935 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-02-19T23:56:16.935 INFO:teuthology.packaging:ref: None 2024-02-19T23:56:16.935 INFO:teuthology.packaging:tag: None 2024-02-19T23:56:16.935 INFO:teuthology.packaging:branch: wip-guits-testing-6-2024-02-19-2119-pacific 2024-02-19T23:56:16.935 INFO:teuthology.packaging:sha1: 19854089e18d4f65dda2b6cd74e737367c2514bd 2024-02-19T23:56:16.935 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-guits-testing-6-2024-02-19-2119-pacific 2024-02-19T23:56:17.080 INFO:teuthology.task.internal:Found packages for ceph version 16.2.14-549.g19854089 2024-02-19T23:56:17.081 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-02-19T23:56:17.088 INFO:teuthology.task.internal:no buildpackages task found 2024-02-19T23:56:17.088 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-02-19T23:56:17.095 INFO:teuthology.task.internal:Saving configuration 2024-02-19T23:56:17.114 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-02-19T23:56:17.120 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-19T23:56:17.143 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi084.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/gabrioux-2024-02-19_22:54:55-orch:cephadm-wip-guits-testing-6-2024-02-19-2119-pacific-distro-default-smithi/7568131', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-19 23:50:31.397255', 'locked_by': 'scheduled_gabrioux@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPNDTO6YWTW+/cXAw1cHDFMHP2Z6Mck+lVm3Zr/19y56KcjLXZcZTPFPbsbMmRH2/7ACg4UxKAjEljm75Wq2Tv4='} 2024-02-19T23:56:17.160 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi132.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/gabrioux-2024-02-19_22:54:55-orch:cephadm-wip-guits-testing-6-2024-02-19-2119-pacific-distro-default-smithi/7568131', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-19 23:50:31.398722', 'locked_by': 'scheduled_gabrioux@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKO7VWT3OFStyyg3pKrCB0HbjeNRE/yRBAu8QNQYTwU8KFhSKWl9zsWsMZ8QipkP2tfE8vtTV0BeqyBYoaWcnfY='} 2024-02-19T23:56:17.160 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-02-19T23:56:17.167 INFO:teuthology.task.internal:roles: ubuntu@smithi084.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2'] 2024-02-19T23:56:17.167 INFO:teuthology.task.internal:roles: ubuntu@smithi132.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5'] 2024-02-19T23:56:17.167 INFO:teuthology.run_tasks:Running task console_log... 2024-02-19T23:56:17.228 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7ffaca281f70>, signals=[15]) 2024-02-19T23:56:17.229 INFO:teuthology.run_tasks:Running task internal.connect... 2024-02-19T23:56:17.235 INFO:teuthology.task.internal:Opening connections... 2024-02-19T23:56:17.235 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi084.front.sepia.ceph.com 2024-02-19T23:56:17.238 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi084.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T23:56:17.311 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi132.front.sepia.ceph.com 2024-02-19T23:56:17.312 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi132.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T23:56:17.383 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-02-19T23:56:17.389 DEBUG:teuthology.orchestra.run.smithi084:> uname -m 2024-02-19T23:56:17.411 INFO:teuthology.orchestra.run.smithi084.stdout:x86_64 2024-02-19T23:56:17.412 DEBUG:teuthology.orchestra.run.smithi084:> cat /etc/os-release 2024-02-19T23:56:17.471 INFO:teuthology.orchestra.run.smithi084.stdout:NAME="CentOS Stream" 2024-02-19T23:56:17.471 INFO:teuthology.orchestra.run.smithi084.stdout:VERSION="8" 2024-02-19T23:56:17.471 INFO:teuthology.orchestra.run.smithi084.stdout:ID="centos" 2024-02-19T23:56:17.471 INFO:teuthology.orchestra.run.smithi084.stdout:ID_LIKE="rhel fedora" 2024-02-19T23:56:17.471 INFO:teuthology.orchestra.run.smithi084.stdout:VERSION_ID="8" 2024-02-19T23:56:17.471 INFO:teuthology.orchestra.run.smithi084.stdout:PLATFORM_ID="platform:el8" 2024-02-19T23:56:17.471 INFO:teuthology.orchestra.run.smithi084.stdout:PRETTY_NAME="CentOS Stream 8" 2024-02-19T23:56:17.472 INFO:teuthology.orchestra.run.smithi084.stdout:ANSI_COLOR="0;31" 2024-02-19T23:56:17.472 INFO:teuthology.orchestra.run.smithi084.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-02-19T23:56:17.472 INFO:teuthology.orchestra.run.smithi084.stdout:HOME_URL="https://centos.org/" 2024-02-19T23:56:17.472 INFO:teuthology.orchestra.run.smithi084.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-02-19T23:56:17.472 INFO:teuthology.orchestra.run.smithi084.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-02-19T23:56:17.472 INFO:teuthology.orchestra.run.smithi084.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-02-19T23:56:17.472 INFO:teuthology.lock.ops:Updating smithi084.front.sepia.ceph.com on lock server 2024-02-19T23:56:17.488 DEBUG:teuthology.orchestra.run.smithi132:> uname -m 2024-02-19T23:56:17.510 INFO:teuthology.orchestra.run.smithi132.stdout:x86_64 2024-02-19T23:56:17.510 DEBUG:teuthology.orchestra.run.smithi132:> cat /etc/os-release 2024-02-19T23:56:17.572 INFO:teuthology.orchestra.run.smithi132.stdout:NAME="CentOS Stream" 2024-02-19T23:56:17.572 INFO:teuthology.orchestra.run.smithi132.stdout:VERSION="8" 2024-02-19T23:56:17.572 INFO:teuthology.orchestra.run.smithi132.stdout:ID="centos" 2024-02-19T23:56:17.572 INFO:teuthology.orchestra.run.smithi132.stdout:ID_LIKE="rhel fedora" 2024-02-19T23:56:17.572 INFO:teuthology.orchestra.run.smithi132.stdout:VERSION_ID="8" 2024-02-19T23:56:17.572 INFO:teuthology.orchestra.run.smithi132.stdout:PLATFORM_ID="platform:el8" 2024-02-19T23:56:17.572 INFO:teuthology.orchestra.run.smithi132.stdout:PRETTY_NAME="CentOS Stream 8" 2024-02-19T23:56:17.572 INFO:teuthology.orchestra.run.smithi132.stdout:ANSI_COLOR="0;31" 2024-02-19T23:56:17.572 INFO:teuthology.orchestra.run.smithi132.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-02-19T23:56:17.572 INFO:teuthology.orchestra.run.smithi132.stdout:HOME_URL="https://centos.org/" 2024-02-19T23:56:17.573 INFO:teuthology.orchestra.run.smithi132.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-02-19T23:56:17.573 INFO:teuthology.orchestra.run.smithi132.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-02-19T23:56:17.573 INFO:teuthology.orchestra.run.smithi132.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-02-19T23:56:17.573 INFO:teuthology.lock.ops:Updating smithi132.front.sepia.ceph.com on lock server 2024-02-19T23:56:17.596 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-02-19T23:56:17.605 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-02-19T23:56:17.611 INFO:teuthology.task.internal:Checking for old test directory... 2024-02-19T23:56:17.611 DEBUG:teuthology.orchestra.run.smithi084:> test '!' -e /home/ubuntu/cephtest 2024-02-19T23:56:17.614 DEBUG:teuthology.orchestra.run.smithi132:> test '!' -e /home/ubuntu/cephtest 2024-02-19T23:56:17.633 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-02-19T23:56:17.639 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-02-19T23:56:17.640 DEBUG:teuthology.orchestra.run.smithi084:> test -z $(ls -A /var/lib/ceph) 2024-02-19T23:56:17.673 DEBUG:teuthology.orchestra.run.smithi132:> test -z $(ls -A /var/lib/ceph) 2024-02-19T23:56:17.705 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-02-19T23:56:17.761 INFO:teuthology.run_tasks:Running task kernel... 2024-02-19T23:56:17.774 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-02-19T23:56:17.775 INFO:teuthology.task.kernel:config {'host.a': {'kdb': True, 'sha1': 'distro'}, 'host.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-02-19T23:56:17.775 DEBUG:teuthology.orchestra.run.smithi084:> test -f /run/.containerenv -o -f /.dockerenv 2024-02-19T23:56:17.775 DEBUG:teuthology.orchestra.run.smithi132:> test -f /run/.containerenv -o -f /.dockerenv 2024-02-19T23:56:17.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-19T23:56:17.794 DEBUG:teuthology.orchestra.run.smithi132:> uname -r 2024-02-19T23:56:17.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-19T23:56:17.795 DEBUG:teuthology.orchestra.run.smithi084:> uname -r 2024-02-19T23:56:17.853 INFO:teuthology.orchestra.run.smithi132.stdout:4.18.0-489.el8.x86_64 2024-02-19T23:56:17.853 INFO:teuthology.task.kernel:Running kernel on smithi132: 4.18.0-489.el8.x86_64 2024-02-19T23:56:17.853 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum install -y kernel 2024-02-19T23:56:17.855 INFO:teuthology.orchestra.run.smithi084.stdout:4.18.0-489.el8.x86_64 2024-02-19T23:56:17.855 INFO:teuthology.task.kernel:Running kernel on smithi084: 4.18.0-489.el8.x86_64 2024-02-19T23:56:17.856 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum install -y kernel 2024-02-19T23:56:26.352 INFO:teuthology.orchestra.run.smithi084.stdout:CentOS Stream 8 - AppStream 4.6 MB/s | 27 MB 00:05 2024-02-19T23:56:26.448 INFO:teuthology.orchestra.run.smithi132.stdout:CentOS Stream 8 - AppStream 4.5 MB/s | 27 MB 00:05 2024-02-19T23:56:32.293 INFO:teuthology.orchestra.run.smithi084.stdout:CentOS Stream 8 - BaseOS 9.3 MB/s | 10 MB 00:01 2024-02-19T23:56:32.503 INFO:teuthology.orchestra.run.smithi132.stdout:CentOS Stream 8 - BaseOS 9.2 MB/s | 10 MB 00:01 2024-02-19T23:56:34.060 INFO:teuthology.orchestra.run.smithi084.stdout:CentOS Stream 8 - Extras 131 kB/s | 18 kB 00:00 2024-02-19T23:56:34.265 INFO:teuthology.orchestra.run.smithi132.stdout:CentOS Stream 8 - Extras 129 kB/s | 18 kB 00:00 2024-02-19T23:56:34.394 INFO:teuthology.orchestra.run.smithi084.stdout:CentOS Stream 8 - Extras common packages 31 kB/s | 6.9 kB 00:00 2024-02-19T23:56:34.600 INFO:teuthology.orchestra.run.smithi132.stdout:CentOS Stream 8 - Extras common packages 31 kB/s | 6.9 kB 00:00 2024-02-19T23:56:34.646 INFO:teuthology.orchestra.run.smithi084.stdout:Copr repo for python3-asyncssh owned by ceph 22 kB/s | 3.5 kB 00:00 2024-02-19T23:56:34.851 INFO:teuthology.orchestra.run.smithi132.stdout:Copr repo for python3-asyncssh owned by ceph 22 kB/s | 3.5 kB 00:00 2024-02-19T23:56:35.891 INFO:teuthology.orchestra.run.smithi084.stdout:Extra Packages for Enterprise Linux 14 MB/s | 16 MB 00:01 2024-02-19T23:56:36.619 INFO:teuthology.orchestra.run.smithi132.stdout:Extra Packages for Enterprise Linux 9.6 MB/s | 16 MB 00:01 2024-02-19T23:56:39.713 INFO:teuthology.orchestra.run.smithi084.stdout:lab-extras 398 kB/s | 24 kB 00:00 2024-02-19T23:56:40.394 INFO:teuthology.orchestra.run.smithi132.stdout:lab-extras 399 kB/s | 24 kB 00:00 2024-02-19T23:56:42.693 INFO:teuthology.orchestra.run.smithi084.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-02-19T23:56:42.693 INFO:teuthology.orchestra.run.smithi084.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-02-19T23:56:42.801 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-02-19T23:56:42.801 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-02-19T23:56:42.801 INFO:teuthology.orchestra.run.smithi084.stdout: Package Architecture Version Repository Size 2024-02-19T23:56:42.802 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-02-19T23:56:42.802 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-02-19T23:56:42.802 INFO:teuthology.orchestra.run.smithi084.stdout: kernel x86_64 4.18.0-539.el8 baseos 10 M 2024-02-19T23:56:42.802 INFO:teuthology.orchestra.run.smithi084.stdout:Installing dependencies: 2024-02-19T23:56:42.802 INFO:teuthology.orchestra.run.smithi084.stdout: kernel-core x86_64 4.18.0-539.el8 baseos 43 M 2024-02-19T23:56:42.802 INFO:teuthology.orchestra.run.smithi084.stdout: kernel-modules x86_64 4.18.0-539.el8 baseos 36 M 2024-02-19T23:56:42.802 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-02-19T23:56:42.802 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-02-19T23:56:42.802 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-02-19T23:56:42.802 INFO:teuthology.orchestra.run.smithi084.stdout:Install 3 Packages 2024-02-19T23:56:42.802 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-02-19T23:56:42.803 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 90 M 2024-02-19T23:56:42.803 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 97 M 2024-02-19T23:56:42.803 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-02-19T23:56:43.291 INFO:teuthology.orchestra.run.smithi132.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-02-19T23:56:43.291 INFO:teuthology.orchestra.run.smithi132.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-02-19T23:56:43.400 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-02-19T23:56:43.401 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-02-19T23:56:43.401 INFO:teuthology.orchestra.run.smithi132.stdout: Package Architecture Version Repository Size 2024-02-19T23:56:43.401 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-02-19T23:56:43.401 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2024-02-19T23:56:43.401 INFO:teuthology.orchestra.run.smithi132.stdout: kernel x86_64 4.18.0-539.el8 baseos 10 M 2024-02-19T23:56:43.401 INFO:teuthology.orchestra.run.smithi132.stdout:Installing dependencies: 2024-02-19T23:56:43.401 INFO:teuthology.orchestra.run.smithi132.stdout: kernel-core x86_64 4.18.0-539.el8 baseos 43 M 2024-02-19T23:56:43.401 INFO:teuthology.orchestra.run.smithi132.stdout: kernel-modules x86_64 4.18.0-539.el8 baseos 36 M 2024-02-19T23:56:43.401 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-19T23:56:43.401 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2024-02-19T23:56:43.401 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-02-19T23:56:43.402 INFO:teuthology.orchestra.run.smithi132.stdout:Install 3 Packages 2024-02-19T23:56:43.402 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-19T23:56:43.402 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 90 M 2024-02-19T23:56:43.402 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 97 M 2024-02-19T23:56:43.402 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2024-02-19T23:56:44.164 INFO:teuthology.orchestra.run.smithi084.stdout:(1/3): kernel-4.18.0-539.el8.x86_64.rpm 9.1 MB/s | 10 MB 00:01 2024-02-19T23:56:45.062 INFO:teuthology.orchestra.run.smithi132.stdout:(1/3): kernel-4.18.0-539.el8.x86_64.rpm 7.0 MB/s | 10 MB 00:01 2024-02-19T23:56:45.514 INFO:teuthology.orchestra.run.smithi084.stdout:(2/3): kernel-modules-4.18.0-539.el8.x86_64.rpm 15 MB/s | 36 MB 00:02 2024-02-19T23:56:46.054 INFO:teuthology.orchestra.run.smithi132.stdout:(2/3): kernel-modules-4.18.0-539.el8.x86_64.rpm 15 MB/s | 36 MB 00:02 2024-02-19T23:56:46.114 INFO:teuthology.orchestra.run.smithi084.stdout:(3/3): kernel-core-4.18.0-539.el8.x86_64.rpm 14 MB/s | 43 MB 00:03 2024-02-19T23:56:46.114 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-02-19T23:56:46.114 INFO:teuthology.orchestra.run.smithi084.stdout:Total 27 MB/s | 90 MB 00:03 2024-02-19T23:56:46.679 INFO:teuthology.orchestra.run.smithi132.stdout:(3/3): kernel-core-4.18.0-539.el8.x86_64.rpm 14 MB/s | 43 MB 00:03 2024-02-19T23:56:46.680 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2024-02-19T23:56:46.680 INFO:teuthology.orchestra.run.smithi132.stdout:Total 27 MB/s | 90 MB 00:03 2024-02-19T23:56:46.796 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-02-19T23:56:47.208 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-02-19T23:56:47.209 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-02-19T23:56:47.363 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2024-02-19T23:56:47.793 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2024-02-19T23:56:47.793 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2024-02-19T23:56:48.445 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-02-19T23:56:48.466 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-02-19T23:56:49.057 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2024-02-19T23:56:49.078 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2024-02-19T23:56:51.224 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-02-19T23:56:51.706 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : kernel-core-4.18.0-539.el8.x86_64 1/3 2024-02-19T23:56:51.865 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2024-02-19T23:56:52.381 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : kernel-core-4.18.0-539.el8.x86_64 1/3 2024-02-19T23:56:55.295 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: kernel-core-4.18.0-539.el8.x86_64 1/3 2024-02-19T23:56:55.647 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : kernel-modules-4.18.0-539.el8.x86_64 2/3 2024-02-19T23:56:55.933 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: kernel-core-4.18.0-539.el8.x86_64 1/3 2024-02-19T23:56:56.338 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : kernel-modules-4.18.0-539.el8.x86_64 2/3 2024-02-19T23:56:59.083 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: kernel-modules-4.18.0-539.el8.x86_64 2/3 2024-02-19T23:56:59.463 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : kernel-4.18.0-539.el8.x86_64 3/3 2024-02-19T23:56:59.765 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: kernel-modules-4.18.0-539.el8.x86_64 2/3 2024-02-19T23:57:00.088 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : kernel-4.18.0-539.el8.x86_64 3/3 2024-02-19T23:57:21.460 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: kernel-core-4.18.0-539.el8.x86_64 3/3 2024-02-19T23:57:21.483 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: kernel-modules-4.18.0-539.el8.x86_64 3/3 2024-02-19T23:57:21.825 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: kernel-4.18.0-539.el8.x86_64 3/3 2024-02-19T23:57:21.825 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : kernel-4.18.0-539.el8.x86_64 1/3 2024-02-19T23:57:21.825 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : kernel-core-4.18.0-539.el8.x86_64 2/3 2024-02-19T23:57:22.232 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: kernel-core-4.18.0-539.el8.x86_64 3/3 2024-02-19T23:57:22.255 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: kernel-modules-4.18.0-539.el8.x86_64 3/3 2024-02-19T23:57:22.283 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : kernel-modules-4.18.0-539.el8.x86_64 3/3 2024-02-19T23:57:22.283 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-02-19T23:57:22.283 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-02-19T23:57:22.283 INFO:teuthology.orchestra.run.smithi084.stdout: kernel-4.18.0-539.el8.x86_64 kernel-core-4.18.0-539.el8.x86_64 2024-02-19T23:57:22.283 INFO:teuthology.orchestra.run.smithi084.stdout: kernel-modules-4.18.0-539.el8.x86_64 2024-02-19T23:57:22.283 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-02-19T23:57:22.283 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-02-19T23:57:22.437 DEBUG:teuthology.orchestra.run.smithi084:> rpm -q kernel | sort -rV | head -n 1 2024-02-19T23:57:22.636 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: kernel-4.18.0-539.el8.x86_64 3/3 2024-02-19T23:57:22.636 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : kernel-4.18.0-539.el8.x86_64 1/3 2024-02-19T23:57:22.636 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : kernel-core-4.18.0-539.el8.x86_64 2/3 2024-02-19T23:57:22.653 INFO:teuthology.orchestra.run.smithi084.stdout:kernel-4.18.0-539.el8.x86_64 2024-02-19T23:57:22.653 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-539.el8.x86_64 2024-02-19T23:57:22.654 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-539.el8.x86_64 2024-02-19T23:57:22.654 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-02-19T23:57:22.654 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-02-19T23:57:22.654 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2024-02-19T23:57:22.654 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi084.front.sepia.ceph.com, path=None, version=distro) 2024-02-19T23:57:22.654 DEBUG:teuthology.orchestra.run.smithi084:> rpm -q kernel | sort -rV | head -n 1 2024-02-19T23:57:22.863 INFO:teuthology.orchestra.run.smithi084.stdout:kernel-4.18.0-539.el8.x86_64 2024-02-19T23:57:22.864 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-539.el8.x86_64 2024-02-19T23:57:22.864 DEBUG:teuthology.orchestra.run.smithi084:> sudo rpm -qi grub2-tools 2024-02-19T23:57:22.897 INFO:teuthology.orchestra.run.smithi084.stdout:Name : grub2-tools 2024-02-19T23:57:22.897 INFO:teuthology.orchestra.run.smithi084.stdout:Epoch : 1 2024-02-19T23:57:22.897 INFO:teuthology.orchestra.run.smithi084.stdout:Version : 2.02 2024-02-19T23:57:22.898 INFO:teuthology.orchestra.run.smithi084.stdout:Release : 129.el8 2024-02-19T23:57:22.898 INFO:teuthology.orchestra.run.smithi084.stdout:Architecture: x86_64 2024-02-19T23:57:22.898 INFO:teuthology.orchestra.run.smithi084.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-02-19T23:57:22.898 INFO:teuthology.orchestra.run.smithi084.stdout:Group : System Environment/Base 2024-02-19T23:57:22.898 INFO:teuthology.orchestra.run.smithi084.stdout:Size : 9265586 2024-02-19T23:57:22.898 INFO:teuthology.orchestra.run.smithi084.stdout:License : GPLv3+ 2024-02-19T23:57:22.898 INFO:teuthology.orchestra.run.smithi084.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-02-19T23:57:22.898 INFO:teuthology.orchestra.run.smithi084.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-02-19T23:57:22.898 INFO:teuthology.orchestra.run.smithi084.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-02-19T23:57:22.898 INFO:teuthology.orchestra.run.smithi084.stdout:Build Host : kbuilder.bsys.centos.org 2024-02-19T23:57:22.898 INFO:teuthology.orchestra.run.smithi084.stdout:Relocations : (not relocatable) 2024-02-19T23:57:22.898 INFO:teuthology.orchestra.run.smithi084.stdout:Packager : CentOS BuildSystem 2024-02-19T23:57:22.898 INFO:teuthology.orchestra.run.smithi084.stdout:Vendor : CentOS 2024-02-19T23:57:22.899 INFO:teuthology.orchestra.run.smithi084.stdout:URL : http://www.gnu.org/software/grub/ 2024-02-19T23:57:22.899 INFO:teuthology.orchestra.run.smithi084.stdout:Summary : Support tools for GRUB. 2024-02-19T23:57:22.899 INFO:teuthology.orchestra.run.smithi084.stdout:Description : 2024-02-19T23:57:22.899 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-02-19T23:57:22.899 INFO:teuthology.orchestra.run.smithi084.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-02-19T23:57:22.899 INFO:teuthology.orchestra.run.smithi084.stdout:customizable bootloader with modular architecture. It supports a rich 2024-02-19T23:57:22.899 INFO:teuthology.orchestra.run.smithi084.stdout:variety of kernel formats, file systems, computer architectures and 2024-02-19T23:57:22.899 INFO:teuthology.orchestra.run.smithi084.stdout:hardware devices. 2024-02-19T23:57:22.899 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-02-19T23:57:22.899 INFO:teuthology.orchestra.run.smithi084.stdout:This subpackage provides tools for support of all platforms. 2024-02-19T23:57:22.900 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-02-19T23:57:22.900 INFO:teuthology.task.kernel:Updating grub on smithi084 to boot 4.18.0-539.el8.x86_64 2024-02-19T23:57:22.900 DEBUG:teuthology.orchestra.run.smithi084:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-02-19T23:57:23.325 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : kernel-modules-4.18.0-539.el8.x86_64 3/3 2024-02-19T23:57:23.325 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-19T23:57:23.325 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2024-02-19T23:57:23.325 INFO:teuthology.orchestra.run.smithi132.stdout: kernel-4.18.0-539.el8.x86_64 kernel-core-4.18.0-539.el8.x86_64 2024-02-19T23:57:23.325 INFO:teuthology.orchestra.run.smithi132.stdout: kernel-modules-4.18.0-539.el8.x86_64 2024-02-19T23:57:23.325 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-19T23:57:23.325 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-02-19T23:57:23.473 DEBUG:teuthology.orchestra.run.smithi132:> rpm -q kernel | sort -rV | head -n 1 2024-02-19T23:57:23.690 INFO:teuthology.orchestra.run.smithi132.stdout:kernel-4.18.0-539.el8.x86_64 2024-02-19T23:57:23.690 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-539.el8.x86_64 2024-02-19T23:57:23.690 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-539.el8.x86_64 2024-02-19T23:57:23.690 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-02-19T23:57:23.690 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-02-19T23:57:23.690 INFO:teuthology.task.kernel:Installing distro kernel on host.b... 2024-02-19T23:57:23.690 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi132.front.sepia.ceph.com, path=None, version=distro) 2024-02-19T23:57:23.691 DEBUG:teuthology.orchestra.run.smithi132:> rpm -q kernel | sort -rV | head -n 1 2024-02-19T23:57:23.902 INFO:teuthology.orchestra.run.smithi132.stdout:kernel-4.18.0-539.el8.x86_64 2024-02-19T23:57:23.902 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-539.el8.x86_64 2024-02-19T23:57:23.902 DEBUG:teuthology.orchestra.run.smithi132:> sudo rpm -qi grub2-tools 2024-02-19T23:57:23.936 INFO:teuthology.orchestra.run.smithi132.stdout:Name : grub2-tools 2024-02-19T23:57:23.936 INFO:teuthology.orchestra.run.smithi132.stdout:Epoch : 1 2024-02-19T23:57:23.936 INFO:teuthology.orchestra.run.smithi132.stdout:Version : 2.02 2024-02-19T23:57:23.936 INFO:teuthology.orchestra.run.smithi132.stdout:Release : 129.el8 2024-02-19T23:57:23.936 INFO:teuthology.orchestra.run.smithi132.stdout:Architecture: x86_64 2024-02-19T23:57:23.936 INFO:teuthology.orchestra.run.smithi132.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-02-19T23:57:23.936 INFO:teuthology.orchestra.run.smithi132.stdout:Group : System Environment/Base 2024-02-19T23:57:23.936 INFO:teuthology.orchestra.run.smithi132.stdout:Size : 9265586 2024-02-19T23:57:23.936 INFO:teuthology.orchestra.run.smithi132.stdout:License : GPLv3+ 2024-02-19T23:57:23.937 INFO:teuthology.orchestra.run.smithi132.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-02-19T23:57:23.937 INFO:teuthology.orchestra.run.smithi132.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-02-19T23:57:23.937 INFO:teuthology.orchestra.run.smithi132.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-02-19T23:57:23.937 INFO:teuthology.orchestra.run.smithi132.stdout:Build Host : kbuilder.bsys.centos.org 2024-02-19T23:57:23.937 INFO:teuthology.orchestra.run.smithi132.stdout:Relocations : (not relocatable) 2024-02-19T23:57:23.937 INFO:teuthology.orchestra.run.smithi132.stdout:Packager : CentOS BuildSystem 2024-02-19T23:57:23.937 INFO:teuthology.orchestra.run.smithi132.stdout:Vendor : CentOS 2024-02-19T23:57:23.937 INFO:teuthology.orchestra.run.smithi132.stdout:URL : http://www.gnu.org/software/grub/ 2024-02-19T23:57:23.937 INFO:teuthology.orchestra.run.smithi132.stdout:Summary : Support tools for GRUB. 2024-02-19T23:57:23.937 INFO:teuthology.orchestra.run.smithi132.stdout:Description : 2024-02-19T23:57:23.937 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-19T23:57:23.937 INFO:teuthology.orchestra.run.smithi132.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-02-19T23:57:23.937 INFO:teuthology.orchestra.run.smithi132.stdout:customizable bootloader with modular architecture. It supports a rich 2024-02-19T23:57:23.937 INFO:teuthology.orchestra.run.smithi132.stdout:variety of kernel formats, file systems, computer architectures and 2024-02-19T23:57:23.937 INFO:teuthology.orchestra.run.smithi132.stdout:hardware devices. 2024-02-19T23:57:23.938 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-19T23:57:23.938 INFO:teuthology.orchestra.run.smithi132.stdout:This subpackage provides tools for support of all platforms. 2024-02-19T23:57:23.938 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-02-19T23:57:23.939 INFO:teuthology.task.kernel:Updating grub on smithi132 to boot 4.18.0-539.el8.x86_64 2024-02-19T23:57:23.939 DEBUG:teuthology.orchestra.run.smithi132:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-02-19T23:57:24.697 INFO:teuthology.orchestra.run.smithi084.stderr:Generating grub configuration file ... 2024-02-19T23:57:25.441 INFO:teuthology.orchestra.run.smithi132.stderr:Generating grub configuration file ... 2024-02-19T23:57:26.533 INFO:teuthology.orchestra.run.smithi084.stderr:done 2024-02-19T23:57:26.536 DEBUG:teuthology.orchestra.run.smithi084:> mktemp 2024-02-19T23:57:26.572 INFO:teuthology.orchestra.run.smithi084.stdout:/tmp/tmp.KsRvLX5yrW 2024-02-19T23:57:26.572 DEBUG:teuthology.orchestra.run.smithi084:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.KsRvLX5yrW 2024-02-19T23:57:26.645 DEBUG:teuthology.orchestra.run.smithi084:> sudo chmod 0666 /tmp/tmp.KsRvLX5yrW 2024-02-19T23:57:26.788 DEBUG:teuthology.orchestra.remote:smithi084:/tmp/tmp.KsRvLX5yrW is 6KB 2024-02-19T23:57:26.837 DEBUG:teuthology.orchestra.run.smithi084:> rm -fr /tmp/tmp.KsRvLX5yrW 2024-02-19T23:57:26.854 DEBUG:teuthology.orchestra.run.smithi084:> sudo /bin/ls /boot/loader/entries || true 2024-02-19T23:57:26.936 INFO:teuthology.orchestra.run.smithi084.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-02-19T23:57:26.937 INFO:teuthology.orchestra.run.smithi084.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-02-19T23:57:26.937 INFO:teuthology.orchestra.run.smithi084.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-02-19T23:57:26.937 INFO:teuthology.orchestra.run.smithi084.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-539.el8.x86_64.conf 2024-02-19T23:57:26.938 DEBUG:teuthology.orchestra.run.smithi084:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-539.el8.x86_64 2024-02-19T23:57:27.096 DEBUG:teuthology.orchestra.run.smithi084:> sudo shutdown -r now 2024-02-19T23:57:27.342 INFO:teuthology.orchestra.run.smithi132.stderr:done 2024-02-19T23:57:27.343 DEBUG:teuthology.orchestra.run.smithi132:> mktemp 2024-02-19T23:57:27.366 INFO:teuthology.orchestra.run.smithi132.stdout:/tmp/tmp.fbfPbfyV4U 2024-02-19T23:57:27.367 DEBUG:teuthology.orchestra.run.smithi132:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.fbfPbfyV4U 2024-02-19T23:57:27.452 DEBUG:teuthology.orchestra.run.smithi132:> sudo chmod 0666 /tmp/tmp.fbfPbfyV4U 2024-02-19T23:57:27.554 DEBUG:teuthology.orchestra.remote:smithi132:/tmp/tmp.fbfPbfyV4U is 6KB 2024-02-19T23:57:27.602 DEBUG:teuthology.orchestra.run.smithi132:> rm -fr /tmp/tmp.fbfPbfyV4U 2024-02-19T23:57:27.619 DEBUG:teuthology.orchestra.run.smithi132:> sudo /bin/ls /boot/loader/entries || true 2024-02-19T23:57:27.699 INFO:teuthology.orchestra.run.smithi132.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-02-19T23:57:27.700 INFO:teuthology.orchestra.run.smithi132.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-02-19T23:57:27.700 INFO:teuthology.orchestra.run.smithi132.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-02-19T23:57:27.700 INFO:teuthology.orchestra.run.smithi132.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-539.el8.x86_64.conf 2024-02-19T23:57:27.701 DEBUG:teuthology.orchestra.run.smithi132:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-539.el8.x86_64 2024-02-19T23:57:27.862 DEBUG:teuthology.orchestra.run.smithi132:> sudo shutdown -r now 2024-02-19T23:57:57.128 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2024-02-19T23:57:57.129 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi084.front.sepia.ceph.com' 2024-02-19T23:57:57.130 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi084.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T23:57:57.865 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2024-02-19T23:57:57.865 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi132.front.sepia.ceph.com' 2024-02-19T23:57:57.866 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi132.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T23:58:15.612 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.84 2024-02-19T23:58:24.620 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi084.front.sepia.ceph.com' 2024-02-19T23:58:24.621 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi084.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T23:58:27.680 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.84 2024-02-19T23:58:39.686 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi084.front.sepia.ceph.com' 2024-02-19T23:58:39.687 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi084.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T23:58:57.884 DEBUG:teuthology.orchestra.remote:timed out 2024-02-19T23:59:06.893 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi132.front.sepia.ceph.com' 2024-02-19T23:59:06.894 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi132.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T23:59:07.154 DEBUG:teuthology.orchestra.run.smithi132:> true 2024-02-19T23:59:07.418 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi132.front.sepia.ceph.com' 2024-02-19T23:59:07.418 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "4.18.0-539.el8.x86_64"... 2024-02-19T23:59:07.418 DEBUG:teuthology.orchestra.run.smithi132:> uname -r 2024-02-19T23:59:07.435 INFO:teuthology.orchestra.run.smithi132.stdout:4.18.0-539.el8.x86_64 2024-02-19T23:59:07.435 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-539.el8.x86_64 vs 4.18.0-539.el8.x86_64 2024-02-19T23:59:07.435 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-02-19T23:59:07.435 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-02-19T23:59:08.436 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2024-02-19T23:59:08.436 DEBUG:teuthology.orchestra.run.smithi132:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-02-19T23:59:08.678 INFO:teuthology.orchestra.run.smithi132.stdout:ttyS1 2024-02-19T23:59:08.700 DEBUG:teuthology.parallel:result is None 2024-02-19T23:59:12.174 DEBUG:teuthology.orchestra.run.smithi084:> true 2024-02-19T23:59:12.620 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi084.front.sepia.ceph.com' 2024-02-19T23:59:12.621 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "4.18.0-539.el8.x86_64"... 2024-02-19T23:59:12.621 DEBUG:teuthology.orchestra.run.smithi084:> uname -r 2024-02-19T23:59:12.675 INFO:teuthology.orchestra.run.smithi084.stdout:4.18.0-539.el8.x86_64 2024-02-19T23:59:12.675 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-539.el8.x86_64 vs 4.18.0-539.el8.x86_64 2024-02-19T23:59:12.675 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-02-19T23:59:12.675 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-02-19T23:59:13.676 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2024-02-19T23:59:13.676 DEBUG:teuthology.orchestra.run.smithi084:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-02-19T23:59:13.760 INFO:teuthology.orchestra.run.smithi084.stdout:ttyS1 2024-02-19T23:59:13.784 DEBUG:teuthology.parallel:result is None 2024-02-19T23:59:13.785 INFO:teuthology.run_tasks:Running task internal.base... 2024-02-19T23:59:13.791 INFO:teuthology.task.internal:Creating test directory... 2024-02-19T23:59:13.791 DEBUG:teuthology.orchestra.run.smithi084:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-02-19T23:59:13.794 DEBUG:teuthology.orchestra.run.smithi132:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-02-19T23:59:13.817 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-02-19T23:59:13.824 INFO:teuthology.run_tasks:Running task internal.archive... 2024-02-19T23:59:13.832 INFO:teuthology.task.internal:Creating archive directory... 2024-02-19T23:59:13.833 DEBUG:teuthology.orchestra.run.smithi084:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-02-19T23:59:13.854 DEBUG:teuthology.orchestra.run.smithi132:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-02-19T23:59:13.891 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-02-19T23:59:13.899 INFO:teuthology.task.internal:Enabling coredump saving... 2024-02-19T23:59:13.899 DEBUG:teuthology.orchestra.run.smithi084:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-02-19T23:59:13.928 DEBUG:teuthology.orchestra.run.smithi132:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-02-19T23:59:13.968 INFO:teuthology.orchestra.run.smithi084.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-19T23:59:13.971 INFO:teuthology.orchestra.run.smithi132.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-19T23:59:13.978 INFO:teuthology.orchestra.run.smithi084.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-19T23:59:13.982 INFO:teuthology.orchestra.run.smithi132.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-19T23:59:13.984 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-02-19T23:59:13.991 INFO:teuthology.task.internal:Configuring sudo... 2024-02-19T23:59:13.991 DEBUG:teuthology.orchestra.run.smithi084:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-02-19T23:59:14.023 DEBUG:teuthology.orchestra.run.smithi132:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-02-19T23:59:14.054 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-02-19T23:59:14.063 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-02-19T23:59:14.063 DEBUG:teuthology.orchestra.run.smithi084:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-02-19T23:59:14.094 DEBUG:teuthology.orchestra.run.smithi132:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-02-19T23:59:14.115 DEBUG:teuthology.orchestra.run.smithi084:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-19T23:59:14.202 DEBUG:teuthology.orchestra.run.smithi084:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-19T23:59:14.252 DEBUG:teuthology.orchestra.run.smithi084:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-19T23:59:14.338 DEBUG:teuthology.orchestra.run.smithi084:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-19T23:59:14.381 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-02-19T23:59:14.381 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-02-19T23:59:14.448 DEBUG:teuthology.orchestra.run.smithi132:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-19T23:59:14.498 DEBUG:teuthology.orchestra.run.smithi132:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-19T23:59:14.552 DEBUG:teuthology.orchestra.run.smithi132:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-19T23:59:14.639 DEBUG:teuthology.orchestra.run.smithi132:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-19T23:59:14.684 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-02-19T23:59:14.685 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-02-19T23:59:14.754 DEBUG:teuthology.orchestra.run.smithi084:> sudo service rsyslog restart 2024-02-19T23:59:14.756 DEBUG:teuthology.orchestra.run.smithi132:> sudo service rsyslog restart 2024-02-19T23:59:14.811 INFO:teuthology.orchestra.run.smithi084.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-19T23:59:14.859 INFO:teuthology.orchestra.run.smithi132.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-19T23:59:15.291 INFO:teuthology.run_tasks:Running task internal.timer... 2024-02-19T23:59:15.299 INFO:teuthology.task.internal:Starting timer... 2024-02-19T23:59:15.299 INFO:teuthology.run_tasks:Running task pcp... 2024-02-19T23:59:15.308 INFO:teuthology.run_tasks:Running task selinux... 2024-02-19T23:59:15.316 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-02-19T23:59:15.317 DEBUG:teuthology.orchestra.run.smithi084:> sudo service auditd rotate 2024-02-19T23:59:15.393 INFO:teuthology.orchestra.run.smithi084.stdout:Rotating logs: 2024-02-19T23:59:15.395 DEBUG:teuthology.orchestra.run.smithi132:> sudo service auditd rotate 2024-02-19T23:59:15.477 INFO:teuthology.orchestra.run.smithi132.stdout:Rotating logs: 2024-02-19T23:59:15.479 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-02-19T23:59:15.479 DEBUG:teuthology.orchestra.run.smithi084:> /usr/sbin/getenforce 2024-02-19T23:59:15.513 INFO:teuthology.orchestra.run.smithi084.stdout:Permissive 2024-02-19T23:59:15.513 DEBUG:teuthology.orchestra.run.smithi132:> /usr/sbin/getenforce 2024-02-19T23:59:15.537 INFO:teuthology.orchestra.run.smithi132.stdout:Permissive 2024-02-19T23:59:15.538 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi084.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi132.front.sepia.ceph.com': 'permissive'} 2024-02-19T23:59:15.538 DEBUG:teuthology.orchestra.run.smithi084:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-19T23:59:15.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-19T23:59:15.582 DEBUG:teuthology.orchestra.run.smithi132:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-19T23:59:15.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-19T23:59:15.609 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-02-19T23:59:15.609 DEBUG:teuthology.orchestra.run.smithi084:> sudo /usr/sbin/setenforce permissive 2024-02-19T23:59:15.650 DEBUG:teuthology.orchestra.run.smithi132:> sudo /usr/sbin/setenforce permissive 2024-02-19T23:59:15.676 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-02-19T23:59:15.687 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-02-19T23:59:15.703 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-02-19T23:59:15.705 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi084.front.sepia.ceph.com,smithi132.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-02-20T00:05:03.446 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi084.front.sepia.ceph.com'), Remote(name='ubuntu@smithi132.front.sepia.ceph.com')] 2024-02-20T00:05:03.448 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi084.front.sepia.ceph.com' 2024-02-20T00:05:03.452 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi084.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T00:05:03.532 DEBUG:teuthology.orchestra.run.smithi084:> true 2024-02-20T00:05:03.607 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi084.front.sepia.ceph.com' 2024-02-20T00:05:03.608 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi132.front.sepia.ceph.com' 2024-02-20T00:05:03.608 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi132.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T00:05:03.680 DEBUG:teuthology.orchestra.run.smithi132:> true 2024-02-20T00:05:03.746 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi132.front.sepia.ceph.com' 2024-02-20T00:05:03.747 INFO:teuthology.run_tasks:Running task clock... 2024-02-20T00:05:03.757 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-02-20T00:05:03.757 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-02-20T00:05:03.758 DEBUG:teuthology.orchestra.run.smithi084:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-20T00:05:03.760 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-02-20T00:05:03.760 DEBUG:teuthology.orchestra.run.smithi132:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-20T00:05:03.790 INFO:teuthology.orchestra.run.smithi084.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-02-20T00:05:03.804 INFO:teuthology.orchestra.run.smithi084.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-02-20T00:05:03.818 INFO:teuthology.orchestra.run.smithi132.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-02-20T00:05:03.831 INFO:teuthology.orchestra.run.smithi132.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-02-20T00:05:03.875 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: ntpd: command not found 2024-02-20T00:05:03.886 INFO:teuthology.orchestra.run.smithi084.stdout:506 Cannot talk to daemon 2024-02-20T00:05:03.894 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: ntpd: command not found 2024-02-20T00:05:03.899 INFO:teuthology.orchestra.run.smithi084.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-02-20T00:05:03.905 INFO:teuthology.orchestra.run.smithi132.stdout:506 Cannot talk to daemon 2024-02-20T00:05:03.912 INFO:teuthology.orchestra.run.smithi084.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-02-20T00:05:03.919 INFO:teuthology.orchestra.run.smithi132.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-02-20T00:05:03.932 INFO:teuthology.orchestra.run.smithi132.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-02-20T00:05:03.972 INFO:teuthology.orchestra.run.smithi084.stderr:bash: ntpq: command not found 2024-02-20T00:05:03.976 INFO:teuthology.orchestra.run.smithi084.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-20T00:05:03.976 INFO:teuthology.orchestra.run.smithi084.stdout:=============================================================================== 2024-02-20T00:05:03.976 INFO:teuthology.orchestra.run.smithi084.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-20T00:05:03.976 INFO:teuthology.orchestra.run.smithi084.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-20T00:05:03.976 INFO:teuthology.orchestra.run.smithi084.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-20T00:05:03.976 INFO:teuthology.orchestra.run.smithi084.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-20T00:05:03.999 INFO:teuthology.orchestra.run.smithi132.stderr:bash: ntpq: command not found 2024-02-20T00:05:04.002 INFO:teuthology.orchestra.run.smithi132.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-20T00:05:04.002 INFO:teuthology.orchestra.run.smithi132.stdout:=============================================================================== 2024-02-20T00:05:04.003 INFO:teuthology.orchestra.run.smithi132.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-20T00:05:04.003 INFO:teuthology.orchestra.run.smithi132.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-20T00:05:04.003 INFO:teuthology.orchestra.run.smithi132.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-20T00:05:04.003 INFO:teuthology.orchestra.run.smithi132.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-20T00:05:04.004 INFO:teuthology.run_tasks:Running task pexec... 2024-02-20T00:05:04.032 INFO:teuthology.task.pexec:Executing custom commands... 2024-02-20T00:05:04.032 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi084.front.sepia.ceph.com 2024-02-20T00:05:04.033 DEBUG:teuthology.orchestra.run.smithi084:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-02-20T00:05:04.033 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi132.front.sepia.ceph.com 2024-02-20T00:05:04.034 DEBUG:teuthology.orchestra.run.smithi132:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-02-20T00:05:04.814 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:01:24 ago on Tue 20 Feb 2024 12:03:40 AM UTC. 2024-02-20T00:05:04.830 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:01:23 ago on Tue 20 Feb 2024 12:03:41 AM UTC. 2024-02-20T00:05:06.837 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-02-20T00:05:06.838 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-02-20T00:05:06.838 INFO:teuthology.orchestra.run.smithi132.stdout: Package Architecture Version Repository Size 2024-02-20T00:05:06.838 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-02-20T00:05:06.838 INFO:teuthology.orchestra.run.smithi132.stdout:Resetting modules: 2024-02-20T00:05:06.838 INFO:teuthology.orchestra.run.smithi132.stdout: container-tools 2024-02-20T00:05:06.838 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:05:06.838 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2024-02-20T00:05:06.838 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-02-20T00:05:06.838 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:05:06.844 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-02-20T00:05:06.846 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-02-20T00:05:06.846 INFO:teuthology.orchestra.run.smithi084.stdout: Package Architecture Version Repository Size 2024-02-20T00:05:06.846 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-02-20T00:05:06.846 INFO:teuthology.orchestra.run.smithi084.stdout:Resetting modules: 2024-02-20T00:05:06.846 INFO:teuthology.orchestra.run.smithi084.stdout: container-tools 2024-02-20T00:05:06.846 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-02-20T00:05:06.846 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-02-20T00:05:06.846 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-02-20T00:05:06.846 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-02-20T00:05:07.092 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-02-20T00:05:07.107 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-02-20T00:05:07.952 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:01:26 ago on Tue 20 Feb 2024 12:03:41 AM UTC. 2024-02-20T00:05:07.952 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:01:27 ago on Tue 20 Feb 2024 12:03:40 AM UTC. 2024-02-20T00:05:09.969 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-02-20T00:05:09.975 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-02-20T00:05:10.005 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-02-20T00:05:10.006 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repository Size 2024-02-20T00:05:10.006 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-02-20T00:05:10.006 INFO:teuthology.orchestra.run.smithi132.stdout:Installing group/module packages: 2024-02-20T00:05:10.006 INFO:teuthology.orchestra.run.smithi132.stdout: aardvark-dns x86_64 2:1.10.0-1.module_el8+871+552be958 appstream 1.0 M 2024-02-20T00:05:10.006 INFO:teuthology.orchestra.run.smithi132.stdout: buildah x86_64 2:1.33.5-1.module_el8+885+7da147f3 appstream 9.7 M 2024-02-20T00:05:10.006 INFO:teuthology.orchestra.run.smithi132.stdout: cockpit-podman noarch 83-1.module_el8+847+7863d4e6 appstream 688 k 2024-02-20T00:05:10.006 INFO:teuthology.orchestra.run.smithi132.stdout: crun x86_64 1.14-1.module_el8+871+552be958 appstream 257 k 2024-02-20T00:05:10.006 INFO:teuthology.orchestra.run.smithi132.stdout: netavark x86_64 2:1.10.2-1.module_el8+885+7da147f3 appstream 4.3 M 2024-02-20T00:05:10.006 INFO:teuthology.orchestra.run.smithi132.stdout: python3-podman noarch 4.9.0-1.module_el8+847+7863d4e6 appstream 180 k 2024-02-20T00:05:10.006 INFO:teuthology.orchestra.run.smithi132.stdout: skopeo x86_64 2:1.14.3-0.1.module_el8+885+7da147f3 appstream 8.8 M 2024-02-20T00:05:10.006 INFO:teuthology.orchestra.run.smithi132.stdout: toolbox x86_64 0.0.99.5-1.module_el8+847+7863d4e6 appstream 2.5 M 2024-02-20T00:05:10.006 INFO:teuthology.orchestra.run.smithi132.stdout: udica noarch 0.2.6-20.module_el8+804+f131391c appstream 53 k 2024-02-20T00:05:10.006 INFO:teuthology.orchestra.run.smithi132.stdout:Installing dependencies: 2024-02-20T00:05:10.006 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2024-02-20T00:05:10.007 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2024-02-20T00:05:10.007 INFO:teuthology.orchestra.run.smithi132.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2024-02-20T00:05:10.007 INFO:teuthology.orchestra.run.smithi132.stdout:Installing weak dependencies: 2024-02-20T00:05:10.007 INFO:teuthology.orchestra.run.smithi132.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2024-02-20T00:05:10.007 INFO:teuthology.orchestra.run.smithi132.stdout:Installing module profiles: 2024-02-20T00:05:10.007 INFO:teuthology.orchestra.run.smithi132.stdout: container-tools/common 2024-02-20T00:05:10.007 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:05:10.007 INFO:teuthology.orchestra.run.smithi132.stdout:Enabling module streams: 2024-02-20T00:05:10.007 INFO:teuthology.orchestra.run.smithi132.stdout: container-tools rhel8 2024-02-20T00:05:10.007 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:05:10.007 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2024-02-20T00:05:10.007 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-02-20T00:05:10.007 INFO:teuthology.orchestra.run.smithi132.stdout:Install 13 Packages 2024-02-20T00:05:10.007 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:05:10.008 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 28 M 2024-02-20T00:05:10.008 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 92 M 2024-02-20T00:05:10.008 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2024-02-20T00:05:10.010 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-02-20T00:05:10.011 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repository Size 2024-02-20T00:05:10.011 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-02-20T00:05:10.011 INFO:teuthology.orchestra.run.smithi084.stdout:Installing group/module packages: 2024-02-20T00:05:10.011 INFO:teuthology.orchestra.run.smithi084.stdout: aardvark-dns x86_64 2:1.10.0-1.module_el8+871+552be958 appstream 1.0 M 2024-02-20T00:05:10.011 INFO:teuthology.orchestra.run.smithi084.stdout: buildah x86_64 2:1.33.5-1.module_el8+885+7da147f3 appstream 9.7 M 2024-02-20T00:05:10.011 INFO:teuthology.orchestra.run.smithi084.stdout: cockpit-podman noarch 83-1.module_el8+847+7863d4e6 appstream 688 k 2024-02-20T00:05:10.011 INFO:teuthology.orchestra.run.smithi084.stdout: crun x86_64 1.14-1.module_el8+871+552be958 appstream 257 k 2024-02-20T00:05:10.011 INFO:teuthology.orchestra.run.smithi084.stdout: netavark x86_64 2:1.10.2-1.module_el8+885+7da147f3 appstream 4.3 M 2024-02-20T00:05:10.011 INFO:teuthology.orchestra.run.smithi084.stdout: python3-podman noarch 4.9.0-1.module_el8+847+7863d4e6 appstream 180 k 2024-02-20T00:05:10.011 INFO:teuthology.orchestra.run.smithi084.stdout: skopeo x86_64 2:1.14.3-0.1.module_el8+885+7da147f3 appstream 8.8 M 2024-02-20T00:05:10.011 INFO:teuthology.orchestra.run.smithi084.stdout: toolbox x86_64 0.0.99.5-1.module_el8+847+7863d4e6 appstream 2.5 M 2024-02-20T00:05:10.011 INFO:teuthology.orchestra.run.smithi084.stdout: udica noarch 0.2.6-20.module_el8+804+f131391c appstream 53 k 2024-02-20T00:05:10.011 INFO:teuthology.orchestra.run.smithi084.stdout:Installing dependencies: 2024-02-20T00:05:10.011 INFO:teuthology.orchestra.run.smithi084.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2024-02-20T00:05:10.011 INFO:teuthology.orchestra.run.smithi084.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2024-02-20T00:05:10.012 INFO:teuthology.orchestra.run.smithi084.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2024-02-20T00:05:10.012 INFO:teuthology.orchestra.run.smithi084.stdout:Installing weak dependencies: 2024-02-20T00:05:10.012 INFO:teuthology.orchestra.run.smithi084.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2024-02-20T00:05:10.012 INFO:teuthology.orchestra.run.smithi084.stdout:Installing module profiles: 2024-02-20T00:05:10.012 INFO:teuthology.orchestra.run.smithi084.stdout: container-tools/common 2024-02-20T00:05:10.012 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-02-20T00:05:10.012 INFO:teuthology.orchestra.run.smithi084.stdout:Enabling module streams: 2024-02-20T00:05:10.012 INFO:teuthology.orchestra.run.smithi084.stdout: container-tools rhel8 2024-02-20T00:05:10.012 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-02-20T00:05:10.012 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-02-20T00:05:10.012 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-02-20T00:05:10.012 INFO:teuthology.orchestra.run.smithi084.stdout:Install 13 Packages 2024-02-20T00:05:10.012 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-02-20T00:05:10.013 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 28 M 2024-02-20T00:05:10.013 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 92 M 2024-02-20T00:05:10.013 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-02-20T00:05:10.187 INFO:teuthology.orchestra.run.smithi084.stdout:(1/13): cockpit-podman-83-1.module_el8+847+7863 5.1 MB/s | 688 kB 00:00 2024-02-20T00:05:10.237 INFO:teuthology.orchestra.run.smithi084.stdout:(2/13): aardvark-dns-1.10.0-1.module_el8+871+55 5.5 MB/s | 1.0 MB 00:00 2024-02-20T00:05:10.262 INFO:teuthology.orchestra.run.smithi084.stdout:(3/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 504 kB/s | 38 kB 00:00 2024-02-20T00:05:10.288 INFO:teuthology.orchestra.run.smithi084.stdout:(4/13): crun-1.14-1.module_el8+871+552be958.x86 5.0 MB/s | 257 kB 00:00 2024-02-20T00:05:10.365 INFO:teuthology.orchestra.run.smithi084.stdout:(5/13): python3-podman-4.9.0-1.module_el8+847+7 2.3 MB/s | 180 kB 00:00 2024-02-20T00:05:10.392 INFO:teuthology.orchestra.run.smithi132.stdout:(1/13): aardvark-dns-1.10.0-1.module_el8+871+55 5.3 MB/s | 1.0 MB 00:00 2024-02-20T00:05:10.421 INFO:teuthology.orchestra.run.smithi084.stdout:(6/13): python3-pytoml-0.1.14-5.git7dea353.el8. 439 kB/s | 25 kB 00:00 2024-02-20T00:05:10.434 INFO:teuthology.orchestra.run.smithi132.stdout:(2/13): cockpit-podman-83-1.module_el8+847+7863 2.9 MB/s | 688 kB 00:00 2024-02-20T00:05:10.459 INFO:teuthology.orchestra.run.smithi132.stdout:(3/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 566 kB/s | 38 kB 00:00 2024-02-20T00:05:10.485 INFO:teuthology.orchestra.run.smithi132.stdout:(4/13): crun-1.14-1.module_el8+871+552be958.x86 5.0 MB/s | 257 kB 00:00 2024-02-20T00:05:10.497 INFO:teuthology.orchestra.run.smithi084.stdout:(7/13): python3-pyxdg-0.25-16.el8.noarch.rpm 1.2 MB/s | 94 kB 00:00 2024-02-20T00:05:10.572 INFO:teuthology.orchestra.run.smithi084.stdout:(8/13): netavark-1.10.2-1.module_el8+885+7da147 14 MB/s | 4.3 MB 00:00 2024-02-20T00:05:10.585 INFO:teuthology.orchestra.run.smithi132.stdout:(5/13): python3-podman-4.9.0-1.module_el8+847+7 1.8 MB/s | 180 kB 00:00 2024-02-20T00:05:10.635 INFO:teuthology.orchestra.run.smithi132.stdout:(6/13): python3-pytoml-0.1.14-5.git7dea353.el8. 496 kB/s | 25 kB 00:00 2024-02-20T00:05:10.686 INFO:teuthology.orchestra.run.smithi132.stdout:(7/13): python3-pyxdg-0.25-16.el8.noarch.rpm 1.8 MB/s | 94 kB 00:00 2024-02-20T00:05:10.764 INFO:teuthology.orchestra.run.smithi084.stdout:(9/13): buildah-1.33.5-1.module_el8+885+7da147f 14 MB/s | 9.7 MB 00:00 2024-02-20T00:05:10.770 INFO:teuthology.orchestra.run.smithi132.stdout:(8/13): netavark-1.10.2-1.module_el8+885+7da147 14 MB/s | 4.3 MB 00:00 2024-02-20T00:05:10.814 INFO:teuthology.orchestra.run.smithi084.stdout:(10/13): toolbox-0.0.99.5-1.module_el8+847+7863 10 MB/s | 2.5 MB 00:00 2024-02-20T00:05:10.890 INFO:teuthology.orchestra.run.smithi084.stdout:(11/13): yajl-2.1.0-12.el8.x86_64.rpm 545 kB/s | 41 kB 00:00 2024-02-20T00:05:10.895 INFO:teuthology.orchestra.run.smithi132.stdout:(9/13): buildah-1.33.5-1.module_el8+885+7da147f 14 MB/s | 9.7 MB 00:00 2024-02-20T00:05:10.915 INFO:teuthology.orchestra.run.smithi084.stdout:(12/13): udica-0.2.6-20.module_el8+804+f131391c 350 kB/s | 53 kB 00:00 2024-02-20T00:05:10.970 INFO:teuthology.orchestra.run.smithi132.stdout:(10/13): toolbox-0.0.99.5-1.module_el8+847+7863 13 MB/s | 2.5 MB 00:00 2024-02-20T00:05:10.996 INFO:teuthology.orchestra.run.smithi132.stdout:(11/13): udica-0.2.6-20.module_el8+804+f131391c 525 kB/s | 53 kB 00:00 2024-02-20T00:05:11.021 INFO:teuthology.orchestra.run.smithi132.stdout:(12/13): yajl-2.1.0-12.el8.x86_64.rpm 815 kB/s | 41 kB 00:00 2024-02-20T00:05:11.048 INFO:teuthology.orchestra.run.smithi084.stdout:(13/13): skopeo-1.14.3-0.1.module_el8+885+7da14 16 MB/s | 8.8 MB 00:00 2024-02-20T00:05:11.049 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-02-20T00:05:11.049 INFO:teuthology.orchestra.run.smithi084.stdout:Total 27 MB/s | 28 MB 00:01 2024-02-20T00:05:11.316 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-02-20T00:05:11.329 INFO:teuthology.orchestra.run.smithi132.stdout:(13/13): skopeo-1.14.3-0.1.module_el8+885+7da14 14 MB/s | 8.8 MB 00:00 2024-02-20T00:05:11.330 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2024-02-20T00:05:11.330 INFO:teuthology.orchestra.run.smithi132.stdout:Total 21 MB/s | 28 MB 00:01 2024-02-20T00:05:11.336 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-02-20T00:05:11.336 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-02-20T00:05:11.567 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-02-20T00:05:11.570 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-02-20T00:05:11.598 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2024-02-20T00:05:11.620 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2024-02-20T00:05:11.620 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2024-02-20T00:05:11.854 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2024-02-20T00:05:11.857 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2024-02-20T00:05:12.244 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-02-20T00:05:12.555 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2024-02-20T00:05:13.195 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2024-02-20T00:05:13.412 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 2/13 2024-02-20T00:05:13.487 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2024-02-20T00:05:13.591 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 3/13 2024-02-20T00:05:13.671 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 2/13 2024-02-20T00:05:13.790 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 4/13 2024-02-20T00:05:13.831 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 3/13 2024-02-20T00:05:14.043 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 5/13 2024-02-20T00:05:14.060 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 4/13 2024-02-20T00:05:14.299 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 5/13 2024-02-20T00:05:14.646 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x8 6/13 2024-02-20T00:05:14.878 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x8 6/13 2024-02-20T00:05:15.597 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : netavark-2:1.10.2-1.module_el8+885+7da147f3.x86_64 7/13 2024-02-20T00:05:15.829 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : buildah-2:1.33.5-1.module_el8+885+7da147f3.x86_64 8/13 2024-02-20T00:05:15.831 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : netavark-2:1.10.2-1.module_el8+885+7da147f3.x86_64 7/13 2024-02-20T00:05:16.056 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : crun-1.14-1.module_el8+871+552be958.x86_64 9/13 2024-02-20T00:05:16.071 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : buildah-2:1.33.5-1.module_el8+885+7da147f3.x86_64 8/13 2024-02-20T00:05:16.274 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : crun-1.14-1.module_el8+871+552be958.x86_64 9/13 2024-02-20T00:05:16.609 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-podman-4.9.0-1.module_el8+847+7863d4e6.noa 10/13 2024-02-20T00:05:16.705 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-podman-4.9.0-1.module_el8+847+7863d4e6.noa 10/13 2024-02-20T00:05:16.836 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : toolbox-0.0.99.5-1.module_el8+847+7863d4e6.x86_64 11/13 2024-02-20T00:05:16.953 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : toolbox-0.0.99.5-1.module_el8+847+7863d4e6.x86_64 11/13 2024-02-20T00:05:17.057 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/13 2024-02-20T00:05:17.187 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/13 2024-02-20T00:05:17.248 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : cockpit-podman-83-1.module_el8+847+7863d4e6.noarch 13/13 2024-02-20T00:05:17.380 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : cockpit-podman-83-1.module_el8+847+7863d4e6.noarch 13/13 2024-02-20T00:05:17.652 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: cockpit-podman-83-1.module_el8+847+7863d4e6.noarch 13/13 2024-02-20T00:05:17.653 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x8 1/13 2024-02-20T00:05:17.653 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : buildah-2:1.33.5-1.module_el8+885+7da147f3.x86_64 2/13 2024-02-20T00:05:17.653 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : cockpit-podman-83-1.module_el8+847+7863d4e6.noarch 3/13 2024-02-20T00:05:17.653 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-02-20T00:05:17.653 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : crun-1.14-1.module_el8+871+552be958.x86_64 5/13 2024-02-20T00:05:17.653 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : netavark-2:1.10.2-1.module_el8+885+7da147f3.x86_64 6/13 2024-02-20T00:05:17.653 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-podman-4.9.0-1.module_el8+847+7863d4e6.noa 7/13 2024-02-20T00:05:17.653 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2024-02-20T00:05:17.653 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2024-02-20T00:05:17.653 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 10/13 2024-02-20T00:05:17.653 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : toolbox-0.0.99.5-1.module_el8+847+7863d4e6.x86_64 11/13 2024-02-20T00:05:17.653 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/13 2024-02-20T00:05:17.838 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: cockpit-podman-83-1.module_el8+847+7863d4e6.noarch 13/13 2024-02-20T00:05:17.838 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x8 1/13 2024-02-20T00:05:17.838 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : buildah-2:1.33.5-1.module_el8+885+7da147f3.x86_64 2/13 2024-02-20T00:05:17.838 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : cockpit-podman-83-1.module_el8+847+7863d4e6.noarch 3/13 2024-02-20T00:05:17.838 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-02-20T00:05:17.838 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : crun-1.14-1.module_el8+871+552be958.x86_64 5/13 2024-02-20T00:05:17.838 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : netavark-2:1.10.2-1.module_el8+885+7da147f3.x86_64 6/13 2024-02-20T00:05:17.838 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-podman-4.9.0-1.module_el8+847+7863d4e6.noa 7/13 2024-02-20T00:05:17.838 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2024-02-20T00:05:17.838 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2024-02-20T00:05:17.839 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 10/13 2024-02-20T00:05:17.839 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : toolbox-0.0.99.5-1.module_el8+847+7863d4e6.x86_64 11/13 2024-02-20T00:05:17.839 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/13 2024-02-20T00:05:18.156 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2024-02-20T00:05:18.156 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-02-20T00:05:18.156 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-02-20T00:05:18.156 INFO:teuthology.orchestra.run.smithi084.stdout: aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x86_64 2024-02-20T00:05:18.156 INFO:teuthology.orchestra.run.smithi084.stdout: buildah-2:1.33.5-1.module_el8+885+7da147f3.x86_64 2024-02-20T00:05:18.156 INFO:teuthology.orchestra.run.smithi084.stdout: cockpit-podman-83-1.module_el8+847+7863d4e6.noarch 2024-02-20T00:05:18.156 INFO:teuthology.orchestra.run.smithi084.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2024-02-20T00:05:18.156 INFO:teuthology.orchestra.run.smithi084.stdout: crun-1.14-1.module_el8+871+552be958.x86_64 2024-02-20T00:05:18.157 INFO:teuthology.orchestra.run.smithi084.stdout: netavark-2:1.10.2-1.module_el8+885+7da147f3.x86_64 2024-02-20T00:05:18.157 INFO:teuthology.orchestra.run.smithi084.stdout: python3-podman-4.9.0-1.module_el8+847+7863d4e6.noarch 2024-02-20T00:05:18.157 INFO:teuthology.orchestra.run.smithi084.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2024-02-20T00:05:18.157 INFO:teuthology.orchestra.run.smithi084.stdout: python3-pyxdg-0.25-16.el8.noarch 2024-02-20T00:05:18.157 INFO:teuthology.orchestra.run.smithi084.stdout: skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 2024-02-20T00:05:18.157 INFO:teuthology.orchestra.run.smithi084.stdout: toolbox-0.0.99.5-1.module_el8+847+7863d4e6.x86_64 2024-02-20T00:05:18.157 INFO:teuthology.orchestra.run.smithi084.stdout: udica-0.2.6-20.module_el8+804+f131391c.noarch 2024-02-20T00:05:18.157 INFO:teuthology.orchestra.run.smithi084.stdout: yajl-2.1.0-12.el8.x86_64 2024-02-20T00:05:18.157 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-02-20T00:05:18.157 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-02-20T00:05:18.441 DEBUG:teuthology.parallel:result is None 2024-02-20T00:05:18.487 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2024-02-20T00:05:18.487 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:05:18.487 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2024-02-20T00:05:18.487 INFO:teuthology.orchestra.run.smithi132.stdout: aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x86_64 2024-02-20T00:05:18.487 INFO:teuthology.orchestra.run.smithi132.stdout: buildah-2:1.33.5-1.module_el8+885+7da147f3.x86_64 2024-02-20T00:05:18.487 INFO:teuthology.orchestra.run.smithi132.stdout: cockpit-podman-83-1.module_el8+847+7863d4e6.noarch 2024-02-20T00:05:18.487 INFO:teuthology.orchestra.run.smithi132.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2024-02-20T00:05:18.488 INFO:teuthology.orchestra.run.smithi132.stdout: crun-1.14-1.module_el8+871+552be958.x86_64 2024-02-20T00:05:18.488 INFO:teuthology.orchestra.run.smithi132.stdout: netavark-2:1.10.2-1.module_el8+885+7da147f3.x86_64 2024-02-20T00:05:18.488 INFO:teuthology.orchestra.run.smithi132.stdout: python3-podman-4.9.0-1.module_el8+847+7863d4e6.noarch 2024-02-20T00:05:18.488 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2024-02-20T00:05:18.488 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pyxdg-0.25-16.el8.noarch 2024-02-20T00:05:18.488 INFO:teuthology.orchestra.run.smithi132.stdout: skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 2024-02-20T00:05:18.488 INFO:teuthology.orchestra.run.smithi132.stdout: toolbox-0.0.99.5-1.module_el8+847+7863d4e6.x86_64 2024-02-20T00:05:18.488 INFO:teuthology.orchestra.run.smithi132.stdout: udica-0.2.6-20.module_el8+804+f131391c.noarch 2024-02-20T00:05:18.488 INFO:teuthology.orchestra.run.smithi132.stdout: yajl-2.1.0-12.el8.x86_64 2024-02-20T00:05:18.488 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:05:18.488 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-02-20T00:05:18.790 DEBUG:teuthology.parallel:result is None 2024-02-20T00:05:18.790 INFO:teuthology.run_tasks:Running task pexec... 2024-02-20T00:05:18.813 INFO:teuthology.task.pexec:Executing custom commands... 2024-02-20T00:05:18.813 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi084.front.sepia.ceph.com 2024-02-20T00:05:18.814 DEBUG:teuthology.orchestra.run.smithi084:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-02-20T00:05:18.814 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi132.front.sepia.ceph.com 2024-02-20T00:05:18.814 DEBUG:teuthology.orchestra.run.smithi132:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-02-20T00:05:18.990 DEBUG:teuthology.parallel:result is None 2024-02-20T00:05:18.992 DEBUG:teuthology.parallel:result is None 2024-02-20T00:05:18.992 INFO:teuthology.run_tasks:Running task install... 2024-02-20T00:05:19.002 DEBUG:teuthology.task.install:project ceph 2024-02-20T00:05:19.002 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '19854089e18d4f65dda2b6cd74e737367c2514bd'}} 2024-02-20T00:05:19.002 DEBUG:teuthology.task.install:config {'exclude_packages': ['ceph-volume'], 'tag': 'v16.2.4', 'flavor': 'default', 'sha1': '19854089e18d4f65dda2b6cd74e737367c2514bd'} 2024-02-20T00:05:19.002 INFO:teuthology.task.install:Using flavor: default 2024-02-20T00:05:19.011 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-02-20T00:05:19.011 INFO:teuthology.task.install:extra packages: [] 2024-02-20T00:05:19.011 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '19854089e18d4f65dda2b6cd74e737367c2514bd', 'tag': 'v16.2.4', 'wait_for_package': False} 2024-02-20T00:05:19.011 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-02-20T00:05:19.011 INFO:teuthology.packaging:ref: None 2024-02-20T00:05:19.011 INFO:teuthology.packaging:tag: v16.2.4 2024-02-20T00:05:19.011 INFO:teuthology.packaging:branch: None 2024-02-20T00:05:19.012 INFO:teuthology.packaging:sha1: 19854089e18d4f65dda2b6cd74e737367c2514bd 2024-02-20T00:05:19.367 DEBUG:teuthology.repo_utils:git ls-remote https://git.ceph.com/ceph-ci v16.2.4^{} -> 3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2024-02-20T00:05:19.368 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2024-02-20T00:05:19.370 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '19854089e18d4f65dda2b6cd74e737367c2514bd', 'tag': 'v16.2.4', 'wait_for_package': False} 2024-02-20T00:05:19.371 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-02-20T00:05:19.371 INFO:teuthology.packaging:ref: None 2024-02-20T00:05:19.371 INFO:teuthology.packaging:tag: v16.2.4 2024-02-20T00:05:19.371 INFO:teuthology.packaging:branch: None 2024-02-20T00:05:19.371 INFO:teuthology.packaging:sha1: 19854089e18d4f65dda2b6cd74e737367c2514bd 2024-02-20T00:05:19.371 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2024-02-20T00:05:19.513 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/ 2024-02-20T00:05:19.513 INFO:teuthology.task.install.rpm:Package version is 16.2.4-0 2024-02-20T00:05:19.526 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/ 2024-02-20T00:05:19.527 INFO:teuthology.task.install.rpm:Package version is 16.2.4-0 2024-02-20T00:05:19.568 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-02-20T00:05:19.569 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-02-20T00:05:19.569 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-02-20T00:05:19.574 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-02-20T00:05:19.574 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-02-20T00:05:19.574 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-02-20T00:05:19.597 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-02-20T00:05:19.597 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-02-20T00:05:19.597 INFO:teuthology.packaging:ref: None 2024-02-20T00:05:19.597 INFO:teuthology.packaging:tag: v16.2.4 2024-02-20T00:05:19.597 INFO:teuthology.packaging:branch: None 2024-02-20T00:05:19.597 INFO:teuthology.packaging:sha1: 19854089e18d4f65dda2b6cd74e737367c2514bd 2024-02-20T00:05:19.598 DEBUG:teuthology.orchestra.run.smithi084:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/v16.2.4/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-02-20T00:05:19.603 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-02-20T00:05:19.603 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-02-20T00:05:19.603 INFO:teuthology.packaging:ref: None 2024-02-20T00:05:19.603 INFO:teuthology.packaging:tag: v16.2.4 2024-02-20T00:05:19.603 INFO:teuthology.packaging:branch: None 2024-02-20T00:05:19.603 INFO:teuthology.packaging:sha1: 19854089e18d4f65dda2b6cd74e737367c2514bd 2024-02-20T00:05:19.603 DEBUG:teuthology.orchestra.run.smithi132:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/v16.2.4/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-02-20T00:05:19.665 DEBUG:teuthology.orchestra.run.smithi084:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-02-20T00:05:19.672 DEBUG:teuthology.orchestra.run.smithi132:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-02-20T00:05:19.742 DEBUG:teuthology.orchestra.run.smithi084:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-02-20T00:05:19.749 DEBUG:teuthology.orchestra.run.smithi132:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-02-20T00:05:19.808 INFO:teuthology.orchestra.run.smithi084.stdout:check_obsoletes = 1 2024-02-20T00:05:19.809 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum clean all 2024-02-20T00:05:19.815 INFO:teuthology.orchestra.run.smithi132.stdout:check_obsoletes = 1 2024-02-20T00:05:19.816 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum clean all 2024-02-20T00:05:20.072 INFO:teuthology.orchestra.run.smithi084.stdout:76 files removed 2024-02-20T00:05:20.080 INFO:teuthology.orchestra.run.smithi132.stdout:76 files removed 2024-02-20T00:05:20.100 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install ceph-radosgw 2024-02-20T00:05:20.108 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install ceph-radosgw 2024-02-20T00:05:20.588 INFO:teuthology.orchestra.run.smithi084.stdout:ceph packages for x86_64 946 kB/s | 74 kB 00:00 2024-02-20T00:05:20.613 INFO:teuthology.orchestra.run.smithi132.stdout:ceph packages for x86_64 741 kB/s | 74 kB 00:00 2024-02-20T00:05:20.697 INFO:teuthology.orchestra.run.smithi084.stdout:ceph noarch packages 214 kB/s | 15 kB 00:00 2024-02-20T00:05:20.730 INFO:teuthology.orchestra.run.smithi132.stdout:ceph noarch packages 191 kB/s | 15 kB 00:00 2024-02-20T00:05:20.814 INFO:teuthology.orchestra.run.smithi132.stdout:ceph source packages 32 kB/s | 1.7 kB 00:00 2024-02-20T00:05:20.831 INFO:teuthology.orchestra.run.smithi084.stdout:ceph source packages 17 kB/s | 1.7 kB 00:00 2024-02-20T00:05:26.776 INFO:teuthology.orchestra.run.smithi132.stdout:CentOS Stream 8 - AppStream 4.6 MB/s | 27 MB 00:05 2024-02-20T00:05:27.123 INFO:teuthology.orchestra.run.smithi084.stdout:CentOS Stream 8 - AppStream 4.3 MB/s | 27 MB 00:06 2024-02-20T00:05:32.982 INFO:teuthology.orchestra.run.smithi132.stdout:CentOS Stream 8 - BaseOS 7.8 MB/s | 10 MB 00:01 2024-02-20T00:05:33.190 INFO:teuthology.orchestra.run.smithi084.stdout:CentOS Stream 8 - BaseOS 8.3 MB/s | 10 MB 00:01 2024-02-20T00:05:34.716 INFO:teuthology.orchestra.run.smithi132.stdout:CentOS Stream 8 - Extras 139 kB/s | 18 kB 00:00 2024-02-20T00:05:35.048 INFO:teuthology.orchestra.run.smithi084.stdout:CentOS Stream 8 - Extras 65 kB/s | 18 kB 00:00 2024-02-20T00:05:35.209 INFO:teuthology.orchestra.run.smithi132.stdout:CentOS Stream 8 - Extras common packages 18 kB/s | 6.9 kB 00:00 2024-02-20T00:05:35.524 INFO:teuthology.orchestra.run.smithi084.stdout:CentOS Stream 8 - Extras common packages 19 kB/s | 6.9 kB 00:00 2024-02-20T00:05:35.910 INFO:teuthology.orchestra.run.smithi132.stdout:CentOS Stream 8 - PowerTools 7.9 MB/s | 4.8 MB 00:00 2024-02-20T00:05:36.225 INFO:teuthology.orchestra.run.smithi084.stdout:CentOS Stream 8 - PowerTools 8.0 MB/s | 4.8 MB 00:00 2024-02-20T00:05:36.962 INFO:teuthology.orchestra.run.smithi132.stdout:Copr repo for python3-asyncssh owned by ceph 25 kB/s | 3.5 kB 00:00 2024-02-20T00:05:37.268 INFO:teuthology.orchestra.run.smithi084.stdout:Copr repo for python3-asyncssh owned by ceph 26 kB/s | 3.5 kB 00:00 2024-02-20T00:05:37.612 INFO:teuthology.orchestra.run.smithi132.stdout:Extra Packages for Enterprise Linux 30 MB/s | 16 MB 00:00 2024-02-20T00:05:38.147 INFO:teuthology.orchestra.run.smithi084.stdout:Extra Packages for Enterprise Linux 21 MB/s | 16 MB 00:00 2024-02-20T00:05:41.354 INFO:teuthology.orchestra.run.smithi132.stdout:lab-extras 613 kB/s | 24 kB 00:00 2024-02-20T00:05:41.960 INFO:teuthology.orchestra.run.smithi084.stdout:lab-extras 432 kB/s | 24 kB 00:00 2024-02-20T00:05:44.891 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-02-20T00:05:44.893 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-02-20T00:05:44.894 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repository Size 2024-02-20T00:05:44.894 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-02-20T00:05:44.894 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2024-02-20T00:05:44.894 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-radosgw x86_64 2:16.2.4-0.el8 ceph 13 M 2024-02-20T00:05:44.894 INFO:teuthology.orchestra.run.smithi132.stdout:Upgrading: 2024-02-20T00:05:44.894 INFO:teuthology.orchestra.run.smithi132.stdout: librados2 x86_64 2:16.2.4-0.el8 ceph 3.7 M 2024-02-20T00:05:44.894 INFO:teuthology.orchestra.run.smithi132.stdout: librbd1 x86_64 2:16.2.4-0.el8 ceph 4.0 M 2024-02-20T00:05:44.894 INFO:teuthology.orchestra.run.smithi132.stdout:Installing dependencies: 2024-02-20T00:05:44.894 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-base x86_64 2:16.2.4-0.el8 ceph 6.0 M 2024-02-20T00:05:44.894 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-common x86_64 2:16.2.4-0.el8 ceph 24 M 2024-02-20T00:05:44.894 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-selinux x86_64 2:16.2.4-0.el8 ceph 25 k 2024-02-20T00:05:44.894 INFO:teuthology.orchestra.run.smithi132.stdout: daxctl-libs x86_64 71.1-7.el8 baseos 42 k 2024-02-20T00:05:44.894 INFO:teuthology.orchestra.run.smithi132.stdout: fmt x86_64 6.2.1-1.el8 epel 116 k 2024-02-20T00:05:44.894 INFO:teuthology.orchestra.run.smithi132.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-02-20T00:05:44.895 INFO:teuthology.orchestra.run.smithi132.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2024-02-20T00:05:44.895 INFO:teuthology.orchestra.run.smithi132.stdout: libcephfs2 x86_64 2:16.2.4-0.el8 ceph 801 k 2024-02-20T00:05:44.895 INFO:teuthology.orchestra.run.smithi132.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-02-20T00:05:44.895 INFO:teuthology.orchestra.run.smithi132.stdout: libpmemobj x86_64 1.6.1-1.el8 appstream 145 k 2024-02-20T00:05:44.895 INFO:teuthology.orchestra.run.smithi132.stdout: librabbitmq x86_64 0.9.0-5.el8 baseos 47 k 2024-02-20T00:05:44.895 INFO:teuthology.orchestra.run.smithi132.stdout: libradosstriper1 x86_64 2:16.2.4-0.el8 ceph 486 k 2024-02-20T00:05:44.895 INFO:teuthology.orchestra.run.smithi132.stdout: librdkafka x86_64 1.6.1-1.el8 appstream 677 k 2024-02-20T00:05:44.895 INFO:teuthology.orchestra.run.smithi132.stdout: librgw2 x86_64 2:16.2.4-0.el8 ceph 4.1 M 2024-02-20T00:05:44.895 INFO:teuthology.orchestra.run.smithi132.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-02-20T00:05:44.895 INFO:teuthology.orchestra.run.smithi132.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2024-02-20T00:05:44.895 INFO:teuthology.orchestra.run.smithi132.stdout: ndctl-libs x86_64 71.1-7.el8 baseos 80 k 2024-02-20T00:05:44.895 INFO:teuthology.orchestra.run.smithi132.stdout: python3-ceph-argparse x86_64 2:16.2.4-0.el8 ceph 44 k 2024-02-20T00:05:44.895 INFO:teuthology.orchestra.run.smithi132.stdout: python3-ceph-common x86_64 2:16.2.4-0.el8 ceph 80 k 2024-02-20T00:05:44.895 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cephfs x86_64 2:16.2.4-0.el8 ceph 213 k 2024-02-20T00:05:44.896 INFO:teuthology.orchestra.run.smithi132.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2024-02-20T00:05:44.896 INFO:teuthology.orchestra.run.smithi132.stdout: python3-rados x86_64 2:16.2.4-0.el8 ceph 387 k 2024-02-20T00:05:44.896 INFO:teuthology.orchestra.run.smithi132.stdout: python3-rbd x86_64 2:16.2.4-0.el8 ceph 368 k 2024-02-20T00:05:44.896 INFO:teuthology.orchestra.run.smithi132.stdout: python3-rgw x86_64 2:16.2.4-0.el8 ceph 114 k 2024-02-20T00:05:44.896 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:05:44.896 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2024-02-20T00:05:44.896 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-02-20T00:05:44.896 INFO:teuthology.orchestra.run.smithi132.stdout:Install 25 Packages 2024-02-20T00:05:44.896 INFO:teuthology.orchestra.run.smithi132.stdout:Upgrade 2 Packages 2024-02-20T00:05:44.896 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:05:44.897 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 59 M 2024-02-20T00:05:44.897 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2024-02-20T00:05:45.347 INFO:teuthology.orchestra.run.smithi132.stdout:(1/27): ceph-base-16.2.4-0.el8.x86_64.rpm 40 MB/s | 6.0 MB 00:00 2024-02-20T00:05:45.372 INFO:teuthology.orchestra.run.smithi132.stdout:(2/27): ceph-selinux-16.2.4-0.el8.x86_64.rpm 991 kB/s | 25 kB 00:00 2024-02-20T00:05:45.405 INFO:teuthology.orchestra.run.smithi132.stdout:(3/27): libcephfs2-16.2.4-0.el8.x86_64.rpm 24 MB/s | 801 kB 00:00 2024-02-20T00:05:45.439 INFO:teuthology.orchestra.run.smithi132.stdout:(4/27): libradosstriper1-16.2.4-0.el8.x86_64.rp 14 MB/s | 486 kB 00:00 2024-02-20T00:05:45.455 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-02-20T00:05:45.459 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-02-20T00:05:45.459 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repository Size 2024-02-20T00:05:45.459 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-02-20T00:05:45.459 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-02-20T00:05:45.459 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-radosgw x86_64 2:16.2.4-0.el8 ceph 13 M 2024-02-20T00:05:45.459 INFO:teuthology.orchestra.run.smithi084.stdout:Upgrading: 2024-02-20T00:05:45.459 INFO:teuthology.orchestra.run.smithi084.stdout: librados2 x86_64 2:16.2.4-0.el8 ceph 3.7 M 2024-02-20T00:05:45.459 INFO:teuthology.orchestra.run.smithi084.stdout: librbd1 x86_64 2:16.2.4-0.el8 ceph 4.0 M 2024-02-20T00:05:45.459 INFO:teuthology.orchestra.run.smithi084.stdout:Installing dependencies: 2024-02-20T00:05:45.459 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-base x86_64 2:16.2.4-0.el8 ceph 6.0 M 2024-02-20T00:05:45.459 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-common x86_64 2:16.2.4-0.el8 ceph 24 M 2024-02-20T00:05:45.460 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-selinux x86_64 2:16.2.4-0.el8 ceph 25 k 2024-02-20T00:05:45.460 INFO:teuthology.orchestra.run.smithi084.stdout: daxctl-libs x86_64 71.1-7.el8 baseos 42 k 2024-02-20T00:05:45.460 INFO:teuthology.orchestra.run.smithi084.stdout: fmt x86_64 6.2.1-1.el8 epel 116 k 2024-02-20T00:05:45.460 INFO:teuthology.orchestra.run.smithi084.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-02-20T00:05:45.460 INFO:teuthology.orchestra.run.smithi084.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2024-02-20T00:05:45.460 INFO:teuthology.orchestra.run.smithi084.stdout: libcephfs2 x86_64 2:16.2.4-0.el8 ceph 801 k 2024-02-20T00:05:45.460 INFO:teuthology.orchestra.run.smithi084.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-02-20T00:05:45.460 INFO:teuthology.orchestra.run.smithi084.stdout: libpmemobj x86_64 1.6.1-1.el8 appstream 145 k 2024-02-20T00:05:45.460 INFO:teuthology.orchestra.run.smithi084.stdout: librabbitmq x86_64 0.9.0-5.el8 baseos 47 k 2024-02-20T00:05:45.460 INFO:teuthology.orchestra.run.smithi084.stdout: libradosstriper1 x86_64 2:16.2.4-0.el8 ceph 486 k 2024-02-20T00:05:45.460 INFO:teuthology.orchestra.run.smithi084.stdout: librdkafka x86_64 1.6.1-1.el8 appstream 677 k 2024-02-20T00:05:45.460 INFO:teuthology.orchestra.run.smithi084.stdout: librgw2 x86_64 2:16.2.4-0.el8 ceph 4.1 M 2024-02-20T00:05:45.460 INFO:teuthology.orchestra.run.smithi084.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-02-20T00:05:45.460 INFO:teuthology.orchestra.run.smithi084.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2024-02-20T00:05:45.461 INFO:teuthology.orchestra.run.smithi084.stdout: ndctl-libs x86_64 71.1-7.el8 baseos 80 k 2024-02-20T00:05:45.461 INFO:teuthology.orchestra.run.smithi084.stdout: python3-ceph-argparse x86_64 2:16.2.4-0.el8 ceph 44 k 2024-02-20T00:05:45.461 INFO:teuthology.orchestra.run.smithi084.stdout: python3-ceph-common x86_64 2:16.2.4-0.el8 ceph 80 k 2024-02-20T00:05:45.461 INFO:teuthology.orchestra.run.smithi084.stdout: python3-cephfs x86_64 2:16.2.4-0.el8 ceph 213 k 2024-02-20T00:05:45.461 INFO:teuthology.orchestra.run.smithi084.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2024-02-20T00:05:45.461 INFO:teuthology.orchestra.run.smithi084.stdout: python3-rados x86_64 2:16.2.4-0.el8 ceph 387 k 2024-02-20T00:05:45.461 INFO:teuthology.orchestra.run.smithi084.stdout: python3-rbd x86_64 2:16.2.4-0.el8 ceph 368 k 2024-02-20T00:05:45.461 INFO:teuthology.orchestra.run.smithi084.stdout: python3-rgw x86_64 2:16.2.4-0.el8 ceph 114 k 2024-02-20T00:05:45.461 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-02-20T00:05:45.461 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-02-20T00:05:45.461 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-02-20T00:05:45.461 INFO:teuthology.orchestra.run.smithi084.stdout:Install 25 Packages 2024-02-20T00:05:45.461 INFO:teuthology.orchestra.run.smithi084.stdout:Upgrade 2 Packages 2024-02-20T00:05:45.461 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-02-20T00:05:45.462 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 59 M 2024-02-20T00:05:45.462 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-02-20T00:05:45.539 INFO:teuthology.orchestra.run.smithi132.stdout:(5/27): librgw2-16.2.4-0.el8.x86_64.rpm 41 MB/s | 4.1 MB 00:00 2024-02-20T00:05:45.565 INFO:teuthology.orchestra.run.smithi132.stdout:(6/27): python3-ceph-argparse-16.2.4-0.el8.x86_ 1.8 MB/s | 44 kB 00:00 2024-02-20T00:05:45.590 INFO:teuthology.orchestra.run.smithi132.stdout:(7/27): python3-ceph-common-16.2.4-0.el8.x86_64 3.1 MB/s | 80 kB 00:00 2024-02-20T00:05:45.624 INFO:teuthology.orchestra.run.smithi132.stdout:(8/27): python3-cephfs-16.2.4-0.el8.x86_64.rpm 6.2 MB/s | 213 kB 00:00 2024-02-20T00:05:45.758 INFO:teuthology.orchestra.run.smithi132.stdout:(9/27): ceph-radosgw-16.2.4-0.el8.x86_64.rpm 23 MB/s | 13 MB 00:00 2024-02-20T00:05:45.783 INFO:teuthology.orchestra.run.smithi132.stdout:(10/27): python3-rados-16.2.4-0.el8.x86_64.rpm 2.4 MB/s | 387 kB 00:00 2024-02-20T00:05:45.808 INFO:teuthology.orchestra.run.smithi132.stdout:(11/27): python3-rbd-16.2.4-0.el8.x86_64.rpm 7.2 MB/s | 368 kB 00:00 2024-02-20T00:05:45.833 INFO:teuthology.orchestra.run.smithi132.stdout:(12/27): python3-rgw-16.2.4-0.el8.x86_64.rpm 2.2 MB/s | 114 kB 00:00 2024-02-20T00:05:45.994 INFO:teuthology.orchestra.run.smithi084.stdout:(1/27): ceph-base-16.2.4-0.el8.x86_64.rpm 26 MB/s | 6.0 MB 00:00 2024-02-20T00:05:46.011 INFO:teuthology.orchestra.run.smithi084.stdout:(2/27): ceph-selinux-16.2.4-0.el8.x86_64.rpm 1.5 MB/s | 25 kB 00:00 2024-02-20T00:05:46.045 INFO:teuthology.orchestra.run.smithi084.stdout:(3/27): libcephfs2-16.2.4-0.el8.x86_64.rpm 24 MB/s | 801 kB 00:00 2024-02-20T00:05:46.079 INFO:teuthology.orchestra.run.smithi084.stdout:(4/27): libradosstriper1-16.2.4-0.el8.x86_64.rp 14 MB/s | 486 kB 00:00 2024-02-20T00:05:46.159 INFO:teuthology.orchestra.run.smithi132.stdout:(13/27): ceph-common-16.2.4-0.el8.x86_64.rpm 25 MB/s | 24 MB 00:00 2024-02-20T00:05:46.184 INFO:teuthology.orchestra.run.smithi132.stdout:(14/27): libpmemobj-1.6.1-1.el8.x86_64.rpm 386 kB/s | 145 kB 00:00 2024-02-20T00:05:46.226 INFO:teuthology.orchestra.run.smithi132.stdout:(15/27): python3-prettytable-0.7.2-14.el8.noarc 1.0 MB/s | 44 kB 00:00 2024-02-20T00:05:46.251 INFO:teuthology.orchestra.run.smithi132.stdout:(16/27): librdkafka-1.6.1-1.el8.x86_64.rpm 1.6 MB/s | 677 kB 00:00 2024-02-20T00:05:46.254 INFO:teuthology.orchestra.run.smithi084.stdout:(5/27): ceph-radosgw-16.2.4-0.el8.x86_64.rpm 26 MB/s | 13 MB 00:00 2024-02-20T00:05:46.279 INFO:teuthology.orchestra.run.smithi084.stdout:(6/27): python3-ceph-argparse-16.2.4-0.el8.x86_ 1.7 MB/s | 44 kB 00:00 2024-02-20T00:05:46.296 INFO:teuthology.orchestra.run.smithi084.stdout:(7/27): python3-ceph-common-16.2.4-0.el8.x86_64 4.7 MB/s | 80 kB 00:00 2024-02-20T00:05:46.301 INFO:teuthology.orchestra.run.smithi132.stdout:(17/27): lttng-ust-2.8.1-11.el8.x86_64.rpm 1.8 MB/s | 259 kB 00:00 2024-02-20T00:05:46.530 INFO:teuthology.orchestra.run.smithi084.stdout:(8/27): ceph-common-16.2.4-0.el8.x86_64.rpm 31 MB/s | 24 MB 00:00 2024-02-20T00:05:46.555 INFO:teuthology.orchestra.run.smithi084.stdout:(9/27): python3-cephfs-16.2.4-0.el8.x86_64.rpm 825 kB/s | 213 kB 00:00 2024-02-20T00:05:46.560 INFO:teuthology.orchestra.run.smithi132.stdout:(18/27): daxctl-libs-71.1-7.el8.x86_64.rpm 126 kB/s | 42 kB 00:00 2024-02-20T00:05:46.577 INFO:teuthology.orchestra.run.smithi132.stdout:(19/27): librabbitmq-0.9.0-5.el8.x86_64.rpm 144 kB/s | 47 kB 00:00 2024-02-20T00:05:46.613 INFO:teuthology.orchestra.run.smithi084.stdout:(10/27): librgw2-16.2.4-0.el8.x86_64.rpm 7.6 MB/s | 4.1 MB 00:00 2024-02-20T00:05:46.639 INFO:teuthology.orchestra.run.smithi084.stdout:(11/27): python3-rados-16.2.4-0.el8.x86_64.rpm 3.5 MB/s | 387 kB 00:00 2024-02-20T00:05:46.664 INFO:teuthology.orchestra.run.smithi084.stdout:(12/27): python3-rgw-16.2.4-0.el8.x86_64.rpm 2.2 MB/s | 114 kB 00:00 2024-02-20T00:05:46.689 INFO:teuthology.orchestra.run.smithi084.stdout:(13/27): python3-rbd-16.2.4-0.el8.x86_64.rpm 2.7 MB/s | 368 kB 00:00 2024-02-20T00:05:46.694 INFO:teuthology.orchestra.run.smithi132.stdout:(20/27): ndctl-libs-71.1-7.el8.x86_64.rpm 204 kB/s | 80 kB 00:00 2024-02-20T00:05:46.748 INFO:teuthology.orchestra.run.smithi084.stdout:(14/27): libpmemobj-1.6.1-1.el8.x86_64.rpm 1.3 MB/s | 145 kB 00:00 2024-02-20T00:05:46.807 INFO:teuthology.orchestra.run.smithi084.stdout:(15/27): python3-prettytable-0.7.2-14.el8.noarc 753 kB/s | 44 kB 00:00 2024-02-20T00:05:46.832 INFO:teuthology.orchestra.run.smithi084.stdout:(16/27): lttng-ust-2.8.1-11.el8.x86_64.rpm 1.8 MB/s | 259 kB 00:00 2024-02-20T00:05:46.865 INFO:teuthology.orchestra.run.smithi084.stdout:(17/27): librdkafka-1.6.1-1.el8.x86_64.rpm 3.3 MB/s | 677 kB 00:00 2024-02-20T00:05:46.911 INFO:teuthology.orchestra.run.smithi132.stdout:(21/27): fmt-6.2.1-1.el8.x86_64.rpm 332 kB/s | 116 kB 00:00 2024-02-20T00:05:47.149 INFO:teuthology.orchestra.run.smithi084.stdout:(18/27): daxctl-libs-71.1-7.el8.x86_64.rpm 123 kB/s | 42 kB 00:00 2024-02-20T00:05:47.162 INFO:teuthology.orchestra.run.smithi132.stdout:(22/27): gperftools-libs-2.7-9.el8.x86_64.rpm 523 kB/s | 306 kB 00:00 2024-02-20T00:05:47.174 INFO:teuthology.orchestra.run.smithi084.stdout:(19/27): librabbitmq-0.9.0-5.el8.x86_64.rpm 137 kB/s | 47 kB 00:00 2024-02-20T00:05:47.225 INFO:teuthology.orchestra.run.smithi084.stdout:(20/27): ndctl-libs-71.1-7.el8.x86_64.rpm 223 kB/s | 80 kB 00:00 2024-02-20T00:05:47.367 INFO:teuthology.orchestra.run.smithi132.stdout:(23/27): liboath-2.6.2-3.el8.x86_64.rpm 129 kB/s | 59 kB 00:00 2024-02-20T00:05:47.613 INFO:teuthology.orchestra.run.smithi132.stdout:(24/27): leveldb-1.22-1.el8.x86_64.rpm 197 kB/s | 181 kB 00:00 2024-02-20T00:05:47.724 INFO:teuthology.orchestra.run.smithi084.stdout:(21/27): fmt-6.2.1-1.el8.x86_64.rpm 203 kB/s | 116 kB 00:00 2024-02-20T00:05:47.860 INFO:teuthology.orchestra.run.smithi084.stdout:(22/27): leveldb-1.22-1.el8.x86_64.rpm 289 kB/s | 181 kB 00:00 2024-02-20T00:05:47.938 INFO:teuthology.orchestra.run.smithi132.stdout:(25/27): libunwind-1.3.1-3.el8.x86_64.rpm 97 kB/s | 75 kB 00:00 2024-02-20T00:05:48.022 INFO:teuthology.orchestra.run.smithi132.stdout:(26/27): librados2-16.2.4-0.el8.x86_64.rpm 5.7 MB/s | 3.7 MB 00:00 2024-02-20T00:05:48.089 INFO:teuthology.orchestra.run.smithi132.stdout:(27/27): librbd1-16.2.4-0.el8.x86_64.rpm 8.5 MB/s | 4.0 MB 00:00 2024-02-20T00:05:48.094 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2024-02-20T00:05:48.094 INFO:teuthology.orchestra.run.smithi132.stdout:Total 18 MB/s | 59 MB 00:03 2024-02-20T00:05:48.134 INFO:teuthology.orchestra.run.smithi084.stdout:(23/27): liboath-2.6.2-3.el8.x86_64.rpm 143 kB/s | 59 kB 00:00 2024-02-20T00:05:48.163 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2024-02-20T00:05:48.212 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2024-02-20T00:05:48.213 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2024-02-20T00:05:48.393 INFO:teuthology.orchestra.run.smithi084.stdout:(24/27): gperftools-libs-2.7-9.el8.x86_64.rpm 251 kB/s | 306 kB 00:01 2024-02-20T00:05:48.418 INFO:teuthology.orchestra.run.smithi084.stdout:(25/27): libunwind-1.3.1-3.el8.x86_64.rpm 134 kB/s | 75 kB 00:00 2024-02-20T00:05:48.502 INFO:teuthology.orchestra.run.smithi084.stdout:(26/27): librados2-16.2.4-0.el8.x86_64.rpm 10 MB/s | 3.7 MB 00:00 2024-02-20T00:05:48.555 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2024-02-20T00:05:48.558 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2024-02-20T00:05:48.583 INFO:teuthology.orchestra.run.smithi084.stdout:(27/27): librbd1-16.2.4-0.el8.x86_64.rpm 21 MB/s | 4.0 MB 00:00 2024-02-20T00:05:48.588 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-02-20T00:05:48.589 INFO:teuthology.orchestra.run.smithi084.stdout:Total 19 MB/s | 59 MB 00:03 2024-02-20T00:05:48.658 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-02-20T00:05:48.706 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-02-20T00:05:48.706 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-02-20T00:05:49.045 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-02-20T00:05:49.048 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-02-20T00:05:49.664 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2024-02-20T00:05:49.881 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : fmt-6.2.1-1.el8.x86_64 1/29 2024-02-20T00:05:50.085 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/29 2024-02-20T00:05:50.210 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-02-20T00:05:50.284 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/29 2024-02-20T00:05:50.435 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : fmt-6.2.1-1.el8.x86_64 1/29 2024-02-20T00:05:50.629 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/29 2024-02-20T00:05:50.687 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/29 2024-02-20T00:05:50.782 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/29 2024-02-20T00:05:50.867 INFO:teuthology.orchestra.run.smithi132.stdout: Upgrading : librados2-2:16.2.4-0.el8.x86_64 4/29 2024-02-20T00:05:50.963 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: librados2-2:16.2.4-0.el8.x86_64 4/29 2024-02-20T00:05:51.179 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/29 2024-02-20T00:05:51.199 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-rados-2:16.2.4-0.el8.x86_64 5/29 2024-02-20T00:05:51.334 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2024-02-20T00:05:51.366 INFO:teuthology.orchestra.run.smithi084.stdout: Upgrading : librados2-2:16.2.4-0.el8.x86_64 4/29 2024-02-20T00:05:51.385 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2024-02-20T00:05:51.463 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: librados2-2:16.2.4-0.el8.x86_64 4/29 2024-02-20T00:05:51.618 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : librabbitmq-0.9.0-5.el8.x86_64 7/29 2024-02-20T00:05:51.706 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-rados-2:16.2.4-0.el8.x86_64 5/29 2024-02-20T00:05:51.858 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2024-02-20T00:05:51.922 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2024-02-20T00:05:52.151 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : librdkafka-1.6.1-1.el8.x86_64 8/29 2024-02-20T00:05:52.158 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : librabbitmq-0.9.0-5.el8.x86_64 7/29 2024-02-20T00:05:52.283 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : librgw2-2:16.2.4-0.el8.x86_64 9/29 2024-02-20T00:05:52.359 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: librgw2-2:16.2.4-0.el8.x86_64 9/29 2024-02-20T00:05:52.555 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/29 2024-02-20T00:05:52.691 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : librdkafka-1.6.1-1.el8.x86_64 8/29 2024-02-20T00:05:52.733 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : daxctl-libs-71.1-7.el8.x86_64 11/29 2024-02-20T00:05:52.798 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: daxctl-libs-71.1-7.el8.x86_64 11/29 2024-02-20T00:05:52.832 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : librgw2-2:16.2.4-0.el8.x86_64 9/29 2024-02-20T00:05:52.912 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: librgw2-2:16.2.4-0.el8.x86_64 9/29 2024-02-20T00:05:52.955 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 12/29 2024-02-20T00:05:53.095 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/29 2024-02-20T00:05:53.100 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-cephfs-2:16.2.4-0.el8.x86_64 13/29 2024-02-20T00:05:53.273 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : daxctl-libs-71.1-7.el8.x86_64 11/29 2024-02-20T00:05:53.316 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ndctl-libs-71.1-7.el8.x86_64 14/29 2024-02-20T00:05:53.344 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: daxctl-libs-71.1-7.el8.x86_64 11/29 2024-02-20T00:05:53.375 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ndctl-libs-71.1-7.el8.x86_64 14/29 2024-02-20T00:05:53.508 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : libpmemobj-1.6.1-1.el8.x86_64 15/29 2024-02-20T00:05:53.512 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 12/29 2024-02-20T00:05:53.707 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-cephfs-2:16.2.4-0.el8.x86_64 13/29 2024-02-20T00:05:53.899 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ndctl-libs-71.1-7.el8.x86_64 14/29 2024-02-20T00:05:53.947 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: libpmemobj-1.6.1-1.el8.x86_64 15/29 2024-02-20T00:05:53.970 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ndctl-libs-71.1-7.el8.x86_64 14/29 2024-02-20T00:05:54.140 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : libpmemobj-1.6.1-1.el8.x86_64 15/29 2024-02-20T00:05:54.158 INFO:teuthology.orchestra.run.smithi132.stdout: Upgrading : librbd1-2:16.2.4-0.el8.x86_64 16/29 2024-02-20T00:05:54.259 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: librbd1-2:16.2.4-0.el8.x86_64 16/29 2024-02-20T00:05:54.470 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-rbd-2:16.2.4-0.el8.x86_64 17/29 2024-02-20T00:05:54.565 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: libpmemobj-1.6.1-1.el8.x86_64 15/29 2024-02-20T00:05:54.676 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-rgw-2:16.2.4-0.el8.x86_64 18/29 2024-02-20T00:05:54.773 INFO:teuthology.orchestra.run.smithi084.stdout: Upgrading : librbd1-2:16.2.4-0.el8.x86_64 16/29 2024-02-20T00:05:54.842 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2024-02-20T00:05:54.875 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: librbd1-2:16.2.4-0.el8.x86_64 16/29 2024-02-20T00:05:54.918 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2024-02-20T00:05:55.094 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-rbd-2:16.2.4-0.el8.x86_64 17/29 2024-02-20T00:05:55.151 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 20/29 2024-02-20T00:05:55.300 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-rgw-2:16.2.4-0.el8.x86_64 18/29 2024-02-20T00:05:55.367 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 21/29 2024-02-20T00:05:55.440 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2024-02-20T00:05:55.531 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 22/29 2024-02-20T00:05:55.543 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2024-02-20T00:05:55.724 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 20/29 2024-02-20T00:05:55.750 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-ceph-common-2:16.2.4-0.el8.x86_64 23/29 2024-02-20T00:05:55.911 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 21/29 2024-02-20T00:05:56.105 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 22/29 2024-02-20T00:05:56.357 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-ceph-common-2:16.2.4-0.el8.x86_64 23/29 2024-02-20T00:05:58.318 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-02-20T00:05:58.533 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-02-20T00:05:58.905 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-02-20T00:05:59.248 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-02-20T00:05:59.272 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-02-20T00:05:59.518 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-base-2:16.2.4-0.el8.x86_64 25/29 2024-02-20T00:05:59.608 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-base-2:16.2.4-0.el8.x86_64 25/29 2024-02-20T00:05:59.741 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2024-02-20T00:05:59.960 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-02-20T00:06:00.273 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-base-2:16.2.4-0.el8.x86_64 25/29 2024-02-20T00:06:00.369 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-base-2:16.2.4-0.el8.x86_64 25/29 2024-02-20T00:06:00.522 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2024-02-20T00:06:18.949 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2024-02-20T00:06:18.950 INFO:teuthology.orchestra.run.smithi132.stdout:skipping the directory /sys 2024-02-20T00:06:18.950 INFO:teuthology.orchestra.run.smithi132.stdout:skipping the directory /proc 2024-02-20T00:06:18.950 INFO:teuthology.orchestra.run.smithi132.stdout:skipping the directory /dev 2024-02-20T00:06:18.950 INFO:teuthology.orchestra.run.smithi132.stdout:skipping the directory /run 2024-02-20T00:06:18.950 INFO:teuthology.orchestra.run.smithi132.stdout:skipping the directory /mnt 2024-02-20T00:06:18.950 INFO:teuthology.orchestra.run.smithi132.stdout:skipping the directory /var/tmp 2024-02-20T00:06:18.950 INFO:teuthology.orchestra.run.smithi132.stdout:skipping the directory /home 2024-02-20T00:06:18.950 INFO:teuthology.orchestra.run.smithi132.stdout:skipping the directory /tmp 2024-02-20T00:06:18.950 INFO:teuthology.orchestra.run.smithi132.stdout:skipping the directory /dev 2024-02-20T00:06:18.950 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:06:19.754 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2024-02-20T00:06:19.754 INFO:teuthology.orchestra.run.smithi084.stdout:skipping the directory /sys 2024-02-20T00:06:19.754 INFO:teuthology.orchestra.run.smithi084.stdout:skipping the directory /proc 2024-02-20T00:06:19.754 INFO:teuthology.orchestra.run.smithi084.stdout:skipping the directory /dev 2024-02-20T00:06:19.754 INFO:teuthology.orchestra.run.smithi084.stdout:skipping the directory /run 2024-02-20T00:06:19.755 INFO:teuthology.orchestra.run.smithi084.stdout:skipping the directory /mnt 2024-02-20T00:06:19.755 INFO:teuthology.orchestra.run.smithi084.stdout:skipping the directory /var/tmp 2024-02-20T00:06:19.755 INFO:teuthology.orchestra.run.smithi084.stdout:skipping the directory /home 2024-02-20T00:06:19.755 INFO:teuthology.orchestra.run.smithi084.stdout:skipping the directory /tmp 2024-02-20T00:06:19.755 INFO:teuthology.orchestra.run.smithi084.stdout:skipping the directory /dev 2024-02-20T00:06:19.755 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-02-20T00:06:20.273 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2024-02-20T00:06:20.349 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2024-02-20T00:06:20.350 INFO:teuthology.orchestra.run.smithi132.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 28/29 2024-02-20T00:06:20.589 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 28/29 2024-02-20T00:06:20.589 INFO:teuthology.orchestra.run.smithi132.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 29/29 2024-02-20T00:06:21.118 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2024-02-20T00:06:21.191 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2024-02-20T00:06:21.191 INFO:teuthology.orchestra.run.smithi084.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 28/29 2024-02-20T00:06:21.246 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 29/29 2024-02-20T00:06:21.246 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-base-2:16.2.4-0.el8.x86_64 1/29 2024-02-20T00:06:21.246 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-common-2:16.2.4-0.el8.x86_64 2/29 2024-02-20T00:06:21.246 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-radosgw-2:16.2.4-0.el8.x86_64 3/29 2024-02-20T00:06:21.246 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-selinux-2:16.2.4-0.el8.x86_64 4/29 2024-02-20T00:06:21.246 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libcephfs2-2:16.2.4-0.el8.x86_64 5/29 2024-02-20T00:06:21.246 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libradosstriper1-2:16.2.4-0.el8.x86_64 6/29 2024-02-20T00:06:21.246 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : librgw2-2:16.2.4-0.el8.x86_64 7/29 2024-02-20T00:06:21.246 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 8/29 2024-02-20T00:06:21.247 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-ceph-common-2:16.2.4-0.el8.x86_64 9/29 2024-02-20T00:06:21.247 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-cephfs-2:16.2.4-0.el8.x86_64 10/29 2024-02-20T00:06:21.247 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-rados-2:16.2.4-0.el8.x86_64 11/29 2024-02-20T00:06:21.247 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-rbd-2:16.2.4-0.el8.x86_64 12/29 2024-02-20T00:06:21.247 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-rgw-2:16.2.4-0.el8.x86_64 13/29 2024-02-20T00:06:21.247 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libpmemobj-1.6.1-1.el8.x86_64 14/29 2024-02-20T00:06:21.247 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : librdkafka-1.6.1-1.el8.x86_64 15/29 2024-02-20T00:06:21.247 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 16/29 2024-02-20T00:06:21.247 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 17/29 2024-02-20T00:06:21.247 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : daxctl-libs-71.1-7.el8.x86_64 18/29 2024-02-20T00:06:21.247 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : librabbitmq-0.9.0-5.el8.x86_64 19/29 2024-02-20T00:06:21.247 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ndctl-libs-71.1-7.el8.x86_64 20/29 2024-02-20T00:06:21.247 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : fmt-6.2.1-1.el8.x86_64 21/29 2024-02-20T00:06:21.247 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 22/29 2024-02-20T00:06:21.247 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : leveldb-1.22-1.el8.x86_64 23/29 2024-02-20T00:06:21.248 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 24/29 2024-02-20T00:06:21.248 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 25/29 2024-02-20T00:06:21.248 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : librados2-2:16.2.4-0.el8.x86_64 26/29 2024-02-20T00:06:21.248 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 27/29 2024-02-20T00:06:21.248 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : librbd1-2:16.2.4-0.el8.x86_64 28/29 2024-02-20T00:06:21.386 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 28/29 2024-02-20T00:06:21.386 INFO:teuthology.orchestra.run.smithi084.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 29/29 2024-02-20T00:06:22.057 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 29/29 2024-02-20T00:06:22.058 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-base-2:16.2.4-0.el8.x86_64 1/29 2024-02-20T00:06:22.058 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-common-2:16.2.4-0.el8.x86_64 2/29 2024-02-20T00:06:22.058 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-radosgw-2:16.2.4-0.el8.x86_64 3/29 2024-02-20T00:06:22.058 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-selinux-2:16.2.4-0.el8.x86_64 4/29 2024-02-20T00:06:22.058 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : libcephfs2-2:16.2.4-0.el8.x86_64 5/29 2024-02-20T00:06:22.058 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : libradosstriper1-2:16.2.4-0.el8.x86_64 6/29 2024-02-20T00:06:22.058 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : librgw2-2:16.2.4-0.el8.x86_64 7/29 2024-02-20T00:06:22.058 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 8/29 2024-02-20T00:06:22.058 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-ceph-common-2:16.2.4-0.el8.x86_64 9/29 2024-02-20T00:06:22.058 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-cephfs-2:16.2.4-0.el8.x86_64 10/29 2024-02-20T00:06:22.058 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-rados-2:16.2.4-0.el8.x86_64 11/29 2024-02-20T00:06:22.058 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-rbd-2:16.2.4-0.el8.x86_64 12/29 2024-02-20T00:06:22.058 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-rgw-2:16.2.4-0.el8.x86_64 13/29 2024-02-20T00:06:22.058 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : libpmemobj-1.6.1-1.el8.x86_64 14/29 2024-02-20T00:06:22.059 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : librdkafka-1.6.1-1.el8.x86_64 15/29 2024-02-20T00:06:22.059 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 16/29 2024-02-20T00:06:22.059 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 17/29 2024-02-20T00:06:22.059 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : daxctl-libs-71.1-7.el8.x86_64 18/29 2024-02-20T00:06:22.059 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : librabbitmq-0.9.0-5.el8.x86_64 19/29 2024-02-20T00:06:22.059 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ndctl-libs-71.1-7.el8.x86_64 20/29 2024-02-20T00:06:22.059 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : fmt-6.2.1-1.el8.x86_64 21/29 2024-02-20T00:06:22.059 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 22/29 2024-02-20T00:06:22.059 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : leveldb-1.22-1.el8.x86_64 23/29 2024-02-20T00:06:22.059 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 24/29 2024-02-20T00:06:22.059 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 25/29 2024-02-20T00:06:22.059 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : librados2-2:16.2.4-0.el8.x86_64 26/29 2024-02-20T00:06:22.059 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 27/29 2024-02-20T00:06:22.059 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : librbd1-2:16.2.4-0.el8.x86_64 28/29 2024-02-20T00:06:22.151 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 29/29 2024-02-20T00:06:22.152 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:06:22.152 INFO:teuthology.orchestra.run.smithi132.stdout:Upgraded: 2024-02-20T00:06:22.152 INFO:teuthology.orchestra.run.smithi132.stdout: librados2-2:16.2.4-0.el8.x86_64 librbd1-2:16.2.4-0.el8.x86_64 2024-02-20T00:06:22.152 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2024-02-20T00:06:22.152 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-base-2:16.2.4-0.el8.x86_64 2024-02-20T00:06:22.152 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-common-2:16.2.4-0.el8.x86_64 2024-02-20T00:06:22.152 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-radosgw-2:16.2.4-0.el8.x86_64 2024-02-20T00:06:22.152 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-selinux-2:16.2.4-0.el8.x86_64 2024-02-20T00:06:22.152 INFO:teuthology.orchestra.run.smithi132.stdout: daxctl-libs-71.1-7.el8.x86_64 2024-02-20T00:06:22.152 INFO:teuthology.orchestra.run.smithi132.stdout: fmt-6.2.1-1.el8.x86_64 2024-02-20T00:06:22.152 INFO:teuthology.orchestra.run.smithi132.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-02-20T00:06:22.153 INFO:teuthology.orchestra.run.smithi132.stdout: leveldb-1.22-1.el8.x86_64 2024-02-20T00:06:22.153 INFO:teuthology.orchestra.run.smithi132.stdout: libcephfs2-2:16.2.4-0.el8.x86_64 2024-02-20T00:06:22.153 INFO:teuthology.orchestra.run.smithi132.stdout: liboath-2.6.2-3.el8.x86_64 2024-02-20T00:06:22.153 INFO:teuthology.orchestra.run.smithi132.stdout: libpmemobj-1.6.1-1.el8.x86_64 2024-02-20T00:06:22.153 INFO:teuthology.orchestra.run.smithi132.stdout: librabbitmq-0.9.0-5.el8.x86_64 2024-02-20T00:06:22.153 INFO:teuthology.orchestra.run.smithi132.stdout: libradosstriper1-2:16.2.4-0.el8.x86_64 2024-02-20T00:06:22.153 INFO:teuthology.orchestra.run.smithi132.stdout: librdkafka-1.6.1-1.el8.x86_64 2024-02-20T00:06:22.153 INFO:teuthology.orchestra.run.smithi132.stdout: librgw2-2:16.2.4-0.el8.x86_64 2024-02-20T00:06:22.153 INFO:teuthology.orchestra.run.smithi132.stdout: libunwind-1.3.1-3.el8.x86_64 2024-02-20T00:06:22.153 INFO:teuthology.orchestra.run.smithi132.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-02-20T00:06:22.153 INFO:teuthology.orchestra.run.smithi132.stdout: ndctl-libs-71.1-7.el8.x86_64 2024-02-20T00:06:22.153 INFO:teuthology.orchestra.run.smithi132.stdout: python3-ceph-argparse-2:16.2.4-0.el8.x86_64 2024-02-20T00:06:22.153 INFO:teuthology.orchestra.run.smithi132.stdout: python3-ceph-common-2:16.2.4-0.el8.x86_64 2024-02-20T00:06:22.153 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cephfs-2:16.2.4-0.el8.x86_64 2024-02-20T00:06:22.154 INFO:teuthology.orchestra.run.smithi132.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-02-20T00:06:22.154 INFO:teuthology.orchestra.run.smithi132.stdout: python3-rados-2:16.2.4-0.el8.x86_64 2024-02-20T00:06:22.154 INFO:teuthology.orchestra.run.smithi132.stdout: python3-rbd-2:16.2.4-0.el8.x86_64 2024-02-20T00:06:22.154 INFO:teuthology.orchestra.run.smithi132.stdout: python3-rgw-2:16.2.4-0.el8.x86_64 2024-02-20T00:06:22.154 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:06:22.154 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-02-20T00:06:22.436 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install ceph-test 2024-02-20T00:06:22.966 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 29/29 2024-02-20T00:06:22.967 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-02-20T00:06:22.967 INFO:teuthology.orchestra.run.smithi084.stdout:Upgraded: 2024-02-20T00:06:22.967 INFO:teuthology.orchestra.run.smithi084.stdout: librados2-2:16.2.4-0.el8.x86_64 librbd1-2:16.2.4-0.el8.x86_64 2024-02-20T00:06:22.967 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-02-20T00:06:22.967 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-base-2:16.2.4-0.el8.x86_64 2024-02-20T00:06:22.967 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-common-2:16.2.4-0.el8.x86_64 2024-02-20T00:06:22.967 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-radosgw-2:16.2.4-0.el8.x86_64 2024-02-20T00:06:22.967 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-selinux-2:16.2.4-0.el8.x86_64 2024-02-20T00:06:22.967 INFO:teuthology.orchestra.run.smithi084.stdout: daxctl-libs-71.1-7.el8.x86_64 2024-02-20T00:06:22.967 INFO:teuthology.orchestra.run.smithi084.stdout: fmt-6.2.1-1.el8.x86_64 2024-02-20T00:06:22.968 INFO:teuthology.orchestra.run.smithi084.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-02-20T00:06:22.968 INFO:teuthology.orchestra.run.smithi084.stdout: leveldb-1.22-1.el8.x86_64 2024-02-20T00:06:22.968 INFO:teuthology.orchestra.run.smithi084.stdout: libcephfs2-2:16.2.4-0.el8.x86_64 2024-02-20T00:06:22.968 INFO:teuthology.orchestra.run.smithi084.stdout: liboath-2.6.2-3.el8.x86_64 2024-02-20T00:06:22.968 INFO:teuthology.orchestra.run.smithi084.stdout: libpmemobj-1.6.1-1.el8.x86_64 2024-02-20T00:06:22.968 INFO:teuthology.orchestra.run.smithi084.stdout: librabbitmq-0.9.0-5.el8.x86_64 2024-02-20T00:06:22.968 INFO:teuthology.orchestra.run.smithi084.stdout: libradosstriper1-2:16.2.4-0.el8.x86_64 2024-02-20T00:06:22.968 INFO:teuthology.orchestra.run.smithi084.stdout: librdkafka-1.6.1-1.el8.x86_64 2024-02-20T00:06:22.968 INFO:teuthology.orchestra.run.smithi084.stdout: librgw2-2:16.2.4-0.el8.x86_64 2024-02-20T00:06:22.968 INFO:teuthology.orchestra.run.smithi084.stdout: libunwind-1.3.1-3.el8.x86_64 2024-02-20T00:06:22.968 INFO:teuthology.orchestra.run.smithi084.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-02-20T00:06:22.968 INFO:teuthology.orchestra.run.smithi084.stdout: ndctl-libs-71.1-7.el8.x86_64 2024-02-20T00:06:22.968 INFO:teuthology.orchestra.run.smithi084.stdout: python3-ceph-argparse-2:16.2.4-0.el8.x86_64 2024-02-20T00:06:22.969 INFO:teuthology.orchestra.run.smithi084.stdout: python3-ceph-common-2:16.2.4-0.el8.x86_64 2024-02-20T00:06:22.969 INFO:teuthology.orchestra.run.smithi084.stdout: python3-cephfs-2:16.2.4-0.el8.x86_64 2024-02-20T00:06:22.969 INFO:teuthology.orchestra.run.smithi084.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-02-20T00:06:22.969 INFO:teuthology.orchestra.run.smithi084.stdout: python3-rados-2:16.2.4-0.el8.x86_64 2024-02-20T00:06:22.969 INFO:teuthology.orchestra.run.smithi084.stdout: python3-rbd-2:16.2.4-0.el8.x86_64 2024-02-20T00:06:22.969 INFO:teuthology.orchestra.run.smithi084.stdout: python3-rgw-2:16.2.4-0.el8.x86_64 2024-02-20T00:06:22.969 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-02-20T00:06:22.969 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-02-20T00:06:23.200 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:00:42 ago on Tue 20 Feb 2024 12:05:41 AM UTC. 2024-02-20T00:06:23.281 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install ceph-test 2024-02-20T00:06:24.035 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:00:43 ago on Tue 20 Feb 2024 12:05:41 AM UTC. 2024-02-20T00:06:24.756 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-02-20T00:06:24.757 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-02-20T00:06:24.757 INFO:teuthology.orchestra.run.smithi132.stdout: Package Architecture Version Repository Size 2024-02-20T00:06:24.758 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-02-20T00:06:24.758 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2024-02-20T00:06:24.758 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-test x86_64 2:16.2.4-0.el8 ceph 52 M 2024-02-20T00:06:24.758 INFO:teuthology.orchestra.run.smithi132.stdout:Installing dependencies: 2024-02-20T00:06:24.758 INFO:teuthology.orchestra.run.smithi132.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2024-02-20T00:06:24.758 INFO:teuthology.orchestra.run.smithi132.stdout: libcephsqlite x86_64 2:16.2.4-0.el8 ceph 169 k 2024-02-20T00:06:24.758 INFO:teuthology.orchestra.run.smithi132.stdout: oniguruma x86_64 6.8.2-3.el8 appstream 188 k 2024-02-20T00:06:24.758 INFO:teuthology.orchestra.run.smithi132.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2024-02-20T00:06:24.758 INFO:teuthology.orchestra.run.smithi132.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2024-02-20T00:06:24.758 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:06:24.758 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2024-02-20T00:06:24.758 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-02-20T00:06:24.758 INFO:teuthology.orchestra.run.smithi132.stdout:Install 6 Packages 2024-02-20T00:06:24.759 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:06:24.759 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 53 M 2024-02-20T00:06:24.759 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 229 M 2024-02-20T00:06:24.759 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2024-02-20T00:06:25.008 INFO:teuthology.orchestra.run.smithi132.stdout:(1/6): libcephsqlite-16.2.4-0.el8.x86_64.rpm 2.9 MB/s | 169 kB 00:00 2024-02-20T00:06:25.050 INFO:teuthology.orchestra.run.smithi132.stdout:(2/6): jq-1.6-8.el8.x86_64.rpm 2.0 MB/s | 203 kB 00:00 2024-02-20T00:06:25.100 INFO:teuthology.orchestra.run.smithi132.stdout:(3/6): socat-1.7.4.1-1.el8.x86_64.rpm 6.3 MB/s | 323 kB 00:00 2024-02-20T00:06:25.125 INFO:teuthology.orchestra.run.smithi132.stdout:(4/6): oniguruma-6.8.2-3.el8.x86_64.rpm 1.6 MB/s | 188 kB 00:00 2024-02-20T00:06:25.151 INFO:teuthology.orchestra.run.smithi132.stdout:(5/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 1.4 MB/s | 71 kB 00:00 2024-02-20T00:06:25.587 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-02-20T00:06:25.588 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-02-20T00:06:25.588 INFO:teuthology.orchestra.run.smithi084.stdout: Package Architecture Version Repository Size 2024-02-20T00:06:25.588 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-02-20T00:06:25.589 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-02-20T00:06:25.589 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-test x86_64 2:16.2.4-0.el8 ceph 52 M 2024-02-20T00:06:25.589 INFO:teuthology.orchestra.run.smithi084.stdout:Installing dependencies: 2024-02-20T00:06:25.589 INFO:teuthology.orchestra.run.smithi084.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2024-02-20T00:06:25.589 INFO:teuthology.orchestra.run.smithi084.stdout: libcephsqlite x86_64 2:16.2.4-0.el8 ceph 169 k 2024-02-20T00:06:25.589 INFO:teuthology.orchestra.run.smithi084.stdout: oniguruma x86_64 6.8.2-3.el8 appstream 188 k 2024-02-20T00:06:25.589 INFO:teuthology.orchestra.run.smithi084.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2024-02-20T00:06:25.589 INFO:teuthology.orchestra.run.smithi084.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2024-02-20T00:06:25.589 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-02-20T00:06:25.589 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-02-20T00:06:25.589 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-02-20T00:06:25.589 INFO:teuthology.orchestra.run.smithi084.stdout:Install 6 Packages 2024-02-20T00:06:25.589 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-02-20T00:06:25.590 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 53 M 2024-02-20T00:06:25.590 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 229 M 2024-02-20T00:06:25.590 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-02-20T00:06:25.734 INFO:teuthology.orchestra.run.smithi132.stdout:(6/6): ceph-test-16.2.4-0.el8.x86_64.rpm 66 MB/s | 52 MB 00:00 2024-02-20T00:06:25.736 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2024-02-20T00:06:25.737 INFO:teuthology.orchestra.run.smithi132.stdout:Total 54 MB/s | 53 MB 00:00 2024-02-20T00:06:25.779 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2024-02-20T00:06:25.800 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2024-02-20T00:06:25.800 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2024-02-20T00:06:25.839 INFO:teuthology.orchestra.run.smithi084.stdout:(1/6): libcephsqlite-16.2.4-0.el8.x86_64.rpm 2.6 MB/s | 169 kB 00:00 2024-02-20T00:06:25.873 INFO:teuthology.orchestra.run.smithi084.stdout:(2/6): jq-1.6-8.el8.x86_64.rpm 2.0 MB/s | 203 kB 00:00 2024-02-20T00:06:25.965 INFO:teuthology.orchestra.run.smithi084.stdout:(3/6): oniguruma-6.8.2-3.el8.x86_64.rpm 1.5 MB/s | 188 kB 00:00 2024-02-20T00:06:26.024 INFO:teuthology.orchestra.run.smithi084.stdout:(4/6): socat-1.7.4.1-1.el8.x86_64.rpm 2.1 MB/s | 323 kB 00:00 2024-02-20T00:06:26.049 INFO:teuthology.orchestra.run.smithi084.stdout:(5/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 846 kB/s | 71 kB 00:00 2024-02-20T00:06:26.092 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2024-02-20T00:06:26.096 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2024-02-20T00:06:26.549 INFO:teuthology.orchestra.run.smithi084.stdout:(6/6): ceph-test-16.2.4-0.el8.x86_64.rpm 67 MB/s | 52 MB 00:00 2024-02-20T00:06:26.551 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-02-20T00:06:26.552 INFO:teuthology.orchestra.run.smithi084.stdout:Total 55 MB/s | 53 MB 00:00 2024-02-20T00:06:26.577 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2024-02-20T00:06:26.594 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-02-20T00:06:26.614 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-02-20T00:06:26.615 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-02-20T00:06:26.789 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2024-02-20T00:06:26.907 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-02-20T00:06:26.910 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-02-20T00:06:26.990 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2024-02-20T00:06:27.128 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : oniguruma-6.8.2-3.el8.x86_64 3/6 2024-02-20T00:06:27.206 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: oniguruma-6.8.2-3.el8.x86_64 3/6 2024-02-20T00:06:27.377 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2024-02-20T00:06:27.410 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-02-20T00:06:27.553 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2024-02-20T00:06:27.619 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2024-02-20T00:06:27.816 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2024-02-20T00:06:27.983 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : oniguruma-6.8.2-3.el8.x86_64 3/6 2024-02-20T00:06:28.062 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: oniguruma-6.8.2-3.el8.x86_64 3/6 2024-02-20T00:06:28.232 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2024-02-20T00:06:28.382 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2024-02-20T00:06:32.453 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2024-02-20T00:06:32.676 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-test-2:16.2.4-0.el8.x86_64 6/6 2024-02-20T00:06:32.979 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-test-2:16.2.4-0.el8.x86_64 6/6 2024-02-20T00:06:32.980 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-test-2:16.2.4-0.el8.x86_64 1/6 2024-02-20T00:06:32.980 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libcephsqlite-2:16.2.4-0.el8.x86_64 2/6 2024-02-20T00:06:32.980 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2024-02-20T00:06:32.980 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : oniguruma-6.8.2-3.el8.x86_64 4/6 2024-02-20T00:06:32.980 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2024-02-20T00:06:33.239 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2024-02-20T00:06:33.461 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2024-02-20T00:06:33.461 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:06:33.461 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2024-02-20T00:06:33.461 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-test-2:16.2.4-0.el8.x86_64 jq-1.6-8.el8.x86_64 2024-02-20T00:06:33.461 INFO:teuthology.orchestra.run.smithi132.stdout: libcephsqlite-2:16.2.4-0.el8.x86_64 oniguruma-6.8.2-3.el8.x86_64 2024-02-20T00:06:33.461 INFO:teuthology.orchestra.run.smithi132.stdout: socat-1.7.4.1-1.el8.x86_64 xmlstarlet-1.6.1-20.el8.x86_64 2024-02-20T00:06:33.461 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:06:33.461 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-02-20T00:06:33.480 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-test-2:16.2.4-0.el8.x86_64 6/6 2024-02-20T00:06:33.675 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install ceph 2024-02-20T00:06:33.765 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-test-2:16.2.4-0.el8.x86_64 6/6 2024-02-20T00:06:33.765 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-test-2:16.2.4-0.el8.x86_64 1/6 2024-02-20T00:06:33.766 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : libcephsqlite-2:16.2.4-0.el8.x86_64 2/6 2024-02-20T00:06:33.766 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2024-02-20T00:06:33.766 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : oniguruma-6.8.2-3.el8.x86_64 4/6 2024-02-20T00:06:33.766 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2024-02-20T00:06:34.289 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2024-02-20T00:06:34.290 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-02-20T00:06:34.290 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-02-20T00:06:34.290 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-test-2:16.2.4-0.el8.x86_64 jq-1.6-8.el8.x86_64 2024-02-20T00:06:34.290 INFO:teuthology.orchestra.run.smithi084.stdout: libcephsqlite-2:16.2.4-0.el8.x86_64 oniguruma-6.8.2-3.el8.x86_64 2024-02-20T00:06:34.290 INFO:teuthology.orchestra.run.smithi084.stdout: socat-1.7.4.1-1.el8.x86_64 xmlstarlet-1.6.1-20.el8.x86_64 2024-02-20T00:06:34.290 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-02-20T00:06:34.290 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-02-20T00:06:34.445 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:00:53 ago on Tue 20 Feb 2024 12:05:41 AM UTC. 2024-02-20T00:06:34.634 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install ceph 2024-02-20T00:06:35.394 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:00:54 ago on Tue 20 Feb 2024 12:05:41 AM UTC. 2024-02-20T00:06:36.005 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-02-20T00:06:36.008 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-02-20T00:06:36.009 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repository Size 2024-02-20T00:06:36.009 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-02-20T00:06:36.009 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2024-02-20T00:06:36.009 INFO:teuthology.orchestra.run.smithi132.stdout: ceph x86_64 2:16.2.4-0.el8 ceph 6.3 k 2024-02-20T00:06:36.009 INFO:teuthology.orchestra.run.smithi132.stdout:Installing dependencies: 2024-02-20T00:06:36.009 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mds x86_64 2:16.2.4-0.el8 ceph 2.4 M 2024-02-20T00:06:36.009 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr x86_64 2:16.2.4-0.el8 ceph 1.6 M 2024-02-20T00:06:36.009 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-modules-core noarch 2:16.2.4-0.el8 ceph-noarch 220 k 2024-02-20T00:06:36.009 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mon x86_64 2:16.2.4-0.el8 ceph 5.0 M 2024-02-20T00:06:36.009 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-osd x86_64 2:16.2.4-0.el8 ceph 18 M 2024-02-20T00:06:36.009 INFO:teuthology.orchestra.run.smithi132.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-02-20T00:06:36.009 INFO:teuthology.orchestra.run.smithi132.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-02-20T00:06:36.009 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2024-02-20T00:06:36.009 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-02-20T00:06:36.010 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-02-20T00:06:36.010 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cryptography x86_64 3.2.1-7.el8 baseos 641 k 2024-02-20T00:06:36.010 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-02-20T00:06:36.010 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-02-20T00:06:36.010 INFO:teuthology.orchestra.run.smithi132.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-02-20T00:06:36.010 INFO:teuthology.orchestra.run.smithi132.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2024-02-20T00:06:36.010 INFO:teuthology.orchestra.run.smithi132.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2024-02-20T00:06:36.010 INFO:teuthology.orchestra.run.smithi132.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-02-20T00:06:36.010 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-02-20T00:06:36.010 INFO:teuthology.orchestra.run.smithi132.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-02-20T00:06:36.010 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2024-02-20T00:06:36.010 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2024-02-20T00:06:36.010 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2024-02-20T00:06:36.010 INFO:teuthology.orchestra.run.smithi132.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-02-20T00:06:36.010 INFO:teuthology.orchestra.run.smithi132.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-02-20T00:06:36.011 INFO:teuthology.orchestra.run.smithi132.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-02-20T00:06:36.011 INFO:teuthology.orchestra.run.smithi132.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-02-20T00:06:36.011 INFO:teuthology.orchestra.run.smithi132.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-02-20T00:06:36.011 INFO:teuthology.orchestra.run.smithi132.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-02-20T00:06:36.011 INFO:teuthology.orchestra.run.smithi132.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-02-20T00:06:36.011 INFO:teuthology.orchestra.run.smithi132.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2024-02-20T00:06:36.011 INFO:teuthology.orchestra.run.smithi132.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-02-20T00:06:36.011 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:06:36.011 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2024-02-20T00:06:36.011 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-02-20T00:06:36.011 INFO:teuthology.orchestra.run.smithi132.stdout:Install 32 Packages 2024-02-20T00:06:36.011 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:06:36.012 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 31 M 2024-02-20T00:06:36.012 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 111 M 2024-02-20T00:06:36.012 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2024-02-20T00:06:36.486 INFO:teuthology.orchestra.run.smithi132.stdout:(1/32): ceph-16.2.4-0.el8.x86_64.rpm 103 kB/s | 6.3 kB 00:00 2024-02-20T00:06:36.553 INFO:teuthology.orchestra.run.smithi132.stdout:(2/32): ceph-mds-16.2.4-0.el8.x86_64.rpm 19 MB/s | 2.4 MB 00:00 2024-02-20T00:06:36.595 INFO:teuthology.orchestra.run.smithi132.stdout:(3/32): ceph-mgr-16.2.4-0.el8.x86_64.rpm 9.2 MB/s | 1.6 MB 00:00 2024-02-20T00:06:36.629 INFO:teuthology.orchestra.run.smithi132.stdout:(4/32): ceph-mgr-modules-core-16.2.4-0.el8.noar 6.5 MB/s | 220 kB 00:00 2024-02-20T00:06:36.712 INFO:teuthology.orchestra.run.smithi132.stdout:(5/32): ceph-mon-16.2.4-0.el8.x86_64.rpm 22 MB/s | 5.0 MB 00:00 2024-02-20T00:06:36.747 INFO:teuthology.orchestra.run.smithi132.stdout:(6/32): python3-mako-1.0.6-14.el8.noarch.rpm 1.3 MB/s | 157 kB 00:00 2024-02-20T00:06:36.813 INFO:teuthology.orchestra.run.smithi132.stdout:(7/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 1.5 MB/s | 103 kB 00:00 2024-02-20T00:06:36.830 INFO:teuthology.orchestra.run.smithi132.stdout:(8/32): python3-markupsafe-0.23-19.el8.x86_64.r 331 kB/s | 39 kB 00:00 2024-02-20T00:06:36.856 INFO:teuthology.orchestra.run.smithi132.stdout:(9/32): python3-pytz-2017.2-11.el8.noarch.rpm 1.3 MB/s | 57 kB 00:00 2024-02-20T00:06:36.952 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-02-20T00:06:36.954 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-02-20T00:06:36.954 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repository Size 2024-02-20T00:06:36.954 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-02-20T00:06:36.955 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-02-20T00:06:36.955 INFO:teuthology.orchestra.run.smithi084.stdout: ceph x86_64 2:16.2.4-0.el8 ceph 6.3 k 2024-02-20T00:06:36.955 INFO:teuthology.orchestra.run.smithi084.stdout:Installing dependencies: 2024-02-20T00:06:36.955 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mds x86_64 2:16.2.4-0.el8 ceph 2.4 M 2024-02-20T00:06:36.955 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mgr x86_64 2:16.2.4-0.el8 ceph 1.6 M 2024-02-20T00:06:36.955 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mgr-modules-core noarch 2:16.2.4-0.el8 ceph-noarch 220 k 2024-02-20T00:06:36.955 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mon x86_64 2:16.2.4-0.el8 ceph 5.0 M 2024-02-20T00:06:36.955 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-osd x86_64 2:16.2.4-0.el8 ceph 18 M 2024-02-20T00:06:36.955 INFO:teuthology.orchestra.run.smithi084.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-02-20T00:06:36.955 INFO:teuthology.orchestra.run.smithi084.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-02-20T00:06:36.955 INFO:teuthology.orchestra.run.smithi084.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2024-02-20T00:06:36.955 INFO:teuthology.orchestra.run.smithi084.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-02-20T00:06:36.955 INFO:teuthology.orchestra.run.smithi084.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-02-20T00:06:36.955 INFO:teuthology.orchestra.run.smithi084.stdout: python3-cryptography x86_64 3.2.1-7.el8 baseos 641 k 2024-02-20T00:06:36.956 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-02-20T00:06:36.956 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-02-20T00:06:36.956 INFO:teuthology.orchestra.run.smithi084.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-02-20T00:06:36.956 INFO:teuthology.orchestra.run.smithi084.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2024-02-20T00:06:36.956 INFO:teuthology.orchestra.run.smithi084.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2024-02-20T00:06:36.956 INFO:teuthology.orchestra.run.smithi084.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-02-20T00:06:36.956 INFO:teuthology.orchestra.run.smithi084.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-02-20T00:06:36.956 INFO:teuthology.orchestra.run.smithi084.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-02-20T00:06:36.956 INFO:teuthology.orchestra.run.smithi084.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2024-02-20T00:06:36.956 INFO:teuthology.orchestra.run.smithi084.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2024-02-20T00:06:36.956 INFO:teuthology.orchestra.run.smithi084.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2024-02-20T00:06:36.956 INFO:teuthology.orchestra.run.smithi084.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-02-20T00:06:36.956 INFO:teuthology.orchestra.run.smithi084.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-02-20T00:06:36.956 INFO:teuthology.orchestra.run.smithi084.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-02-20T00:06:36.957 INFO:teuthology.orchestra.run.smithi084.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-02-20T00:06:36.957 INFO:teuthology.orchestra.run.smithi084.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-02-20T00:06:36.957 INFO:teuthology.orchestra.run.smithi084.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-02-20T00:06:36.957 INFO:teuthology.orchestra.run.smithi084.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-02-20T00:06:36.957 INFO:teuthology.orchestra.run.smithi084.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2024-02-20T00:06:36.957 INFO:teuthology.orchestra.run.smithi084.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-02-20T00:06:36.957 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-02-20T00:06:36.957 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-02-20T00:06:36.957 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-02-20T00:06:36.957 INFO:teuthology.orchestra.run.smithi084.stdout:Install 32 Packages 2024-02-20T00:06:36.957 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-02-20T00:06:36.958 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 31 M 2024-02-20T00:06:36.958 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 111 M 2024-02-20T00:06:36.958 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-02-20T00:06:37.081 INFO:teuthology.orchestra.run.smithi132.stdout:(10/32): ceph-osd-16.2.4-0.el8.x86_64.rpm 34 MB/s | 18 MB 00:00 2024-02-20T00:06:37.106 INFO:teuthology.orchestra.run.smithi132.stdout:(11/32): python3-werkzeug-0.12.2-4.el8.noarch.r 1.6 MB/s | 457 kB 00:00 2024-02-20T00:06:37.346 INFO:teuthology.orchestra.run.smithi084.stdout:(1/32): ceph-16.2.4-0.el8.x86_64.rpm 88 kB/s | 6.3 kB 00:00 2024-02-20T00:06:37.408 INFO:teuthology.orchestra.run.smithi084.stdout:(2/32): ceph-mgr-16.2.4-0.el8.x86_64.rpm 12 MB/s | 1.6 MB 00:00 2024-02-20T00:06:37.466 INFO:teuthology.orchestra.run.smithi084.stdout:(3/32): ceph-mds-16.2.4-0.el8.x86_64.rpm 12 MB/s | 2.4 MB 00:00 2024-02-20T00:06:37.491 INFO:teuthology.orchestra.run.smithi084.stdout:(4/32): ceph-mgr-modules-core-16.2.4-0.el8.noar 8.6 MB/s | 220 kB 00:00 2024-02-20T00:06:37.498 INFO:teuthology.orchestra.run.smithi132.stdout:(12/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 387 kB/s | 249 kB 00:00 2024-02-20T00:06:37.515 INFO:teuthology.orchestra.run.smithi132.stdout:(13/32): python3-pycparser-2.14-14.el8.noarch.r 267 kB/s | 109 kB 00:00 2024-02-20T00:06:37.600 INFO:teuthology.orchestra.run.smithi084.stdout:(5/32): ceph-mon-16.2.4-0.el8.x86_64.rpm 20 MB/s | 5.0 MB 00:00 2024-02-20T00:06:37.625 INFO:teuthology.orchestra.run.smithi084.stdout:(6/32): python3-mako-1.0.6-14.el8.noarch.rpm 1.1 MB/s | 157 kB 00:00 2024-02-20T00:06:37.676 INFO:teuthology.orchestra.run.smithi084.stdout:(7/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 2.0 MB/s | 103 kB 00:00 2024-02-20T00:06:37.682 INFO:teuthology.orchestra.run.smithi132.stdout:(14/32): python3-cryptography-3.2.1-7.el8.x86_6 1.0 MB/s | 641 kB 00:00 2024-02-20T00:06:37.701 INFO:teuthology.orchestra.run.smithi084.stdout:(8/32): python3-markupsafe-0.23-19.el8.x86_64.r 386 kB/s | 39 kB 00:00 2024-02-20T00:06:37.718 INFO:teuthology.orchestra.run.smithi084.stdout:(9/32): python3-pytz-2017.2-11.el8.noarch.rpm 1.3 MB/s | 57 kB 00:00 2024-02-20T00:06:37.765 INFO:teuthology.orchestra.run.smithi132.stdout:(15/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 163 kB/s | 44 kB 00:00 2024-02-20T00:06:37.874 INFO:teuthology.orchestra.run.smithi132.stdout:(16/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 516 kB/s | 185 kB 00:00 2024-02-20T00:06:37.952 INFO:teuthology.orchestra.run.smithi084.stdout:(10/32): ceph-osd-16.2.4-0.el8.x86_64.rpm 33 MB/s | 18 MB 00:00 2024-02-20T00:06:37.966 INFO:teuthology.orchestra.run.smithi132.stdout:(17/32): python3-jaraco-6.2-6.el8.noarch.rpm 121 kB/s | 11 kB 00:00 2024-02-20T00:06:37.977 INFO:teuthology.orchestra.run.smithi084.stdout:(11/32): python3-werkzeug-0.12.2-4.el8.noarch.r 1.6 MB/s | 457 kB 00:00 2024-02-20T00:06:38.008 INFO:teuthology.orchestra.run.smithi132.stdout:(18/32): python3-cherrypy-18.4.0-1.el8.noarch.r 1.6 MB/s | 384 kB 00:00 2024-02-20T00:06:38.033 INFO:teuthology.orchestra.run.smithi132.stdout:(19/32): python3-jaraco-functools-2.0-4.el8.noa 262 kB/s | 18 kB 00:00 2024-02-20T00:06:38.059 INFO:teuthology.orchestra.run.smithi132.stdout:(20/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 459 kB/s | 173 kB 00:00 2024-02-20T00:06:38.084 INFO:teuthology.orchestra.run.smithi132.stdout:(21/32): python3-logutils-0.3.5-11.el8.noarch.r 647 kB/s | 49 kB 00:00 2024-02-20T00:06:38.117 INFO:teuthology.orchestra.run.smithi132.stdout:(22/32): python3-more-itertools-7.2.0-3.el8.noa 701 kB/s | 59 kB 00:00 2024-02-20T00:06:38.159 INFO:teuthology.orchestra.run.smithi132.stdout:(23/32): python3-portend-2.6-1.el8.noarch.rpm 207 kB/s | 16 kB 00:00 2024-02-20T00:06:38.202 INFO:teuthology.orchestra.run.smithi132.stdout:(24/32): python3-simplegeneric-0.8.1-17.el8.noa 223 kB/s | 19 kB 00:00 2024-02-20T00:06:38.227 INFO:teuthology.orchestra.run.smithi132.stdout:(25/32): python3-pecan-1.3.2-9.el8.noarch.rpm 1.6 MB/s | 283 kB 00:00 2024-02-20T00:06:38.252 INFO:teuthology.orchestra.run.smithi132.stdout:(26/32): python3-singledispatch-3.4.0.3-18.el8. 258 kB/s | 24 kB 00:00 2024-02-20T00:06:38.286 INFO:teuthology.orchestra.run.smithi132.stdout:(27/32): python3-tempora-1.14.1-5.el8.noarch.rp 348 kB/s | 29 kB 00:00 2024-02-20T00:06:38.311 INFO:teuthology.orchestra.run.smithi132.stdout:(28/32): python3-trustme-0.6.0-4.el8.noarch.rpm 328 kB/s | 27 kB 00:00 2024-02-20T00:06:38.345 INFO:teuthology.orchestra.run.smithi132.stdout:(29/32): python3-waitress-1.4.3-1.el8.noarch.rp 2.7 MB/s | 255 kB 00:00 2024-02-20T00:06:38.377 INFO:teuthology.orchestra.run.smithi084.stdout:(12/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 377 kB/s | 249 kB 00:00 2024-02-20T00:06:38.395 INFO:teuthology.orchestra.run.smithi132.stdout:(30/32): python3-webtest-2.0.33-1.el8.noarch.rp 1.0 MB/s | 85 kB 00:00 2024-02-20T00:06:38.402 INFO:teuthology.orchestra.run.smithi084.stdout:(13/32): python3-pycparser-2.14-14.el8.noarch.r 256 kB/s | 109 kB 00:00 2024-02-20T00:06:38.420 INFO:teuthology.orchestra.run.smithi132.stdout:(31/32): python3-zc-lockfile-2.0-2.el8.noarch.r 305 kB/s | 23 kB 00:00 2024-02-20T00:06:38.445 INFO:teuthology.orchestra.run.smithi132.stdout:(32/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 1.5 MB/s | 251 kB 00:00 2024-02-20T00:06:38.449 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2024-02-20T00:06:38.450 INFO:teuthology.orchestra.run.smithi132.stdout:Total 13 MB/s | 31 MB 00:02 2024-02-20T00:06:38.531 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2024-02-20T00:06:38.570 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2024-02-20T00:06:38.570 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2024-02-20T00:06:38.578 INFO:teuthology.orchestra.run.smithi084.stdout:(14/32): python3-cryptography-3.2.1-7.el8.x86_6 1.0 MB/s | 641 kB 00:00 2024-02-20T00:06:38.797 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2024-02-20T00:06:38.800 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2024-02-20T00:06:38.820 INFO:teuthology.orchestra.run.smithi084.stdout:(15/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 99 kB/s | 44 kB 00:00 2024-02-20T00:06:38.979 INFO:teuthology.orchestra.run.smithi084.stdout:(16/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 322 kB/s | 185 kB 00:00 2024-02-20T00:06:39.087 INFO:teuthology.orchestra.run.smithi084.stdout:(17/32): python3-jaraco-6.2-6.el8.noarch.rpm 103 kB/s | 11 kB 00:00 2024-02-20T00:06:39.154 INFO:teuthology.orchestra.run.smithi084.stdout:(18/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 300 kB/s | 173 kB 00:00 2024-02-20T00:06:39.196 INFO:teuthology.orchestra.run.smithi084.stdout:(19/32): python3-jaraco-functools-2.0-4.el8.noa 162 kB/s | 18 kB 00:00 2024-02-20T00:06:39.221 INFO:teuthology.orchestra.run.smithi084.stdout:(20/32): python3-cherrypy-18.4.0-1.el8.noarch.r 959 kB/s | 384 kB 00:00 2024-02-20T00:06:39.272 INFO:teuthology.orchestra.run.smithi084.stdout:(21/32): python3-logutils-0.3.5-11.el8.noarch.r 416 kB/s | 49 kB 00:00 2024-02-20T00:06:39.322 INFO:teuthology.orchestra.run.smithi084.stdout:(22/32): python3-more-itertools-7.2.0-3.el8.noa 468 kB/s | 59 kB 00:00 2024-02-20T00:06:39.372 INFO:teuthology.orchestra.run.smithi084.stdout:(23/32): python3-pecan-1.3.2-9.el8.noarch.rpm 1.8 MB/s | 283 kB 00:00 2024-02-20T00:06:39.397 INFO:teuthology.orchestra.run.smithi084.stdout:(24/32): python3-portend-2.6-1.el8.noarch.rpm 124 kB/s | 16 kB 00:00 2024-02-20T00:06:39.431 INFO:teuthology.orchestra.run.smithi084.stdout:(25/32): python3-simplegeneric-0.8.1-17.el8.noa 172 kB/s | 19 kB 00:00 2024-02-20T00:06:39.481 INFO:teuthology.orchestra.run.smithi084.stdout:(26/32): python3-singledispatch-3.4.0.3-18.el8. 220 kB/s | 24 kB 00:00 2024-02-20T00:06:39.515 INFO:teuthology.orchestra.run.smithi084.stdout:(27/32): python3-tempora-1.14.1-5.el8.noarch.rp 249 kB/s | 29 kB 00:00 2024-02-20T00:06:39.540 INFO:teuthology.orchestra.run.smithi084.stdout:(28/32): python3-trustme-0.6.0-4.el8.noarch.rpm 252 kB/s | 27 kB 00:00 2024-02-20T00:06:39.624 INFO:teuthology.orchestra.run.smithi084.stdout:(29/32): python3-waitress-1.4.3-1.el8.noarch.rp 1.8 MB/s | 255 kB 00:00 2024-02-20T00:06:39.674 INFO:teuthology.orchestra.run.smithi084.stdout:(30/32): python3-webtest-2.0.33-1.el8.noarch.rp 640 kB/s | 85 kB 00:00 2024-02-20T00:06:39.724 INFO:teuthology.orchestra.run.smithi084.stdout:(31/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 1.2 MB/s | 251 kB 00:00 2024-02-20T00:06:39.749 INFO:teuthology.orchestra.run.smithi084.stdout:(32/32): python3-zc-lockfile-2.0-2.el8.noarch.r 183 kB/s | 23 kB 00:00 2024-02-20T00:06:39.753 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-02-20T00:06:39.753 INFO:teuthology.orchestra.run.smithi084.stdout:Total 11 MB/s | 31 MB 00:02 2024-02-20T00:06:39.834 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-02-20T00:06:39.874 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-02-20T00:06:39.874 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-02-20T00:06:39.933 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2024-02-20T00:06:40.099 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-02-20T00:06:40.102 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-02-20T00:06:40.139 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2024-02-20T00:06:40.342 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2024-02-20T00:06:40.583 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2024-02-20T00:06:40.790 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2024-02-20T00:06:40.997 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2024-02-20T00:06:41.168 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2024-02-20T00:06:41.233 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-02-20T00:06:41.328 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2024-02-20T00:06:41.443 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2024-02-20T00:06:41.481 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2024-02-20T00:06:41.637 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2024-02-20T00:06:41.716 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2024-02-20T00:06:41.879 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2024-02-20T00:06:41.928 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2024-02-20T00:06:42.116 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2024-02-20T00:06:42.127 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2024-02-20T00:06:42.317 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2024-02-20T00:06:42.364 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2024-02-20T00:06:42.497 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2024-02-20T00:06:42.699 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2024-02-20T00:06:42.728 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2024-02-20T00:06:42.868 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2024-02-20T00:06:43.020 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-cryptography-3.2.1-7.el8.x86_64 14/32 2024-02-20T00:06:43.069 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2024-02-20T00:06:43.191 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2024-02-20T00:06:43.314 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2024-02-20T00:06:43.543 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2024-02-20T00:06:43.544 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2024-02-20T00:06:43.726 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-02-20T00:06:43.801 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2024-02-20T00:06:44.010 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2024-02-20T00:06:44.161 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2024-02-20T00:06:44.214 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2024-02-20T00:06:44.369 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2024-02-20T00:06:44.448 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-cryptography-3.2.1-7.el8.x86_64 14/32 2024-02-20T00:06:44.563 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2024-02-20T00:06:44.612 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2024-02-20T00:06:44.805 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2024-02-20T00:06:44.879 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2024-02-20T00:06:45.003 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2024-02-20T00:06:45.088 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-02-20T00:06:45.266 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2024-02-20T00:06:45.382 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2024-02-20T00:06:45.594 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2024-02-20T00:06:45.618 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2024-02-20T00:06:45.807 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2024-02-20T00:06:45.932 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2024-02-20T00:06:46.001 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2024-02-20T00:06:46.273 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2024-02-20T00:06:46.306 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 27/32 2024-02-20T00:06:46.499 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2024-02-20T00:06:46.552 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2024-02-20T00:06:46.739 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2024-02-20T00:06:47.070 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2024-02-20T00:06:47.402 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2024-02-20T00:06:47.783 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 27/32 2024-02-20T00:06:48.004 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2024-02-20T00:06:48.078 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2024-02-20T00:06:48.319 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2024-02-20T00:06:48.751 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2024-02-20T00:06:49.010 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2024-02-20T00:06:49.232 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2024-02-20T00:06:49.442 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2024-02-20T00:06:49.459 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2024-02-20T00:06:49.534 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2024-02-20T00:06:49.591 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-2:16.2.4-0.el8.x86_64 32/32 2024-02-20T00:06:49.795 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2024-02-20T00:06:49.996 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-2:16.2.4-0.el8.x86_64 32/32 2024-02-20T00:06:49.996 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-2:16.2.4-0.el8.x86_64 1/32 2024-02-20T00:06:49.996 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-mds-2:16.2.4-0.el8.x86_64 2/32 2024-02-20T00:06:49.996 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-mgr-2:16.2.4-0.el8.x86_64 3/32 2024-02-20T00:06:49.996 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-mon-2:16.2.4-0.el8.x86_64 4/32 2024-02-20T00:06:49.996 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-osd-2:16.2.4-0.el8.x86_64 5/32 2024-02-20T00:06:49.997 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 6/32 2024-02-20T00:06:49.997 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2024-02-20T00:06:49.997 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2024-02-20T00:06:49.997 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2024-02-20T00:06:49.997 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2024-02-20T00:06:49.997 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2024-02-20T00:06:49.997 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2024-02-20T00:06:49.997 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-cryptography-3.2.1-7.el8.x86_64 13/32 2024-02-20T00:06:49.997 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2024-02-20T00:06:49.997 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2024-02-20T00:06:49.997 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2024-02-20T00:06:49.997 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-02-20T00:06:49.997 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2024-02-20T00:06:49.997 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2024-02-20T00:06:49.998 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2024-02-20T00:06:49.998 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2024-02-20T00:06:49.998 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2024-02-20T00:06:49.998 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2024-02-20T00:06:49.998 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2024-02-20T00:06:49.998 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2024-02-20T00:06:49.998 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2024-02-20T00:06:49.999 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2024-02-20T00:06:49.999 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2024-02-20T00:06:49.999 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2024-02-20T00:06:49.999 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2024-02-20T00:06:49.999 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2024-02-20T00:06:50.228 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2024-02-20T00:06:50.453 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2024-02-20T00:06:50.675 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2024-02-20T00:06:50.772 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2024-02-20T00:06:50.773 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:06:50.773 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2024-02-20T00:06:50.773 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-2:16.2.4-0.el8.x86_64 2024-02-20T00:06:50.773 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mds-2:16.2.4-0.el8.x86_64 2024-02-20T00:06:50.773 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-2:16.2.4-0.el8.x86_64 2024-02-20T00:06:50.773 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 2024-02-20T00:06:50.773 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mon-2:16.2.4-0.el8.x86_64 2024-02-20T00:06:50.773 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-osd-2:16.2.4-0.el8.x86_64 2024-02-20T00:06:50.773 INFO:teuthology.orchestra.run.smithi132.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-02-20T00:06:50.773 INFO:teuthology.orchestra.run.smithi132.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-02-20T00:06:50.773 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-02-20T00:06:50.773 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-02-20T00:06:50.773 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-02-20T00:06:50.773 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cryptography-3.2.1-7.el8.x86_64 2024-02-20T00:06:50.774 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jaraco-6.2-6.el8.noarch 2024-02-20T00:06:50.774 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-02-20T00:06:50.774 INFO:teuthology.orchestra.run.smithi132.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-02-20T00:06:50.774 INFO:teuthology.orchestra.run.smithi132.stdout: python3-mako-1.0.6-14.el8.noarch 2024-02-20T00:06:50.774 INFO:teuthology.orchestra.run.smithi132.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-02-20T00:06:50.774 INFO:teuthology.orchestra.run.smithi132.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-02-20T00:06:50.774 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-02-20T00:06:50.775 INFO:teuthology.orchestra.run.smithi132.stdout: python3-portend-2.6-1.el8.noarch 2024-02-20T00:06:50.775 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-02-20T00:06:50.775 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pycparser-2.14-14.el8.noarch 2024-02-20T00:06:50.775 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pytz-2017.2-11.el8.noarch 2024-02-20T00:06:50.775 INFO:teuthology.orchestra.run.smithi132.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-02-20T00:06:50.775 INFO:teuthology.orchestra.run.smithi132.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-02-20T00:06:50.775 INFO:teuthology.orchestra.run.smithi132.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-02-20T00:06:50.775 INFO:teuthology.orchestra.run.smithi132.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-02-20T00:06:50.775 INFO:teuthology.orchestra.run.smithi132.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-02-20T00:06:50.775 INFO:teuthology.orchestra.run.smithi132.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-02-20T00:06:50.775 INFO:teuthology.orchestra.run.smithi132.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-02-20T00:06:50.775 INFO:teuthology.orchestra.run.smithi132.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-02-20T00:06:50.775 INFO:teuthology.orchestra.run.smithi132.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-02-20T00:06:50.776 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:06:50.776 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-02-20T00:06:50.886 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2024-02-20T00:06:50.904 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2024-02-20T00:06:51.028 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-2:16.2.4-0.el8.x86_64 32/32 2024-02-20T00:06:51.036 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install ceph-base 2024-02-20T00:06:51.428 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-2:16.2.4-0.el8.x86_64 32/32 2024-02-20T00:06:51.428 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-2:16.2.4-0.el8.x86_64 1/32 2024-02-20T00:06:51.428 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-mds-2:16.2.4-0.el8.x86_64 2/32 2024-02-20T00:06:51.428 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-mgr-2:16.2.4-0.el8.x86_64 3/32 2024-02-20T00:06:51.428 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-mon-2:16.2.4-0.el8.x86_64 4/32 2024-02-20T00:06:51.428 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-osd-2:16.2.4-0.el8.x86_64 5/32 2024-02-20T00:06:51.428 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 6/32 2024-02-20T00:06:51.428 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2024-02-20T00:06:51.428 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2024-02-20T00:06:51.428 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2024-02-20T00:06:51.429 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2024-02-20T00:06:51.429 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2024-02-20T00:06:51.429 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2024-02-20T00:06:51.429 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-cryptography-3.2.1-7.el8.x86_64 13/32 2024-02-20T00:06:51.429 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2024-02-20T00:06:51.429 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2024-02-20T00:06:51.429 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2024-02-20T00:06:51.429 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-02-20T00:06:51.429 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2024-02-20T00:06:51.429 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2024-02-20T00:06:51.429 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2024-02-20T00:06:51.429 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2024-02-20T00:06:51.429 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2024-02-20T00:06:51.429 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2024-02-20T00:06:51.430 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2024-02-20T00:06:51.430 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2024-02-20T00:06:51.430 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2024-02-20T00:06:51.430 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2024-02-20T00:06:51.430 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2024-02-20T00:06:51.430 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2024-02-20T00:06:51.431 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2024-02-20T00:06:51.431 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2024-02-20T00:06:51.790 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:01:10 ago on Tue 20 Feb 2024 12:05:41 AM UTC. 2024-02-20T00:06:52.197 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2024-02-20T00:06:52.197 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-02-20T00:06:52.197 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-02-20T00:06:52.197 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-2:16.2.4-0.el8.x86_64 2024-02-20T00:06:52.197 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mds-2:16.2.4-0.el8.x86_64 2024-02-20T00:06:52.197 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mgr-2:16.2.4-0.el8.x86_64 2024-02-20T00:06:52.197 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 2024-02-20T00:06:52.198 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mon-2:16.2.4-0.el8.x86_64 2024-02-20T00:06:52.198 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-osd-2:16.2.4-0.el8.x86_64 2024-02-20T00:06:52.198 INFO:teuthology.orchestra.run.smithi084.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-02-20T00:06:52.198 INFO:teuthology.orchestra.run.smithi084.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-02-20T00:06:52.198 INFO:teuthology.orchestra.run.smithi084.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-02-20T00:06:52.198 INFO:teuthology.orchestra.run.smithi084.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-02-20T00:06:52.198 INFO:teuthology.orchestra.run.smithi084.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-02-20T00:06:52.198 INFO:teuthology.orchestra.run.smithi084.stdout: python3-cryptography-3.2.1-7.el8.x86_64 2024-02-20T00:06:52.198 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jaraco-6.2-6.el8.noarch 2024-02-20T00:06:52.198 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-02-20T00:06:52.198 INFO:teuthology.orchestra.run.smithi084.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-02-20T00:06:52.198 INFO:teuthology.orchestra.run.smithi084.stdout: python3-mako-1.0.6-14.el8.noarch 2024-02-20T00:06:52.198 INFO:teuthology.orchestra.run.smithi084.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-02-20T00:06:52.198 INFO:teuthology.orchestra.run.smithi084.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-02-20T00:06:52.199 INFO:teuthology.orchestra.run.smithi084.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-02-20T00:06:52.199 INFO:teuthology.orchestra.run.smithi084.stdout: python3-portend-2.6-1.el8.noarch 2024-02-20T00:06:52.199 INFO:teuthology.orchestra.run.smithi084.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-02-20T00:06:52.199 INFO:teuthology.orchestra.run.smithi084.stdout: python3-pycparser-2.14-14.el8.noarch 2024-02-20T00:06:52.199 INFO:teuthology.orchestra.run.smithi084.stdout: python3-pytz-2017.2-11.el8.noarch 2024-02-20T00:06:52.199 INFO:teuthology.orchestra.run.smithi084.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-02-20T00:06:52.199 INFO:teuthology.orchestra.run.smithi084.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-02-20T00:06:52.199 INFO:teuthology.orchestra.run.smithi084.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-02-20T00:06:52.199 INFO:teuthology.orchestra.run.smithi084.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-02-20T00:06:52.199 INFO:teuthology.orchestra.run.smithi084.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-02-20T00:06:52.199 INFO:teuthology.orchestra.run.smithi084.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-02-20T00:06:52.199 INFO:teuthology.orchestra.run.smithi084.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-02-20T00:06:52.199 INFO:teuthology.orchestra.run.smithi084.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-02-20T00:06:52.199 INFO:teuthology.orchestra.run.smithi084.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-02-20T00:06:52.199 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-02-20T00:06:52.199 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-02-20T00:06:52.463 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install ceph-base 2024-02-20T00:06:53.218 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:01:12 ago on Tue 20 Feb 2024 12:05:41 AM UTC. 2024-02-20T00:06:53.295 INFO:teuthology.orchestra.run.smithi132.stdout:Package ceph-base-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-20T00:06:53.343 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-02-20T00:06:53.344 INFO:teuthology.orchestra.run.smithi132.stdout:Nothing to do. 2024-02-20T00:06:53.344 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-02-20T00:06:53.442 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install cephadm 2024-02-20T00:06:54.197 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:01:13 ago on Tue 20 Feb 2024 12:05:41 AM UTC. 2024-02-20T00:06:54.705 INFO:teuthology.orchestra.run.smithi084.stdout:Package ceph-base-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-20T00:06:54.751 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-02-20T00:06:54.752 INFO:teuthology.orchestra.run.smithi084.stdout:Nothing to do. 2024-02-20T00:06:54.752 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-02-20T00:06:54.867 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install cephadm 2024-02-20T00:06:55.611 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:01:14 ago on Tue 20 Feb 2024 12:05:41 AM UTC. 2024-02-20T00:06:55.740 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-02-20T00:06:55.741 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-02-20T00:06:55.741 INFO:teuthology.orchestra.run.smithi132.stdout: Package Architecture Version Repository Size 2024-02-20T00:06:55.741 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-02-20T00:06:55.741 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2024-02-20T00:06:55.741 INFO:teuthology.orchestra.run.smithi132.stdout: cephadm noarch 2:16.2.4-0.el8 ceph-noarch 71 k 2024-02-20T00:06:55.741 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:06:55.741 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2024-02-20T00:06:55.741 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-02-20T00:06:55.741 INFO:teuthology.orchestra.run.smithi132.stdout:Install 1 Package 2024-02-20T00:06:55.742 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:06:55.742 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 71 k 2024-02-20T00:06:55.742 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 290 k 2024-02-20T00:06:55.742 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2024-02-20T00:06:55.809 INFO:teuthology.orchestra.run.smithi132.stdout:cephadm-16.2.4-0.el8.noarch.rpm 1.0 MB/s | 71 kB 00:00 2024-02-20T00:06:55.809 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2024-02-20T00:06:55.809 INFO:teuthology.orchestra.run.smithi132.stdout:Total 1.0 MB/s | 71 kB 00:00 2024-02-20T00:06:55.810 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2024-02-20T00:06:55.816 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2024-02-20T00:06:55.816 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2024-02-20T00:06:55.818 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2024-02-20T00:06:55.821 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2024-02-20T00:06:55.871 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2024-02-20T00:06:56.377 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2024-02-20T00:06:56.605 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : cephadm-2:16.2.4-0.el8.noarch 1/1 2024-02-20T00:06:56.837 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2024-02-20T00:06:57.154 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-02-20T00:06:57.155 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-02-20T00:06:57.155 INFO:teuthology.orchestra.run.smithi084.stdout: Package Architecture Version Repository Size 2024-02-20T00:06:57.155 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-02-20T00:06:57.155 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-02-20T00:06:57.155 INFO:teuthology.orchestra.run.smithi084.stdout: cephadm noarch 2:16.2.4-0.el8 ceph-noarch 71 k 2024-02-20T00:06:57.155 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-02-20T00:06:57.155 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-02-20T00:06:57.155 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-02-20T00:06:57.156 INFO:teuthology.orchestra.run.smithi084.stdout:Install 1 Package 2024-02-20T00:06:57.156 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-02-20T00:06:57.156 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 71 k 2024-02-20T00:06:57.156 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 290 k 2024-02-20T00:06:57.156 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-02-20T00:06:57.218 INFO:teuthology.orchestra.run.smithi084.stdout:cephadm-16.2.4-0.el8.noarch.rpm 1.1 MB/s | 71 kB 00:00 2024-02-20T00:06:57.219 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-02-20T00:06:57.219 INFO:teuthology.orchestra.run.smithi084.stdout:Total 1.1 MB/s | 71 kB 00:00 2024-02-20T00:06:57.219 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-02-20T00:06:57.226 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-02-20T00:06:57.226 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-02-20T00:06:57.226 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : cephadm-2:16.2.4-0.el8.noarch 1/1 2024-02-20T00:06:57.226 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:06:57.226 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2024-02-20T00:06:57.227 INFO:teuthology.orchestra.run.smithi132.stdout: cephadm-2:16.2.4-0.el8.noarch 2024-02-20T00:06:57.227 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:06:57.227 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-02-20T00:06:57.227 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-02-20T00:06:57.230 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-02-20T00:06:57.272 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-02-20T00:06:57.434 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install ceph-immutable-object-cache 2024-02-20T00:06:57.778 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2024-02-20T00:06:57.939 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : cephadm-2:16.2.4-0.el8.noarch 1/1 2024-02-20T00:06:58.167 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2024-02-20T00:06:58.189 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:01:17 ago on Tue 20 Feb 2024 12:05:41 AM UTC. 2024-02-20T00:06:58.518 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : cephadm-2:16.2.4-0.el8.noarch 1/1 2024-02-20T00:06:58.519 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-02-20T00:06:58.519 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-02-20T00:06:58.519 INFO:teuthology.orchestra.run.smithi084.stdout: cephadm-2:16.2.4-0.el8.noarch 2024-02-20T00:06:58.519 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-02-20T00:06:58.519 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-02-20T00:06:58.749 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install ceph-immutable-object-cache 2024-02-20T00:06:59.506 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:01:18 ago on Tue 20 Feb 2024 12:05:41 AM UTC. 2024-02-20T00:06:59.747 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-02-20T00:06:59.747 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-02-20T00:06:59.747 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repo Size 2024-02-20T00:06:59.747 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-02-20T00:06:59.747 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2024-02-20T00:06:59.748 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-immutable-object-cache x86_64 2:16.2.4-0.el8 ceph 163 k 2024-02-20T00:06:59.748 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:06:59.748 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2024-02-20T00:06:59.748 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-02-20T00:06:59.748 INFO:teuthology.orchestra.run.smithi132.stdout:Install 1 Package 2024-02-20T00:06:59.748 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:06:59.748 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 163 k 2024-02-20T00:06:59.749 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 587 k 2024-02-20T00:06:59.749 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2024-02-20T00:07:00.065 INFO:teuthology.orchestra.run.smithi132.stdout:ceph-immutable-object-cache-16.2.4-0.el8.x86_64 515 kB/s | 163 kB 00:00 2024-02-20T00:07:00.065 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2024-02-20T00:07:00.066 INFO:teuthology.orchestra.run.smithi132.stdout:Total 513 kB/s | 163 kB 00:00 2024-02-20T00:07:00.066 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2024-02-20T00:07:00.077 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2024-02-20T00:07:00.077 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2024-02-20T00:07:00.124 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2024-02-20T00:07:00.127 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2024-02-20T00:07:00.244 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2024-02-20T00:07:00.423 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-02-20T00:07:00.833 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-02-20T00:07:01.037 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-02-20T00:07:01.037 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-02-20T00:07:01.037 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repo Size 2024-02-20T00:07:01.037 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-02-20T00:07:01.038 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-02-20T00:07:01.038 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-immutable-object-cache x86_64 2:16.2.4-0.el8 ceph 163 k 2024-02-20T00:07:01.038 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-02-20T00:07:01.038 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-02-20T00:07:01.038 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-02-20T00:07:01.038 INFO:teuthology.orchestra.run.smithi084.stdout:Install 1 Package 2024-02-20T00:07:01.038 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-02-20T00:07:01.038 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 163 k 2024-02-20T00:07:01.039 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 587 k 2024-02-20T00:07:01.039 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-02-20T00:07:01.212 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-02-20T00:07:01.212 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:07:01.212 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2024-02-20T00:07:01.212 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 2024-02-20T00:07:01.213 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:07:01.213 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-02-20T00:07:01.455 INFO:teuthology.orchestra.run.smithi084.stdout:ceph-immutable-object-cache-16.2.4-0.el8.x86_64 391 kB/s | 163 kB 00:00 2024-02-20T00:07:01.456 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-02-20T00:07:01.456 INFO:teuthology.orchestra.run.smithi084.stdout:Total 390 kB/s | 163 kB 00:00 2024-02-20T00:07:01.457 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-02-20T00:07:01.463 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install ceph-mgr 2024-02-20T00:07:01.468 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-02-20T00:07:01.468 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-02-20T00:07:01.516 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-02-20T00:07:01.519 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-02-20T00:07:02.223 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:01:21 ago on Tue 20 Feb 2024 12:05:41 AM UTC. 2024-02-20T00:07:02.233 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-02-20T00:07:02.592 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-02-20T00:07:03.026 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-02-20T00:07:03.401 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-02-20T00:07:03.402 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-02-20T00:07:03.402 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-02-20T00:07:03.402 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 2024-02-20T00:07:03.402 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-02-20T00:07:03.402 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-02-20T00:07:03.607 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install ceph-mgr 2024-02-20T00:07:03.731 INFO:teuthology.orchestra.run.smithi132.stdout:Package ceph-mgr-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-20T00:07:03.777 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-02-20T00:07:03.778 INFO:teuthology.orchestra.run.smithi132.stdout:Nothing to do. 2024-02-20T00:07:03.778 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-02-20T00:07:03.896 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install ceph-mgr-dashboard 2024-02-20T00:07:04.367 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:01:23 ago on Tue 20 Feb 2024 12:05:41 AM UTC. 2024-02-20T00:07:04.646 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:01:23 ago on Tue 20 Feb 2024 12:05:41 AM UTC. 2024-02-20T00:07:05.856 INFO:teuthology.orchestra.run.smithi084.stdout:Package ceph-mgr-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-20T00:07:05.903 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-02-20T00:07:05.904 INFO:teuthology.orchestra.run.smithi084.stdout:Nothing to do. 2024-02-20T00:07:05.904 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-02-20T00:07:06.017 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install ceph-mgr-dashboard 2024-02-20T00:07:06.189 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-02-20T00:07:06.190 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-02-20T00:07:06.190 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repository Size 2024-02-20T00:07:06.190 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-02-20T00:07:06.190 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2024-02-20T00:07:06.190 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-dashboard noarch 2:16.2.4-0.el8 ceph-noarch 2.7 M 2024-02-20T00:07:06.190 INFO:teuthology.orchestra.run.smithi132.stdout:Installing dependencies: 2024-02-20T00:07:06.190 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-grafana-dashboards noarch 2:16.2.4-0.el8 ceph-noarch 24 k 2024-02-20T00:07:06.190 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-prometheus-alerts noarch 2:16.2.4-0.el8 ceph-noarch 8.9 k 2024-02-20T00:07:06.190 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2024-02-20T00:07:06.190 INFO:teuthology.orchestra.run.smithi132.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-02-20T00:07:06.191 INFO:teuthology.orchestra.run.smithi132.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-02-20T00:07:06.191 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:07:06.191 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2024-02-20T00:07:06.191 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-02-20T00:07:06.191 INFO:teuthology.orchestra.run.smithi132.stdout:Install 6 Packages 2024-02-20T00:07:06.191 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:07:06.191 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 3.0 M 2024-02-20T00:07:06.192 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 15 M 2024-02-20T00:07:06.192 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2024-02-20T00:07:06.461 INFO:teuthology.orchestra.run.smithi132.stdout:(1/6): ceph-grafana-dashboards-16.2.4-0.el8.noa 524 kB/s | 24 kB 00:00 2024-02-20T00:07:06.486 INFO:teuthology.orchestra.run.smithi132.stdout:(2/6): ceph-prometheus-alerts-16.2.4-0.el8.noar 126 kB/s | 8.9 kB 00:00 2024-02-20T00:07:06.562 INFO:teuthology.orchestra.run.smithi132.stdout:(3/6): ceph-mgr-dashboard-16.2.4-0.el8.noarch.r 18 MB/s | 2.7 MB 00:00 2024-02-20T00:07:06.761 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:01:25 ago on Tue 20 Feb 2024 12:05:41 AM UTC. 2024-02-20T00:07:06.803 INFO:teuthology.orchestra.run.smithi132.stdout:(4/6): python3-jwt-1.6.1-2.el8.noarch.rpm 124 kB/s | 43 kB 00:00 2024-02-20T00:07:07.478 INFO:teuthology.orchestra.run.smithi132.stdout:(5/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 34 kB/s | 34 kB 00:00 2024-02-20T00:07:08.120 INFO:teuthology.orchestra.run.smithi132.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 126 kB/s | 196 kB 00:01 2024-02-20T00:07:08.123 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2024-02-20T00:07:08.124 INFO:teuthology.orchestra.run.smithi132.stdout:Total 1.5 MB/s | 3.0 MB 00:01 2024-02-20T00:07:08.135 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2024-02-20T00:07:08.146 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2024-02-20T00:07:08.147 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2024-02-20T00:07:08.233 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2024-02-20T00:07:08.236 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2024-02-20T00:07:08.314 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-02-20T00:07:08.315 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-02-20T00:07:08.315 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repository Size 2024-02-20T00:07:08.315 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-02-20T00:07:08.315 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-02-20T00:07:08.315 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mgr-dashboard noarch 2:16.2.4-0.el8 ceph-noarch 2.7 M 2024-02-20T00:07:08.315 INFO:teuthology.orchestra.run.smithi084.stdout:Installing dependencies: 2024-02-20T00:07:08.315 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-grafana-dashboards noarch 2:16.2.4-0.el8 ceph-noarch 24 k 2024-02-20T00:07:08.315 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-prometheus-alerts noarch 2:16.2.4-0.el8 ceph-noarch 8.9 k 2024-02-20T00:07:08.315 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2024-02-20T00:07:08.315 INFO:teuthology.orchestra.run.smithi084.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-02-20T00:07:08.316 INFO:teuthology.orchestra.run.smithi084.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-02-20T00:07:08.316 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-02-20T00:07:08.316 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-02-20T00:07:08.316 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-02-20T00:07:08.316 INFO:teuthology.orchestra.run.smithi084.stdout:Install 6 Packages 2024-02-20T00:07:08.316 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-02-20T00:07:08.316 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 3.0 M 2024-02-20T00:07:08.317 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 15 M 2024-02-20T00:07:08.317 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-02-20T00:07:08.522 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2024-02-20T00:07:08.716 INFO:teuthology.orchestra.run.smithi084.stdout:(1/6): ceph-grafana-dashboards-16.2.4-0.el8.noa 398 kB/s | 24 kB 00:00 2024-02-20T00:07:08.735 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2024-02-20T00:07:08.741 INFO:teuthology.orchestra.run.smithi084.stdout:(2/6): ceph-prometheus-alerts-16.2.4-0.el8.noar 104 kB/s | 8.9 kB 00:00 2024-02-20T00:07:08.791 INFO:teuthology.orchestra.run.smithi084.stdout:(3/6): ceph-mgr-dashboard-16.2.4-0.el8.noarch.r 20 MB/s | 2.7 MB 00:00 2024-02-20T00:07:08.842 INFO:teuthology.orchestra.run.smithi084.stdout:(4/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 335 kB/s | 34 kB 00:00 2024-02-20T00:07:08.875 INFO:teuthology.orchestra.run.smithi084.stdout:(5/6): python3-routes-2.4.1-12.el8.noarch.rpm 2.3 MB/s | 196 kB 00:00 2024-02-20T00:07:08.972 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2024-02-20T00:07:09.050 INFO:teuthology.orchestra.run.smithi084.stdout:(6/6): python3-jwt-1.6.1-2.el8.noarch.rpm 127 kB/s | 43 kB 00:00 2024-02-20T00:07:09.052 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-02-20T00:07:09.052 INFO:teuthology.orchestra.run.smithi084.stdout:Total 4.0 MB/s | 3.0 MB 00:00 2024-02-20T00:07:09.063 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-02-20T00:07:09.073 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-02-20T00:07:09.073 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-02-20T00:07:09.156 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-02-20T00:07:09.159 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-02-20T00:07:09.223 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2024-02-20T00:07:09.336 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 4/6 2024-02-20T00:07:09.437 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-02-20T00:07:09.671 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2024-02-20T00:07:09.958 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2024-02-20T00:07:10.167 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2024-02-20T00:07:10.271 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 4/6 2024-02-20T00:07:10.319 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 5/6 2024-02-20T00:07:10.590 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2024-02-20T00:07:10.881 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2024-02-20T00:07:10.882 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 1/6 2024-02-20T00:07:10.882 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2/6 2024-02-20T00:07:10.882 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 3/6 2024-02-20T00:07:10.882 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2024-02-20T00:07:10.882 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2024-02-20T00:07:11.298 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 5/6 2024-02-20T00:07:11.446 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2024-02-20T00:07:11.446 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:07:11.446 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2024-02-20T00:07:11.446 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 2024-02-20T00:07:11.446 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2024-02-20T00:07:11.446 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 2024-02-20T00:07:11.446 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-02-20T00:07:11.446 INFO:teuthology.orchestra.run.smithi132.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-02-20T00:07:11.446 INFO:teuthology.orchestra.run.smithi132.stdout: python3-routes-2.4.1-12.el8.noarch 2024-02-20T00:07:11.446 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:07:11.446 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-02-20T00:07:11.572 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install ceph-mgr-diskprediction-local 2024-02-20T00:07:11.590 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2024-02-20T00:07:11.873 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2024-02-20T00:07:11.873 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 1/6 2024-02-20T00:07:11.873 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2/6 2024-02-20T00:07:11.874 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 3/6 2024-02-20T00:07:11.874 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2024-02-20T00:07:11.874 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2024-02-20T00:07:12.244 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2024-02-20T00:07:12.245 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-02-20T00:07:12.245 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-02-20T00:07:12.245 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 2024-02-20T00:07:12.245 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2024-02-20T00:07:12.245 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 2024-02-20T00:07:12.245 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-02-20T00:07:12.246 INFO:teuthology.orchestra.run.smithi084.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-02-20T00:07:12.246 INFO:teuthology.orchestra.run.smithi084.stdout: python3-routes-2.4.1-12.el8.noarch 2024-02-20T00:07:12.246 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-02-20T00:07:12.246 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-02-20T00:07:12.328 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:01:31 ago on Tue 20 Feb 2024 12:05:41 AM UTC. 2024-02-20T00:07:12.378 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install ceph-mgr-diskprediction-local 2024-02-20T00:07:13.131 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:01:32 ago on Tue 20 Feb 2024 12:05:41 AM UTC. 2024-02-20T00:07:13.876 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-02-20T00:07:13.877 INFO:teuthology.orchestra.run.smithi132.stdout:============================================================================================ 2024-02-20T00:07:13.877 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repository Size 2024-02-20T00:07:13.877 INFO:teuthology.orchestra.run.smithi132.stdout:============================================================================================ 2024-02-20T00:07:13.877 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2024-02-20T00:07:13.877 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-diskprediction-local noarch 2:16.2.4-0.el8 ceph-noarch 8.6 M 2024-02-20T00:07:13.877 INFO:teuthology.orchestra.run.smithi132.stdout:Installing dependencies: 2024-02-20T00:07:13.877 INFO:teuthology.orchestra.run.smithi132.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2024-02-20T00:07:13.878 INFO:teuthology.orchestra.run.smithi132.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2024-02-20T00:07:13.878 INFO:teuthology.orchestra.run.smithi132.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2024-02-20T00:07:13.878 INFO:teuthology.orchestra.run.smithi132.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2024-02-20T00:07:13.878 INFO:teuthology.orchestra.run.smithi132.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2024-02-20T00:07:13.878 INFO:teuthology.orchestra.run.smithi132.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2024-02-20T00:07:13.878 INFO:teuthology.orchestra.run.smithi132.stdout: python3-scipy x86_64 1.0.0-21.module_el8+762+77bd8591 appstream 14 M 2024-02-20T00:07:13.878 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:07:13.878 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2024-02-20T00:07:13.878 INFO:teuthology.orchestra.run.smithi132.stdout:============================================================================================ 2024-02-20T00:07:13.878 INFO:teuthology.orchestra.run.smithi132.stdout:Install 8 Packages 2024-02-20T00:07:13.878 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:07:13.879 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 37 M 2024-02-20T00:07:13.879 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 210 M 2024-02-20T00:07:13.879 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2024-02-20T00:07:14.259 INFO:teuthology.orchestra.run.smithi132.stdout:(1/8): ceph-mgr-diskprediction-local-16.2.4-0.e 59 MB/s | 8.6 MB 00:00 2024-02-20T00:07:14.542 INFO:teuthology.orchestra.run.smithi132.stdout:(2/8): openblas-0.3.15-4.el8.x86_64.rpm 11 MB/s | 4.7 MB 00:00 2024-02-20T00:07:14.626 INFO:teuthology.orchestra.run.smithi132.stdout:(3/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 9.5 MB/s | 4.9 MB 00:00 2024-02-20T00:07:14.651 INFO:teuthology.orchestra.run.smithi132.stdout:(4/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 2.0 MB/s | 225 kB 00:00 2024-02-20T00:07:14.676 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-02-20T00:07:14.678 INFO:teuthology.orchestra.run.smithi084.stdout:============================================================================================ 2024-02-20T00:07:14.678 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repository Size 2024-02-20T00:07:14.678 INFO:teuthology.orchestra.run.smithi084.stdout:============================================================================================ 2024-02-20T00:07:14.678 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-02-20T00:07:14.678 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mgr-diskprediction-local noarch 2:16.2.4-0.el8 ceph-noarch 8.6 M 2024-02-20T00:07:14.678 INFO:teuthology.orchestra.run.smithi084.stdout:Installing dependencies: 2024-02-20T00:07:14.678 INFO:teuthology.orchestra.run.smithi084.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2024-02-20T00:07:14.678 INFO:teuthology.orchestra.run.smithi084.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2024-02-20T00:07:14.679 INFO:teuthology.orchestra.run.smithi084.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2024-02-20T00:07:14.679 INFO:teuthology.orchestra.run.smithi084.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2024-02-20T00:07:14.679 INFO:teuthology.orchestra.run.smithi084.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2024-02-20T00:07:14.679 INFO:teuthology.orchestra.run.smithi084.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2024-02-20T00:07:14.679 INFO:teuthology.orchestra.run.smithi084.stdout: python3-scipy x86_64 1.0.0-21.module_el8+762+77bd8591 appstream 14 M 2024-02-20T00:07:14.679 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-02-20T00:07:14.679 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-02-20T00:07:14.679 INFO:teuthology.orchestra.run.smithi084.stdout:============================================================================================ 2024-02-20T00:07:14.679 INFO:teuthology.orchestra.run.smithi084.stdout:Install 8 Packages 2024-02-20T00:07:14.679 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-02-20T00:07:14.680 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 37 M 2024-02-20T00:07:14.680 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 210 M 2024-02-20T00:07:14.680 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-02-20T00:07:14.803 INFO:teuthology.orchestra.run.smithi132.stdout:(5/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 6.9 MB/s | 3.7 MB 00:00 2024-02-20T00:07:14.904 INFO:teuthology.orchestra.run.smithi084.stdout:(1/8): ceph-mgr-diskprediction-local-16.2.4-0.e 61 MB/s | 8.6 MB 00:00 2024-02-20T00:07:15.085 INFO:teuthology.orchestra.run.smithi132.stdout:(6/8): python3-scipy-1.0.0-21.module_el8+762+77 31 MB/s | 14 MB 00:00 2024-02-20T00:07:15.229 INFO:teuthology.orchestra.run.smithi084.stdout:(2/8): openblas-0.3.15-4.el8.x86_64.rpm 10 MB/s | 4.7 MB 00:00 2024-02-20T00:07:15.260 INFO:teuthology.orchestra.run.smithi132.stdout:(7/8): libquadmath-8.5.0-21.el8.x86_64.rpm 376 kB/s | 172 kB 00:00 2024-02-20T00:07:15.296 INFO:teuthology.orchestra.run.smithi084.stdout:(3/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 9.1 MB/s | 4.9 MB 00:00 2024-02-20T00:07:15.302 INFO:teuthology.orchestra.run.smithi132.stdout:(8/8): libgfortran-8.5.0-21.el8.x86_64.rpm 992 kB/s | 645 kB 00:00 2024-02-20T00:07:15.302 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2024-02-20T00:07:15.303 INFO:teuthology.orchestra.run.smithi132.stdout:Total 26 MB/s | 37 MB 00:01 2024-02-20T00:07:15.321 INFO:teuthology.orchestra.run.smithi084.stdout:(4/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 2.4 MB/s | 225 kB 00:00 2024-02-20T00:07:15.388 INFO:teuthology.orchestra.run.smithi084.stdout:(5/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 7.7 MB/s | 3.7 MB 00:00 2024-02-20T00:07:15.531 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2024-02-20T00:07:15.549 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2024-02-20T00:07:15.549 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2024-02-20T00:07:15.789 INFO:teuthology.orchestra.run.smithi084.stdout:(6/8): python3-scipy-1.0.0-21.module_el8+762+77 29 MB/s | 14 MB 00:00 2024-02-20T00:07:15.847 INFO:teuthology.orchestra.run.smithi084.stdout:(7/8): libquadmath-8.5.0-21.el8.x86_64.rpm 376 kB/s | 172 kB 00:00 2024-02-20T00:07:15.859 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2024-02-20T00:07:15.862 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2024-02-20T00:07:16.014 INFO:teuthology.orchestra.run.smithi084.stdout:(8/8): libgfortran-8.5.0-21.el8.x86_64.rpm 932 kB/s | 645 kB 00:00 2024-02-20T00:07:16.015 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-02-20T00:07:16.015 INFO:teuthology.orchestra.run.smithi084.stdout:Total 28 MB/s | 37 MB 00:01 2024-02-20T00:07:16.244 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-02-20T00:07:16.263 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-02-20T00:07:16.263 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-02-20T00:07:16.479 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2024-02-20T00:07:16.574 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-02-20T00:07:16.577 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-02-20T00:07:16.837 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2024-02-20T00:07:17.229 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-02-20T00:07:17.380 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2024-02-20T00:07:17.414 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2024-02-20T00:07:17.524 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2024-02-20T00:07:17.575 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2024-02-20T00:07:17.682 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2024-02-20T00:07:18.173 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2024-02-20T00:07:18.271 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2024-02-20T00:07:18.345 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2024-02-20T00:07:18.663 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2024-02-20T00:07:18.957 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2024-02-20T00:07:19.129 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-02-20T00:07:19.274 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2024-02-20T00:07:19.446 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-02-20T00:07:19.828 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-02-20T00:07:20.136 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-02-20T00:07:20.140 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2024-02-20T00:07:20.420 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-02-20T00:07:20.499 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2024-02-20T00:07:20.763 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-02-20T00:07:22.192 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-02-20T00:07:22.518 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-02-20T00:07:23.835 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 7/8 2024-02-20T00:07:23.958 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 7/8 2024-02-20T00:07:24.153 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2024-02-20T00:07:24.465 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2024-02-20T00:07:24.466 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 1/8 2024-02-20T00:07:24.466 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2024-02-20T00:07:24.466 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2024-02-20T00:07:24.466 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2024-02-20T00:07:24.466 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2024-02-20T00:07:24.466 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 6/8 2024-02-20T00:07:24.466 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2024-02-20T00:07:24.569 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2024-02-20T00:07:24.881 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2024-02-20T00:07:24.881 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 1/8 2024-02-20T00:07:24.881 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2024-02-20T00:07:24.881 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2024-02-20T00:07:24.881 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2024-02-20T00:07:24.881 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2024-02-20T00:07:24.881 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 6/8 2024-02-20T00:07:24.881 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2024-02-20T00:07:24.891 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2024-02-20T00:07:24.891 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:07:24.892 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2024-02-20T00:07:24.892 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 2024-02-20T00:07:24.892 INFO:teuthology.orchestra.run.smithi132.stdout: libgfortran-8.5.0-21.el8.x86_64 2024-02-20T00:07:24.892 INFO:teuthology.orchestra.run.smithi132.stdout: libquadmath-8.5.0-21.el8.x86_64 2024-02-20T00:07:24.892 INFO:teuthology.orchestra.run.smithi132.stdout: openblas-0.3.15-4.el8.x86_64 2024-02-20T00:07:24.892 INFO:teuthology.orchestra.run.smithi132.stdout: openblas-threads-0.3.15-4.el8.x86_64 2024-02-20T00:07:24.892 INFO:teuthology.orchestra.run.smithi132.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-02-20T00:07:24.892 INFO:teuthology.orchestra.run.smithi132.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-02-20T00:07:24.892 INFO:teuthology.orchestra.run.smithi132.stdout: python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_64 2024-02-20T00:07:24.892 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:07:24.892 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-02-20T00:07:25.037 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install ceph-mgr-rook 2024-02-20T00:07:25.274 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2024-02-20T00:07:25.274 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-02-20T00:07:25.274 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-02-20T00:07:25.274 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 2024-02-20T00:07:25.274 INFO:teuthology.orchestra.run.smithi084.stdout: libgfortran-8.5.0-21.el8.x86_64 2024-02-20T00:07:25.274 INFO:teuthology.orchestra.run.smithi084.stdout: libquadmath-8.5.0-21.el8.x86_64 2024-02-20T00:07:25.275 INFO:teuthology.orchestra.run.smithi084.stdout: openblas-0.3.15-4.el8.x86_64 2024-02-20T00:07:25.275 INFO:teuthology.orchestra.run.smithi084.stdout: openblas-threads-0.3.15-4.el8.x86_64 2024-02-20T00:07:25.275 INFO:teuthology.orchestra.run.smithi084.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-02-20T00:07:25.275 INFO:teuthology.orchestra.run.smithi084.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-02-20T00:07:25.275 INFO:teuthology.orchestra.run.smithi084.stdout: python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_64 2024-02-20T00:07:25.275 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-02-20T00:07:25.275 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-02-20T00:07:25.393 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install ceph-mgr-rook 2024-02-20T00:07:25.794 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:01:44 ago on Tue 20 Feb 2024 12:05:41 AM UTC. 2024-02-20T00:07:26.149 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:01:45 ago on Tue 20 Feb 2024 12:05:41 AM UTC. 2024-02-20T00:07:27.343 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-02-20T00:07:27.345 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-02-20T00:07:27.345 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repository Size 2024-02-20T00:07:27.345 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-02-20T00:07:27.345 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2024-02-20T00:07:27.345 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-rook noarch 2:16.2.4-0.el8 ceph-noarch 27 k 2024-02-20T00:07:27.345 INFO:teuthology.orchestra.run.smithi132.stdout:Installing dependencies: 2024-02-20T00:07:27.345 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-02-20T00:07:27.345 INFO:teuthology.orchestra.run.smithi132.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-02-20T00:07:27.346 INFO:teuthology.orchestra.run.smithi132.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-02-20T00:07:27.346 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2024-02-20T00:07:27.346 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2024-02-20T00:07:27.346 INFO:teuthology.orchestra.run.smithi132.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-02-20T00:07:27.346 INFO:teuthology.orchestra.run.smithi132.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2024-02-20T00:07:27.346 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2024-02-20T00:07:27.346 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2024-02-20T00:07:27.346 INFO:teuthology.orchestra.run.smithi132.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2024-02-20T00:07:27.346 INFO:teuthology.orchestra.run.smithi132.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-02-20T00:07:27.346 INFO:teuthology.orchestra.run.smithi132.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-02-20T00:07:27.346 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:07:27.346 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2024-02-20T00:07:27.346 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-02-20T00:07:27.346 INFO:teuthology.orchestra.run.smithi132.stdout:Install 13 Packages 2024-02-20T00:07:27.347 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:07:27.347 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 1.9 M 2024-02-20T00:07:27.347 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 24 M 2024-02-20T00:07:27.347 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2024-02-20T00:07:27.717 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-02-20T00:07:27.719 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-02-20T00:07:27.719 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repository Size 2024-02-20T00:07:27.719 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-02-20T00:07:27.719 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-02-20T00:07:27.719 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mgr-rook noarch 2:16.2.4-0.el8 ceph-noarch 27 k 2024-02-20T00:07:27.719 INFO:teuthology.orchestra.run.smithi084.stdout:Installing dependencies: 2024-02-20T00:07:27.719 INFO:teuthology.orchestra.run.smithi084.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-02-20T00:07:27.719 INFO:teuthology.orchestra.run.smithi084.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-02-20T00:07:27.720 INFO:teuthology.orchestra.run.smithi084.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-02-20T00:07:27.720 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2024-02-20T00:07:27.720 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2024-02-20T00:07:27.720 INFO:teuthology.orchestra.run.smithi084.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-02-20T00:07:27.720 INFO:teuthology.orchestra.run.smithi084.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2024-02-20T00:07:27.720 INFO:teuthology.orchestra.run.smithi084.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2024-02-20T00:07:27.720 INFO:teuthology.orchestra.run.smithi084.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2024-02-20T00:07:27.720 INFO:teuthology.orchestra.run.smithi084.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2024-02-20T00:07:27.720 INFO:teuthology.orchestra.run.smithi084.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-02-20T00:07:27.720 INFO:teuthology.orchestra.run.smithi084.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-02-20T00:07:27.720 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-02-20T00:07:27.720 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-02-20T00:07:27.720 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-02-20T00:07:27.720 INFO:teuthology.orchestra.run.smithi084.stdout:Install 13 Packages 2024-02-20T00:07:27.720 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-02-20T00:07:27.721 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 1.9 M 2024-02-20T00:07:27.721 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 24 M 2024-02-20T00:07:27.721 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-02-20T00:07:27.746 INFO:teuthology.orchestra.run.smithi132.stdout:(1/13): ceph-mgr-rook-16.2.4-0.el8.noarch.rpm 571 kB/s | 27 kB 00:00 2024-02-20T00:07:27.822 INFO:teuthology.orchestra.run.smithi132.stdout:(2/13): python3-jsonpointer-1.10-11.el8.noarch. 163 kB/s | 20 kB 00:00 2024-02-20T00:07:27.892 INFO:teuthology.orchestra.run.smithi132.stdout:(3/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 138 kB/s | 27 kB 00:00 2024-02-20T00:07:27.965 INFO:teuthology.orchestra.run.smithi132.stdout:(4/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 581 kB/s | 126 kB 00:00 2024-02-20T00:07:28.007 INFO:teuthology.orchestra.run.smithi132.stdout:(5/13): python3-pyasn1-modules-0.3.7-6.el8.noar 597 kB/s | 110 kB 00:00 2024-02-20T00:07:28.048 INFO:teuthology.orchestra.run.smithi084.stdout:(1/13): ceph-mgr-rook-16.2.4-0.el8.noarch.rpm 598 kB/s | 27 kB 00:00 2024-02-20T00:07:28.106 INFO:teuthology.orchestra.run.smithi084.stdout:(2/13): python3-jsonpointer-1.10-11.el8.noarch. 193 kB/s | 20 kB 00:00 2024-02-20T00:07:28.131 INFO:teuthology.orchestra.run.smithi084.stdout:(3/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 206 kB/s | 27 kB 00:00 2024-02-20T00:07:28.156 INFO:teuthology.orchestra.run.smithi084.stdout:(4/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 1.1 MB/s | 126 kB 00:00 2024-02-20T00:07:28.165 INFO:teuthology.orchestra.run.smithi132.stdout:(6/13): python3-cachetools-3.1.1-4.el8.noarch.r 208 kB/s | 33 kB 00:00 2024-02-20T00:07:28.182 INFO:teuthology.orchestra.run.smithi084.stdout:(5/13): python3-pyasn1-modules-0.3.7-6.el8.noar 1.4 MB/s | 110 kB 00:00 2024-02-20T00:07:28.232 INFO:teuthology.orchestra.run.smithi132.stdout:(7/13): python3-certifi-2018.10.15-7.el8.noarch 244 kB/s | 16 kB 00:00 2024-02-20T00:07:28.274 INFO:teuthology.orchestra.run.smithi132.stdout:(8/13): python3-requests-oauthlib-1.0.0-1.el8.n 138 kB/s | 43 kB 00:00 2024-02-20T00:07:28.342 INFO:teuthology.orchestra.run.smithi132.stdout:(9/13): python3-google-auth-1.1.1-10.el8.noarch 746 kB/s | 81 kB 00:00 2024-02-20T00:07:28.357 INFO:teuthology.orchestra.run.smithi084.stdout:(6/13): python3-cachetools-3.1.1-4.el8.noarch.r 188 kB/s | 33 kB 00:00 2024-02-20T00:07:28.417 INFO:teuthology.orchestra.run.smithi132.stdout:(10/13): python3-rsa-4.9-2.el8.noarch.rpm 857 kB/s | 64 kB 00:00 2024-02-20T00:07:28.432 INFO:teuthology.orchestra.run.smithi084.stdout:(7/13): python3-certifi-2018.10.15-7.el8.noarch 217 kB/s | 16 kB 00:00 2024-02-20T00:07:28.442 INFO:teuthology.orchestra.run.smithi132.stdout:(11/13): python3-oauthlib-2.1.0-1.el8.noarch.rp 282 kB/s | 155 kB 00:00 2024-02-20T00:07:28.483 INFO:teuthology.orchestra.run.smithi132.stdout:(12/13): python3-websocket-client-0.56.0-5.el8. 917 kB/s | 61 kB 00:00 2024-02-20T00:07:28.491 INFO:teuthology.orchestra.run.smithi084.stdout:(8/13): python3-requests-oauthlib-1.0.0-1.el8.n 127 kB/s | 43 kB 00:00 2024-02-20T00:07:28.583 INFO:teuthology.orchestra.run.smithi084.stdout:(9/13): python3-google-auth-1.1.1-10.el8.noarch 540 kB/s | 81 kB 00:00 2024-02-20T00:07:28.642 INFO:teuthology.orchestra.run.smithi132.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 3.2 MB/s | 1.2 MB 00:00 2024-02-20T00:07:28.642 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2024-02-20T00:07:28.643 INFO:teuthology.orchestra.run.smithi132.stdout:Total 1.5 MB/s | 1.9 MB 00:01 2024-02-20T00:07:28.667 INFO:teuthology.orchestra.run.smithi084.stdout:(10/13): python3-oauthlib-2.1.0-1.el8.noarch.rp 289 kB/s | 155 kB 00:00 2024-02-20T00:07:28.684 INFO:teuthology.orchestra.run.smithi084.stdout:(11/13): python3-rsa-4.9-2.el8.noarch.rpm 640 kB/s | 64 kB 00:00 2024-02-20T00:07:28.699 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2024-02-20T00:07:28.715 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2024-02-20T00:07:28.715 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2024-02-20T00:07:28.804 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2024-02-20T00:07:28.807 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2024-02-20T00:07:28.859 INFO:teuthology.orchestra.run.smithi084.stdout:(12/13): python3-websocket-client-0.56.0-5.el8. 319 kB/s | 61 kB 00:00 2024-02-20T00:07:28.901 INFO:teuthology.orchestra.run.smithi084.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 2.8 MB/s | 1.2 MB 00:00 2024-02-20T00:07:28.901 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-02-20T00:07:28.901 INFO:teuthology.orchestra.run.smithi084.stdout:Total 1.6 MB/s | 1.9 MB 00:01 2024-02-20T00:07:28.955 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-02-20T00:07:28.971 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-02-20T00:07:28.971 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-02-20T00:07:29.057 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-02-20T00:07:29.060 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-02-20T00:07:29.342 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2024-02-20T00:07:29.587 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2024-02-20T00:07:29.614 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-02-20T00:07:29.752 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2024-02-20T00:07:29.862 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2024-02-20T00:07:29.970 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2024-02-20T00:07:30.068 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2024-02-20T00:07:30.208 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2024-02-20T00:07:30.346 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2024-02-20T00:07:30.407 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2024-02-20T00:07:30.583 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2024-02-20T00:07:30.629 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2024-02-20T00:07:30.823 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2024-02-20T00:07:30.922 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2024-02-20T00:07:31.087 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2024-02-20T00:07:31.138 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2024-02-20T00:07:31.412 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2024-02-20T00:07:31.655 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2024-02-20T00:07:32.179 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2024-02-20T00:07:32.432 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2024-02-20T00:07:32.650 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2024-02-20T00:07:32.691 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2024-02-20T00:07:32.825 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2024-02-20T00:07:32.958 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2024-02-20T00:07:33.055 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2024-02-20T00:07:33.200 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2024-02-20T00:07:33.353 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2024-02-20T00:07:33.354 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-mgr-rook-2:16.2.4-0.el8.noarch 1/13 2024-02-20T00:07:33.354 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2024-02-20T00:07:33.354 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2024-02-20T00:07:33.354 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2024-02-20T00:07:33.354 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2024-02-20T00:07:33.354 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2024-02-20T00:07:33.354 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2024-02-20T00:07:33.354 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2024-02-20T00:07:33.354 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2024-02-20T00:07:33.354 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2024-02-20T00:07:33.354 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2024-02-20T00:07:33.354 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2024-02-20T00:07:33.400 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2024-02-20T00:07:33.551 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2024-02-20T00:07:33.809 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2024-02-20T00:07:33.810 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:07:33.810 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2024-02-20T00:07:33.810 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-rook-2:16.2.4-0.el8.noarch 2024-02-20T00:07:33.810 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-02-20T00:07:33.810 INFO:teuthology.orchestra.run.smithi132.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-02-20T00:07:33.810 INFO:teuthology.orchestra.run.smithi132.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-02-20T00:07:33.810 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-02-20T00:07:33.810 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-02-20T00:07:33.810 INFO:teuthology.orchestra.run.smithi132.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-02-20T00:07:33.810 INFO:teuthology.orchestra.run.smithi132.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-02-20T00:07:33.810 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-02-20T00:07:33.810 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-02-20T00:07:33.810 INFO:teuthology.orchestra.run.smithi132.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-02-20T00:07:33.811 INFO:teuthology.orchestra.run.smithi132.stdout: python3-rsa-4.9-2.el8.noarch 2024-02-20T00:07:33.811 INFO:teuthology.orchestra.run.smithi132.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-02-20T00:07:33.811 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:07:33.811 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-02-20T00:07:33.843 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2024-02-20T00:07:33.843 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-mgr-rook-2:16.2.4-0.el8.noarch 1/13 2024-02-20T00:07:33.844 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2024-02-20T00:07:33.844 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2024-02-20T00:07:33.844 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2024-02-20T00:07:33.844 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2024-02-20T00:07:33.844 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2024-02-20T00:07:33.844 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2024-02-20T00:07:33.844 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2024-02-20T00:07:33.844 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2024-02-20T00:07:33.844 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2024-02-20T00:07:33.844 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2024-02-20T00:07:33.844 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2024-02-20T00:07:33.934 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install ceph-mgr-cephadm 2024-02-20T00:07:34.272 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2024-02-20T00:07:34.273 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-02-20T00:07:34.273 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-02-20T00:07:34.273 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mgr-rook-2:16.2.4-0.el8.noarch 2024-02-20T00:07:34.273 INFO:teuthology.orchestra.run.smithi084.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-02-20T00:07:34.273 INFO:teuthology.orchestra.run.smithi084.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-02-20T00:07:34.273 INFO:teuthology.orchestra.run.smithi084.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-02-20T00:07:34.273 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-02-20T00:07:34.273 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-02-20T00:07:34.273 INFO:teuthology.orchestra.run.smithi084.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-02-20T00:07:34.273 INFO:teuthology.orchestra.run.smithi084.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-02-20T00:07:34.273 INFO:teuthology.orchestra.run.smithi084.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-02-20T00:07:34.273 INFO:teuthology.orchestra.run.smithi084.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-02-20T00:07:34.273 INFO:teuthology.orchestra.run.smithi084.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-02-20T00:07:34.273 INFO:teuthology.orchestra.run.smithi084.stdout: python3-rsa-4.9-2.el8.noarch 2024-02-20T00:07:34.273 INFO:teuthology.orchestra.run.smithi084.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-02-20T00:07:34.274 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-02-20T00:07:34.274 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-02-20T00:07:34.400 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install ceph-mgr-cephadm 2024-02-20T00:07:34.686 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:01:53 ago on Tue 20 Feb 2024 12:05:41 AM UTC. 2024-02-20T00:07:35.150 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:01:54 ago on Tue 20 Feb 2024 12:05:41 AM UTC. 2024-02-20T00:07:36.237 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-02-20T00:07:36.238 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-02-20T00:07:36.239 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repository Size 2024-02-20T00:07:36.239 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-02-20T00:07:36.239 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2024-02-20T00:07:36.239 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-cephadm noarch 2:16.2.4-0.el8 ceph-noarch 89 k 2024-02-20T00:07:36.239 INFO:teuthology.orchestra.run.smithi132.stdout:Installing dependencies: 2024-02-20T00:07:36.239 INFO:teuthology.orchestra.run.smithi132.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2024-02-20T00:07:36.239 INFO:teuthology.orchestra.run.smithi132.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2024-02-20T00:07:36.239 INFO:teuthology.orchestra.run.smithi132.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2024-02-20T00:07:36.239 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jinja2 noarch 2.10.1-4.el8 appstream 578 k 2024-02-20T00:07:36.239 INFO:teuthology.orchestra.run.smithi132.stdout: python3-remoto noarch 1.2.1-1.el8 epel 49 k 2024-02-20T00:07:36.239 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:07:36.239 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2024-02-20T00:07:36.239 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-02-20T00:07:36.239 INFO:teuthology.orchestra.run.smithi132.stdout:Install 6 Packages 2024-02-20T00:07:36.239 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:07:36.240 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 5.9 M 2024-02-20T00:07:36.240 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 24 M 2024-02-20T00:07:36.240 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2024-02-20T00:07:36.367 INFO:teuthology.orchestra.run.smithi132.stdout:(1/6): ceph-mgr-cephadm-16.2.4-0.el8.noarch.rpm 2.0 MB/s | 89 kB 00:00 2024-02-20T00:07:36.534 INFO:teuthology.orchestra.run.smithi132.stdout:(2/6): python3-apipkg-1.5-6.el8.noarch.rpm 127 kB/s | 21 kB 00:00 2024-02-20T00:07:36.609 INFO:teuthology.orchestra.run.smithi132.stdout:(3/6): python3-jinja2-2.10.1-4.el8.noarch.rpm 2.0 MB/s | 578 kB 00:00 2024-02-20T00:07:36.718 INFO:teuthology.orchestra.run.smithi132.stdout:(4/6): python3-execnet-1.7.1-1.el8.noarch.rpm 1.9 MB/s | 355 kB 00:00 2024-02-20T00:07:36.752 INFO:teuthology.orchestra.run.smithi132.stdout:(5/6): python3-remoto-1.2.1-1.el8.noarch.rpm 344 kB/s | 49 kB 00:00 2024-02-20T00:07:36.762 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-02-20T00:07:36.763 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-02-20T00:07:36.763 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repository Size 2024-02-20T00:07:36.763 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-02-20T00:07:36.763 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-02-20T00:07:36.763 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mgr-cephadm noarch 2:16.2.4-0.el8 ceph-noarch 89 k 2024-02-20T00:07:36.764 INFO:teuthology.orchestra.run.smithi084.stdout:Installing dependencies: 2024-02-20T00:07:36.764 INFO:teuthology.orchestra.run.smithi084.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2024-02-20T00:07:36.764 INFO:teuthology.orchestra.run.smithi084.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2024-02-20T00:07:36.764 INFO:teuthology.orchestra.run.smithi084.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2024-02-20T00:07:36.764 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jinja2 noarch 2.10.1-4.el8 appstream 578 k 2024-02-20T00:07:36.764 INFO:teuthology.orchestra.run.smithi084.stdout: python3-remoto noarch 1.2.1-1.el8 epel 49 k 2024-02-20T00:07:36.764 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-02-20T00:07:36.764 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-02-20T00:07:36.765 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-02-20T00:07:36.765 INFO:teuthology.orchestra.run.smithi084.stdout:Install 6 Packages 2024-02-20T00:07:36.765 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-02-20T00:07:36.766 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 5.9 M 2024-02-20T00:07:36.766 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 24 M 2024-02-20T00:07:36.766 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-02-20T00:07:36.952 INFO:teuthology.orchestra.run.smithi132.stdout:(6/6): python3-babel-2.5.1-7.el8.noarch.rpm 7.6 MB/s | 4.8 MB 00:00 2024-02-20T00:07:36.952 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2024-02-20T00:07:36.952 INFO:teuthology.orchestra.run.smithi132.stdout:Total 8.2 MB/s | 5.9 MB 00:00 2024-02-20T00:07:37.002 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2024-02-20T00:07:37.012 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2024-02-20T00:07:37.012 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2024-02-20T00:07:37.106 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2024-02-20T00:07:37.109 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2024-02-20T00:07:37.190 INFO:teuthology.orchestra.run.smithi084.stdout:(1/6): ceph-mgr-cephadm-16.2.4-0.el8.noarch.rpm 1.8 MB/s | 89 kB 00:00 2024-02-20T00:07:37.282 INFO:teuthology.orchestra.run.smithi084.stdout:(2/6): python3-apipkg-1.5-6.el8.noarch.rpm 231 kB/s | 21 kB 00:00 2024-02-20T00:07:37.308 INFO:teuthology.orchestra.run.smithi084.stdout:(3/6): python3-jinja2-2.10.1-4.el8.noarch.rpm 3.4 MB/s | 578 kB 00:00 2024-02-20T00:07:37.393 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2024-02-20T00:07:37.399 INFO:teuthology.orchestra.run.smithi084.stdout:(4/6): python3-remoto-1.2.1-1.el8.noarch.rpm 534 kB/s | 49 kB 00:00 2024-02-20T00:07:37.441 INFO:teuthology.orchestra.run.smithi084.stdout:(5/6): python3-execnet-1.7.1-1.el8.noarch.rpm 2.2 MB/s | 355 kB 00:00 2024-02-20T00:07:37.600 INFO:teuthology.orchestra.run.smithi084.stdout:(6/6): python3-babel-2.5.1-7.el8.noarch.rpm 10 MB/s | 4.8 MB 00:00 2024-02-20T00:07:37.600 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-02-20T00:07:37.600 INFO:teuthology.orchestra.run.smithi084.stdout:Total 7.0 MB/s | 5.9 MB 00:00 2024-02-20T00:07:37.651 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-02-20T00:07:37.661 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-02-20T00:07:37.661 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-02-20T00:07:37.674 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2024-02-20T00:07:37.753 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-02-20T00:07:37.756 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-02-20T00:07:37.991 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2024-02-20T00:07:38.032 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-02-20T00:07:38.316 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2024-02-20T00:07:38.604 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2024-02-20T00:07:38.896 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-remoto-1.2.1-1.el8.noarch 3/6 2024-02-20T00:07:39.270 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 4/6 2024-02-20T00:07:39.480 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-jinja2-2.10.1-4.el8.noarch 5/6 2024-02-20T00:07:39.531 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-remoto-1.2.1-1.el8.noarch 3/6 2024-02-20T00:07:39.725 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2024-02-20T00:07:39.995 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2024-02-20T00:07:39.995 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 1/6 2024-02-20T00:07:39.995 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/6 2024-02-20T00:07:39.995 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-jinja2-2.10.1-4.el8.noarch 3/6 2024-02-20T00:07:39.996 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2024-02-20T00:07:39.996 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2024-02-20T00:07:40.263 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 4/6 2024-02-20T00:07:40.398 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-remoto-1.2.1-1.el8.noarch 6/6 2024-02-20T00:07:40.398 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:07:40.398 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2024-02-20T00:07:40.398 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch python3-apipkg-1.5-6.el8.noarch 2024-02-20T00:07:40.398 INFO:teuthology.orchestra.run.smithi132.stdout: python3-babel-2.5.1-7.el8.noarch python3-execnet-1.7.1-1.el8.noarch 2024-02-20T00:07:40.398 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jinja2-2.10.1-4.el8.noarch python3-remoto-1.2.1-1.el8.noarch 2024-02-20T00:07:40.398 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:07:40.398 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-02-20T00:07:40.496 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-jinja2-2.10.1-4.el8.noarch 5/6 2024-02-20T00:07:40.526 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install ceph-fuse 2024-02-20T00:07:40.750 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2024-02-20T00:07:41.038 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2024-02-20T00:07:41.039 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 1/6 2024-02-20T00:07:41.039 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/6 2024-02-20T00:07:41.039 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-jinja2-2.10.1-4.el8.noarch 3/6 2024-02-20T00:07:41.039 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2024-02-20T00:07:41.039 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2024-02-20T00:07:41.293 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:02:00 ago on Tue 20 Feb 2024 12:05:41 AM UTC. 2024-02-20T00:07:41.415 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-remoto-1.2.1-1.el8.noarch 6/6 2024-02-20T00:07:41.415 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-02-20T00:07:41.416 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-02-20T00:07:41.416 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch python3-apipkg-1.5-6.el8.noarch 2024-02-20T00:07:41.416 INFO:teuthology.orchestra.run.smithi084.stdout: python3-babel-2.5.1-7.el8.noarch python3-execnet-1.7.1-1.el8.noarch 2024-02-20T00:07:41.416 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jinja2-2.10.1-4.el8.noarch python3-remoto-1.2.1-1.el8.noarch 2024-02-20T00:07:41.416 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-02-20T00:07:41.417 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-02-20T00:07:41.538 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install ceph-fuse 2024-02-20T00:07:42.289 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:02:01 ago on Tue 20 Feb 2024 12:05:41 AM UTC. 2024-02-20T00:07:42.844 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-02-20T00:07:42.845 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-02-20T00:07:42.845 INFO:teuthology.orchestra.run.smithi132.stdout: Package Architecture Version Repository Size 2024-02-20T00:07:42.845 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-02-20T00:07:42.845 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2024-02-20T00:07:42.845 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-fuse x86_64 2:16.2.4-0.el8 ceph 885 k 2024-02-20T00:07:42.845 INFO:teuthology.orchestra.run.smithi132.stdout:Installing dependencies: 2024-02-20T00:07:42.845 INFO:teuthology.orchestra.run.smithi132.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2024-02-20T00:07:42.846 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:07:42.846 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2024-02-20T00:07:42.846 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-02-20T00:07:42.846 INFO:teuthology.orchestra.run.smithi132.stdout:Install 2 Packages 2024-02-20T00:07:42.846 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:07:42.846 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 968 k 2024-02-20T00:07:42.846 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 3.2 M 2024-02-20T00:07:42.846 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2024-02-20T00:07:42.944 INFO:teuthology.orchestra.run.smithi132.stdout:(1/2): ceph-fuse-16.2.4-0.el8.x86_64.rpm 15 MB/s | 885 kB 00:00 2024-02-20T00:07:43.473 INFO:teuthology.orchestra.run.smithi132.stdout:(2/2): fuse-2.9.7-17.el8.x86_64.rpm 142 kB/s | 83 kB 00:00 2024-02-20T00:07:43.473 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2024-02-20T00:07:43.474 INFO:teuthology.orchestra.run.smithi132.stdout:Total 1.5 MB/s | 968 kB 00:00 2024-02-20T00:07:43.485 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2024-02-20T00:07:43.507 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2024-02-20T00:07:43.507 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2024-02-20T00:07:43.601 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2024-02-20T00:07:43.604 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2024-02-20T00:07:43.813 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-02-20T00:07:43.814 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-02-20T00:07:43.814 INFO:teuthology.orchestra.run.smithi084.stdout: Package Architecture Version Repository Size 2024-02-20T00:07:43.814 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-02-20T00:07:43.814 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-02-20T00:07:43.814 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-fuse x86_64 2:16.2.4-0.el8 ceph 885 k 2024-02-20T00:07:43.814 INFO:teuthology.orchestra.run.smithi084.stdout:Installing dependencies: 2024-02-20T00:07:43.814 INFO:teuthology.orchestra.run.smithi084.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2024-02-20T00:07:43.814 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-02-20T00:07:43.814 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-02-20T00:07:43.814 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-02-20T00:07:43.814 INFO:teuthology.orchestra.run.smithi084.stdout:Install 2 Packages 2024-02-20T00:07:43.815 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-02-20T00:07:43.815 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 968 k 2024-02-20T00:07:43.815 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 3.2 M 2024-02-20T00:07:43.815 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-02-20T00:07:43.908 INFO:teuthology.orchestra.run.smithi084.stdout:(1/2): ceph-fuse-16.2.4-0.el8.x86_64.rpm 17 MB/s | 885 kB 00:00 2024-02-20T00:07:43.915 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2024-02-20T00:07:44.174 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2024-02-20T00:07:44.241 INFO:teuthology.orchestra.run.smithi084.stdout:(2/2): fuse-2.9.7-17.el8.x86_64.rpm 215 kB/s | 83 kB 00:00 2024-02-20T00:07:44.242 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-02-20T00:07:44.242 INFO:teuthology.orchestra.run.smithi084.stdout:Total 2.2 MB/s | 968 kB 00:00 2024-02-20T00:07:44.252 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-02-20T00:07:44.273 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-02-20T00:07:44.273 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-02-20T00:07:44.358 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2024-02-20T00:07:44.359 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-02-20T00:07:44.362 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-02-20T00:07:44.535 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-02-20T00:07:44.766 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2024-02-20T00:07:44.766 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-fuse-2:16.2.4-0.el8.x86_64 1/2 2024-02-20T00:07:44.840 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2024-02-20T00:07:45.024 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2024-02-20T00:07:45.156 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2024-02-20T00:07:45.156 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:07:45.156 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2024-02-20T00:07:45.156 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-fuse-2:16.2.4-0.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2024-02-20T00:07:45.157 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:07:45.157 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-02-20T00:07:45.370 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install librados-devel 2024-02-20T00:07:45.412 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2024-02-20T00:07:45.412 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-fuse-2:16.2.4-0.el8.x86_64 1/2 2024-02-20T00:07:45.812 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2024-02-20T00:07:45.812 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-02-20T00:07:45.812 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-02-20T00:07:45.813 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-fuse-2:16.2.4-0.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2024-02-20T00:07:45.813 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-02-20T00:07:45.813 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-02-20T00:07:46.057 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install librados-devel 2024-02-20T00:07:46.141 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:02:05 ago on Tue 20 Feb 2024 12:05:41 AM UTC. 2024-02-20T00:07:46.823 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:02:05 ago on Tue 20 Feb 2024 12:05:41 AM UTC. 2024-02-20T00:07:47.699 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-02-20T00:07:47.700 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-02-20T00:07:47.700 INFO:teuthology.orchestra.run.smithi132.stdout: Package Architecture Version Repository Size 2024-02-20T00:07:47.700 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-02-20T00:07:47.700 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2024-02-20T00:07:47.700 INFO:teuthology.orchestra.run.smithi132.stdout: librados-devel x86_64 2:16.2.4-0.el8 ceph 131 k 2024-02-20T00:07:47.700 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:07:47.700 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2024-02-20T00:07:47.700 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-02-20T00:07:47.700 INFO:teuthology.orchestra.run.smithi132.stdout:Install 1 Package 2024-02-20T00:07:47.700 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:07:47.701 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 131 k 2024-02-20T00:07:47.701 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 439 k 2024-02-20T00:07:47.701 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2024-02-20T00:07:47.743 INFO:teuthology.orchestra.run.smithi132.stdout:librados-devel-16.2.4-0.el8.x86_64.rpm 3.1 MB/s | 131 kB 00:00 2024-02-20T00:07:47.743 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2024-02-20T00:07:47.743 INFO:teuthology.orchestra.run.smithi132.stdout:Total 3.0 MB/s | 131 kB 00:00 2024-02-20T00:07:47.744 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2024-02-20T00:07:47.749 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2024-02-20T00:07:47.749 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2024-02-20T00:07:47.798 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2024-02-20T00:07:47.801 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2024-02-20T00:07:47.923 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2024-02-20T00:07:48.164 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-20T00:07:48.380 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-02-20T00:07:48.380 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-02-20T00:07:48.380 INFO:teuthology.orchestra.run.smithi084.stdout: Package Architecture Version Repository Size 2024-02-20T00:07:48.380 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-02-20T00:07:48.380 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-02-20T00:07:48.381 INFO:teuthology.orchestra.run.smithi084.stdout: librados-devel x86_64 2:16.2.4-0.el8 ceph 131 k 2024-02-20T00:07:48.381 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-02-20T00:07:48.381 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-02-20T00:07:48.381 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-02-20T00:07:48.381 INFO:teuthology.orchestra.run.smithi084.stdout:Install 1 Package 2024-02-20T00:07:48.381 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-02-20T00:07:48.381 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 131 k 2024-02-20T00:07:48.381 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 439 k 2024-02-20T00:07:48.382 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-02-20T00:07:48.421 INFO:teuthology.orchestra.run.smithi084.stdout:librados-devel-16.2.4-0.el8.x86_64.rpm 3.2 MB/s | 131 kB 00:00 2024-02-20T00:07:48.422 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-02-20T00:07:48.422 INFO:teuthology.orchestra.run.smithi084.stdout:Total 3.1 MB/s | 131 kB 00:00 2024-02-20T00:07:48.422 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-02-20T00:07:48.428 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-02-20T00:07:48.428 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-02-20T00:07:48.459 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-20T00:07:48.478 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-02-20T00:07:48.481 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-02-20T00:07:48.597 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-02-20T00:07:48.863 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-20T00:07:48.868 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-20T00:07:48.868 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:07:48.868 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2024-02-20T00:07:48.868 INFO:teuthology.orchestra.run.smithi132.stdout: librados-devel-2:16.2.4-0.el8.x86_64 2024-02-20T00:07:48.869 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:07:48.869 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-02-20T00:07:49.161 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-20T00:07:49.546 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-20T00:07:49.546 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-02-20T00:07:49.546 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-02-20T00:07:49.546 INFO:teuthology.orchestra.run.smithi084.stdout: librados-devel-2:16.2.4-0.el8.x86_64 2024-02-20T00:07:49.546 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-02-20T00:07:49.546 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-02-20T00:07:49.776 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install libcephfs2 2024-02-20T00:07:50.153 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install libcephfs2 2024-02-20T00:07:50.540 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:02:09 ago on Tue 20 Feb 2024 12:05:41 AM UTC. 2024-02-20T00:07:50.925 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:02:09 ago on Tue 20 Feb 2024 12:05:41 AM UTC. 2024-02-20T00:07:52.041 INFO:teuthology.orchestra.run.smithi084.stdout:Package libcephfs2-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-20T00:07:52.088 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-02-20T00:07:52.089 INFO:teuthology.orchestra.run.smithi084.stdout:Nothing to do. 2024-02-20T00:07:52.089 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-02-20T00:07:52.195 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install libcephfs-devel 2024-02-20T00:07:52.422 INFO:teuthology.orchestra.run.smithi132.stdout:Package libcephfs2-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-20T00:07:52.468 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-02-20T00:07:52.469 INFO:teuthology.orchestra.run.smithi132.stdout:Nothing to do. 2024-02-20T00:07:52.470 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-02-20T00:07:52.562 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install libcephfs-devel 2024-02-20T00:07:52.949 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:02:11 ago on Tue 20 Feb 2024 12:05:41 AM UTC. 2024-02-20T00:07:53.318 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:02:12 ago on Tue 20 Feb 2024 12:05:41 AM UTC. 2024-02-20T00:07:54.484 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-02-20T00:07:54.484 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-02-20T00:07:54.485 INFO:teuthology.orchestra.run.smithi084.stdout: Package Architecture Version Repository Size 2024-02-20T00:07:54.485 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-02-20T00:07:54.485 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-02-20T00:07:54.485 INFO:teuthology.orchestra.run.smithi084.stdout: libcephfs-devel x86_64 2:16.2.4-0.el8 ceph 24 k 2024-02-20T00:07:54.485 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-02-20T00:07:54.485 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-02-20T00:07:54.485 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-02-20T00:07:54.485 INFO:teuthology.orchestra.run.smithi084.stdout:Install 1 Package 2024-02-20T00:07:54.485 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-02-20T00:07:54.486 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 24 k 2024-02-20T00:07:54.486 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 94 k 2024-02-20T00:07:54.486 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-02-20T00:07:54.525 INFO:teuthology.orchestra.run.smithi084.stdout:libcephfs-devel-16.2.4-0.el8.x86_64.rpm 626 kB/s | 24 kB 00:00 2024-02-20T00:07:54.525 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-02-20T00:07:54.525 INFO:teuthology.orchestra.run.smithi084.stdout:Total 607 kB/s | 24 kB 00:00 2024-02-20T00:07:54.526 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-02-20T00:07:54.529 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-02-20T00:07:54.529 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-02-20T00:07:54.531 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-02-20T00:07:54.534 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-02-20T00:07:54.588 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-02-20T00:07:54.816 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-20T00:07:54.875 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-02-20T00:07:54.876 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-02-20T00:07:54.876 INFO:teuthology.orchestra.run.smithi132.stdout: Package Architecture Version Repository Size 2024-02-20T00:07:54.876 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-02-20T00:07:54.876 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2024-02-20T00:07:54.876 INFO:teuthology.orchestra.run.smithi132.stdout: libcephfs-devel x86_64 2:16.2.4-0.el8 ceph 24 k 2024-02-20T00:07:54.876 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:07:54.876 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2024-02-20T00:07:54.877 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-02-20T00:07:54.877 INFO:teuthology.orchestra.run.smithi132.stdout:Install 1 Package 2024-02-20T00:07:54.877 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:07:54.877 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 24 k 2024-02-20T00:07:54.877 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 94 k 2024-02-20T00:07:54.877 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2024-02-20T00:07:54.931 INFO:teuthology.orchestra.run.smithi132.stdout:libcephfs-devel-16.2.4-0.el8.x86_64.rpm 451 kB/s | 24 kB 00:00 2024-02-20T00:07:54.931 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2024-02-20T00:07:54.932 INFO:teuthology.orchestra.run.smithi132.stdout:Total 439 kB/s | 24 kB 00:00 2024-02-20T00:07:54.932 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2024-02-20T00:07:54.935 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2024-02-20T00:07:54.935 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2024-02-20T00:07:54.937 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2024-02-20T00:07:54.941 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2024-02-20T00:07:54.989 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2024-02-20T00:07:55.084 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-20T00:07:55.273 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-20T00:07:55.418 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-20T00:07:55.419 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-02-20T00:07:55.419 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-02-20T00:07:55.419 INFO:teuthology.orchestra.run.smithi084.stdout: libcephfs-devel-2:16.2.4-0.el8.x86_64 2024-02-20T00:07:55.419 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-02-20T00:07:55.419 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-02-20T00:07:55.535 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install librados2 2024-02-20T00:07:55.540 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-20T00:07:56.296 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:02:15 ago on Tue 20 Feb 2024 12:05:41 AM UTC. 2024-02-20T00:07:56.739 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-20T00:07:56.739 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:07:56.739 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2024-02-20T00:07:56.739 INFO:teuthology.orchestra.run.smithi132.stdout: libcephfs-devel-2:16.2.4-0.el8.x86_64 2024-02-20T00:07:56.739 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:07:56.739 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-02-20T00:07:56.862 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install librados2 2024-02-20T00:07:57.624 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:02:16 ago on Tue 20 Feb 2024 12:05:41 AM UTC. 2024-02-20T00:07:57.799 INFO:teuthology.orchestra.run.smithi084.stdout:Package librados2-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-20T00:07:57.845 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-02-20T00:07:57.847 INFO:teuthology.orchestra.run.smithi084.stdout:Nothing to do. 2024-02-20T00:07:57.847 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-02-20T00:07:57.946 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install librbd1 2024-02-20T00:07:58.697 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:02:17 ago on Tue 20 Feb 2024 12:05:41 AM UTC. 2024-02-20T00:07:59.133 INFO:teuthology.orchestra.run.smithi132.stdout:Package librados2-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-20T00:07:59.180 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-02-20T00:07:59.182 INFO:teuthology.orchestra.run.smithi132.stdout:Nothing to do. 2024-02-20T00:07:59.182 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-02-20T00:07:59.280 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install librbd1 2024-02-20T00:08:00.029 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:02:19 ago on Tue 20 Feb 2024 12:05:41 AM UTC. 2024-02-20T00:08:00.226 INFO:teuthology.orchestra.run.smithi084.stdout:Package librbd1-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-20T00:08:00.271 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-02-20T00:08:00.273 INFO:teuthology.orchestra.run.smithi084.stdout:Nothing to do. 2024-02-20T00:08:00.273 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-02-20T00:08:00.371 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install python3-rados 2024-02-20T00:08:01.118 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:02:20 ago on Tue 20 Feb 2024 12:05:41 AM UTC. 2024-02-20T00:08:01.533 INFO:teuthology.orchestra.run.smithi132.stdout:Package librbd1-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-20T00:08:01.579 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-02-20T00:08:01.581 INFO:teuthology.orchestra.run.smithi132.stdout:Nothing to do. 2024-02-20T00:08:01.581 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-02-20T00:08:01.681 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install python3-rados 2024-02-20T00:08:02.435 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:02:21 ago on Tue 20 Feb 2024 12:05:41 AM UTC. 2024-02-20T00:08:02.610 INFO:teuthology.orchestra.run.smithi084.stdout:Package python3-rados-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-20T00:08:02.657 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-02-20T00:08:02.658 INFO:teuthology.orchestra.run.smithi084.stdout:Nothing to do. 2024-02-20T00:08:02.658 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-02-20T00:08:02.754 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install python3-rgw 2024-02-20T00:08:03.514 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:02:22 ago on Tue 20 Feb 2024 12:05:41 AM UTC. 2024-02-20T00:08:03.958 INFO:teuthology.orchestra.run.smithi132.stdout:Package python3-rados-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-20T00:08:04.005 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-02-20T00:08:04.006 INFO:teuthology.orchestra.run.smithi132.stdout:Nothing to do. 2024-02-20T00:08:04.006 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-02-20T00:08:04.123 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install python3-rgw 2024-02-20T00:08:04.879 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:02:23 ago on Tue 20 Feb 2024 12:05:41 AM UTC. 2024-02-20T00:08:05.022 INFO:teuthology.orchestra.run.smithi084.stdout:Package python3-rgw-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-20T00:08:05.069 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-02-20T00:08:05.070 INFO:teuthology.orchestra.run.smithi084.stdout:Nothing to do. 2024-02-20T00:08:05.070 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-02-20T00:08:05.186 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install python3-cephfs 2024-02-20T00:08:05.940 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:02:24 ago on Tue 20 Feb 2024 12:05:41 AM UTC. 2024-02-20T00:08:06.374 INFO:teuthology.orchestra.run.smithi132.stdout:Package python3-rgw-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-20T00:08:06.421 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-02-20T00:08:06.422 INFO:teuthology.orchestra.run.smithi132.stdout:Nothing to do. 2024-02-20T00:08:06.422 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-02-20T00:08:06.548 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install python3-cephfs 2024-02-20T00:08:07.302 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:02:26 ago on Tue 20 Feb 2024 12:05:41 AM UTC. 2024-02-20T00:08:07.431 INFO:teuthology.orchestra.run.smithi084.stdout:Package python3-cephfs-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-20T00:08:07.477 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-02-20T00:08:07.478 INFO:teuthology.orchestra.run.smithi084.stdout:Nothing to do. 2024-02-20T00:08:07.479 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-02-20T00:08:07.571 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install python3-rbd 2024-02-20T00:08:08.322 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:02:27 ago on Tue 20 Feb 2024 12:05:41 AM UTC. 2024-02-20T00:08:08.803 INFO:teuthology.orchestra.run.smithi132.stdout:Package python3-cephfs-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-20T00:08:08.849 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-02-20T00:08:08.850 INFO:teuthology.orchestra.run.smithi132.stdout:Nothing to do. 2024-02-20T00:08:08.850 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-02-20T00:08:08.952 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install python3-rbd 2024-02-20T00:08:09.705 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:02:28 ago on Tue 20 Feb 2024 12:05:41 AM UTC. 2024-02-20T00:08:09.821 INFO:teuthology.orchestra.run.smithi084.stdout:Package python3-rbd-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-20T00:08:09.868 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-02-20T00:08:09.869 INFO:teuthology.orchestra.run.smithi084.stdout:Nothing to do. 2024-02-20T00:08:09.869 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-02-20T00:08:09.975 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install rbd-fuse 2024-02-20T00:08:10.729 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:02:29 ago on Tue 20 Feb 2024 12:05:41 AM UTC. 2024-02-20T00:08:11.208 INFO:teuthology.orchestra.run.smithi132.stdout:Package python3-rbd-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-20T00:08:11.256 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-02-20T00:08:11.256 INFO:teuthology.orchestra.run.smithi132.stdout:Nothing to do. 2024-02-20T00:08:11.256 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-02-20T00:08:11.366 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install rbd-fuse 2024-02-20T00:08:12.139 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:02:31 ago on Tue 20 Feb 2024 12:05:41 AM UTC. 2024-02-20T00:08:12.274 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-02-20T00:08:12.275 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-02-20T00:08:12.275 INFO:teuthology.orchestra.run.smithi084.stdout: Package Architecture Version Repository Size 2024-02-20T00:08:12.275 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-02-20T00:08:12.275 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-02-20T00:08:12.275 INFO:teuthology.orchestra.run.smithi084.stdout: rbd-fuse x86_64 2:16.2.4-0.el8 ceph 89 k 2024-02-20T00:08:12.275 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-02-20T00:08:12.275 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-02-20T00:08:12.276 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-02-20T00:08:12.276 INFO:teuthology.orchestra.run.smithi084.stdout:Install 1 Package 2024-02-20T00:08:12.276 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-02-20T00:08:12.279 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 89 k 2024-02-20T00:08:12.279 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 298 k 2024-02-20T00:08:12.279 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-02-20T00:08:12.322 INFO:teuthology.orchestra.run.smithi084.stdout:rbd-fuse-16.2.4-0.el8.x86_64.rpm 1.9 MB/s | 89 kB 00:00 2024-02-20T00:08:12.322 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-02-20T00:08:12.322 INFO:teuthology.orchestra.run.smithi084.stdout:Total 1.9 MB/s | 89 kB 00:00 2024-02-20T00:08:12.323 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-02-20T00:08:12.332 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-02-20T00:08:12.332 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-02-20T00:08:12.384 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-02-20T00:08:12.387 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-02-20T00:08:12.915 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-02-20T00:08:13.142 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-02-20T00:08:13.441 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-02-20T00:08:13.682 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-02-20T00:08:13.682 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-02-20T00:08:13.682 INFO:teuthology.orchestra.run.smithi132.stdout: Package Architecture Version Repository Size 2024-02-20T00:08:13.683 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-02-20T00:08:13.683 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2024-02-20T00:08:13.683 INFO:teuthology.orchestra.run.smithi132.stdout: rbd-fuse x86_64 2:16.2.4-0.el8 ceph 89 k 2024-02-20T00:08:13.683 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:08:13.683 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2024-02-20T00:08:13.683 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-02-20T00:08:13.683 INFO:teuthology.orchestra.run.smithi132.stdout:Install 1 Package 2024-02-20T00:08:13.683 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:08:13.684 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 89 k 2024-02-20T00:08:13.684 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 298 k 2024-02-20T00:08:13.684 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2024-02-20T00:08:13.740 INFO:teuthology.orchestra.run.smithi132.stdout:rbd-fuse-16.2.4-0.el8.x86_64.rpm 1.6 MB/s | 89 kB 00:00 2024-02-20T00:08:13.740 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2024-02-20T00:08:13.741 INFO:teuthology.orchestra.run.smithi132.stdout:Total 1.5 MB/s | 89 kB 00:00 2024-02-20T00:08:13.741 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2024-02-20T00:08:13.751 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2024-02-20T00:08:13.751 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2024-02-20T00:08:13.800 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2024-02-20T00:08:13.803 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2024-02-20T00:08:13.811 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-02-20T00:08:13.811 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-02-20T00:08:13.811 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-02-20T00:08:13.811 INFO:teuthology.orchestra.run.smithi084.stdout: rbd-fuse-2:16.2.4-0.el8.x86_64 2024-02-20T00:08:13.811 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-02-20T00:08:13.811 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-02-20T00:08:13.923 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2024-02-20T00:08:14.057 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install rbd-mirror 2024-02-20T00:08:14.110 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-02-20T00:08:14.405 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-02-20T00:08:14.799 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-02-20T00:08:14.800 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:08:14.800 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2024-02-20T00:08:14.800 INFO:teuthology.orchestra.run.smithi132.stdout: rbd-fuse-2:16.2.4-0.el8.x86_64 2024-02-20T00:08:14.800 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:08:14.800 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-02-20T00:08:14.817 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:02:33 ago on Tue 20 Feb 2024 12:05:41 AM UTC. 2024-02-20T00:08:15.014 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install rbd-mirror 2024-02-20T00:08:15.781 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:02:34 ago on Tue 20 Feb 2024 12:05:41 AM UTC. 2024-02-20T00:08:16.384 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-02-20T00:08:16.385 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-02-20T00:08:16.385 INFO:teuthology.orchestra.run.smithi084.stdout: Package Architecture Version Repository Size 2024-02-20T00:08:16.385 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-02-20T00:08:16.385 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-02-20T00:08:16.385 INFO:teuthology.orchestra.run.smithi084.stdout: rbd-mirror x86_64 2:16.2.4-0.el8 ceph 3.7 M 2024-02-20T00:08:16.385 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-02-20T00:08:16.385 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-02-20T00:08:16.385 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-02-20T00:08:16.385 INFO:teuthology.orchestra.run.smithi084.stdout:Install 1 Package 2024-02-20T00:08:16.385 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-02-20T00:08:16.386 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 3.7 M 2024-02-20T00:08:16.386 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 18 M 2024-02-20T00:08:16.386 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-02-20T00:08:16.476 INFO:teuthology.orchestra.run.smithi084.stdout:rbd-mirror-16.2.4-0.el8.x86_64.rpm 41 MB/s | 3.7 MB 00:00 2024-02-20T00:08:16.476 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-02-20T00:08:16.476 INFO:teuthology.orchestra.run.smithi084.stdout:Total 41 MB/s | 3.7 MB 00:00 2024-02-20T00:08:16.477 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-02-20T00:08:16.488 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-02-20T00:08:16.489 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-02-20T00:08:16.550 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-02-20T00:08:16.553 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-02-20T00:08:17.020 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-02-20T00:08:17.234 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-02-20T00:08:17.338 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-02-20T00:08:17.338 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-02-20T00:08:17.338 INFO:teuthology.orchestra.run.smithi132.stdout: Package Architecture Version Repository Size 2024-02-20T00:08:17.339 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-02-20T00:08:17.339 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2024-02-20T00:08:17.339 INFO:teuthology.orchestra.run.smithi132.stdout: rbd-mirror x86_64 2:16.2.4-0.el8 ceph 3.7 M 2024-02-20T00:08:17.339 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:08:17.339 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2024-02-20T00:08:17.339 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-02-20T00:08:17.339 INFO:teuthology.orchestra.run.smithi132.stdout:Install 1 Package 2024-02-20T00:08:17.339 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:08:17.339 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 3.7 M 2024-02-20T00:08:17.340 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 18 M 2024-02-20T00:08:17.340 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2024-02-20T00:08:17.443 INFO:teuthology.orchestra.run.smithi132.stdout:rbd-mirror-16.2.4-0.el8.x86_64.rpm 36 MB/s | 3.7 MB 00:00 2024-02-20T00:08:17.443 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2024-02-20T00:08:17.444 INFO:teuthology.orchestra.run.smithi132.stdout:Total 35 MB/s | 3.7 MB 00:00 2024-02-20T00:08:17.444 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2024-02-20T00:08:17.456 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2024-02-20T00:08:17.456 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2024-02-20T00:08:17.519 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2024-02-20T00:08:17.522 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2024-02-20T00:08:17.681 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-02-20T00:08:18.056 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-02-20T00:08:18.056 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-02-20T00:08:18.056 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-02-20T00:08:18.056 INFO:teuthology.orchestra.run.smithi084.stdout: rbd-mirror-2:16.2.4-0.el8.x86_64 2024-02-20T00:08:18.056 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-02-20T00:08:18.057 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-02-20T00:08:18.275 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install rbd-nbd 2024-02-20T00:08:18.293 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2024-02-20T00:08:18.519 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-02-20T00:08:18.935 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-02-20T00:08:19.035 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:02:38 ago on Tue 20 Feb 2024 12:05:41 AM UTC. 2024-02-20T00:08:19.306 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-02-20T00:08:19.306 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:08:19.306 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2024-02-20T00:08:19.307 INFO:teuthology.orchestra.run.smithi132.stdout: rbd-mirror-2:16.2.4-0.el8.x86_64 2024-02-20T00:08:19.307 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:08:19.307 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-02-20T00:08:19.543 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install rbd-nbd 2024-02-20T00:08:20.311 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:02:39 ago on Tue 20 Feb 2024 12:05:41 AM UTC. 2024-02-20T00:08:20.574 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-02-20T00:08:20.574 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-02-20T00:08:20.574 INFO:teuthology.orchestra.run.smithi084.stdout: Package Architecture Version Repository Size 2024-02-20T00:08:20.574 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-02-20T00:08:20.574 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-02-20T00:08:20.574 INFO:teuthology.orchestra.run.smithi084.stdout: rbd-nbd x86_64 2:16.2.4-0.el8 ceph 173 k 2024-02-20T00:08:20.574 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-02-20T00:08:20.575 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-02-20T00:08:20.575 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-02-20T00:08:20.575 INFO:teuthology.orchestra.run.smithi084.stdout:Install 1 Package 2024-02-20T00:08:20.575 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-02-20T00:08:20.575 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 173 k 2024-02-20T00:08:20.575 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 649 k 2024-02-20T00:08:20.576 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-02-20T00:08:20.619 INFO:teuthology.orchestra.run.smithi084.stdout:rbd-nbd-16.2.4-0.el8.x86_64.rpm 3.9 MB/s | 173 kB 00:00 2024-02-20T00:08:20.620 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-02-20T00:08:20.620 INFO:teuthology.orchestra.run.smithi084.stdout:Total 3.8 MB/s | 173 kB 00:00 2024-02-20T00:08:20.621 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-02-20T00:08:20.631 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-02-20T00:08:20.631 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-02-20T00:08:20.682 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-02-20T00:08:20.684 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-02-20T00:08:20.803 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-02-20T00:08:21.020 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-02-20T00:08:21.293 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-02-20T00:08:21.687 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-02-20T00:08:21.687 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-02-20T00:08:21.687 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-02-20T00:08:21.687 INFO:teuthology.orchestra.run.smithi084.stdout: rbd-nbd-2:16.2.4-0.el8.x86_64 2024-02-20T00:08:21.687 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-02-20T00:08:21.687 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-02-20T00:08:21.905 DEBUG:teuthology.parallel:result is None 2024-02-20T00:08:21.929 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-02-20T00:08:21.930 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-02-20T00:08:21.930 INFO:teuthology.orchestra.run.smithi132.stdout: Package Architecture Version Repository Size 2024-02-20T00:08:21.930 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-02-20T00:08:21.930 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2024-02-20T00:08:21.930 INFO:teuthology.orchestra.run.smithi132.stdout: rbd-nbd x86_64 2:16.2.4-0.el8 ceph 173 k 2024-02-20T00:08:21.930 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:08:21.931 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2024-02-20T00:08:21.931 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-02-20T00:08:21.931 INFO:teuthology.orchestra.run.smithi132.stdout:Install 1 Package 2024-02-20T00:08:21.931 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:08:21.931 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 173 k 2024-02-20T00:08:21.931 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 649 k 2024-02-20T00:08:21.931 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2024-02-20T00:08:21.979 INFO:teuthology.orchestra.run.smithi132.stdout:rbd-nbd-16.2.4-0.el8.x86_64.rpm 3.5 MB/s | 173 kB 00:00 2024-02-20T00:08:21.980 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2024-02-20T00:08:21.980 INFO:teuthology.orchestra.run.smithi132.stdout:Total 3.4 MB/s | 173 kB 00:00 2024-02-20T00:08:21.980 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2024-02-20T00:08:21.991 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2024-02-20T00:08:21.991 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2024-02-20T00:08:22.042 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2024-02-20T00:08:22.045 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2024-02-20T00:08:22.175 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2024-02-20T00:08:22.396 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-02-20T00:08:22.671 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-02-20T00:08:23.066 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-02-20T00:08:23.066 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:08:23.066 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2024-02-20T00:08:23.066 INFO:teuthology.orchestra.run.smithi132.stdout: rbd-nbd-2:16.2.4-0.el8.x86_64 2024-02-20T00:08:23.066 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:08:23.066 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-02-20T00:08:23.374 DEBUG:teuthology.parallel:result is None 2024-02-20T00:08:23.374 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-02-20T00:08:23.374 INFO:teuthology.packaging:ref: None 2024-02-20T00:08:23.375 INFO:teuthology.packaging:tag: v16.2.4 2024-02-20T00:08:23.375 INFO:teuthology.packaging:branch: None 2024-02-20T00:08:23.375 INFO:teuthology.packaging:sha1: 19854089e18d4f65dda2b6cd74e737367c2514bd 2024-02-20T00:08:23.375 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2024-02-20T00:08:23.519 DEBUG:teuthology.orchestra.run.smithi084:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-02-20T00:08:23.546 INFO:teuthology.orchestra.run.smithi084.stdout:16.2.4-0.el8 2024-02-20T00:08:23.547 INFO:teuthology.packaging:The installed version of ceph is 16.2.4-0.el8 2024-02-20T00:08:23.547 INFO:teuthology.task.install:The correct ceph version 16.2.4-0 is installed. 2024-02-20T00:08:23.549 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-02-20T00:08:23.549 INFO:teuthology.packaging:ref: None 2024-02-20T00:08:23.549 INFO:teuthology.packaging:tag: v16.2.4 2024-02-20T00:08:23.549 INFO:teuthology.packaging:branch: None 2024-02-20T00:08:23.549 INFO:teuthology.packaging:sha1: 19854089e18d4f65dda2b6cd74e737367c2514bd 2024-02-20T00:08:23.549 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2024-02-20T00:08:23.761 DEBUG:teuthology.orchestra.run.smithi132:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-02-20T00:08:23.784 INFO:teuthology.orchestra.run.smithi132.stdout:16.2.4-0.el8 2024-02-20T00:08:23.784 INFO:teuthology.packaging:The installed version of ceph is 16.2.4-0.el8 2024-02-20T00:08:23.784 INFO:teuthology.task.install:The correct ceph version 16.2.4-0 is installed. 2024-02-20T00:08:23.786 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-02-20T00:08:23.787 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-02-20T00:08:23.787 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-02-20T00:08:23.821 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-02-20T00:08:23.821 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-02-20T00:08:23.852 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-02-20T00:08:23.852 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-02-20T00:08:23.852 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd of=/usr/bin/daemon-helper 2024-02-20T00:08:23.892 DEBUG:teuthology.orchestra.run.smithi084:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-02-20T00:08:23.962 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-02-20T00:08:23.962 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd of=/usr/bin/daemon-helper 2024-02-20T00:08:23.989 DEBUG:teuthology.orchestra.run.smithi132:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-02-20T00:08:24.056 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-02-20T00:08:24.056 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-02-20T00:08:24.056 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd of=/usr/bin/adjust-ulimits 2024-02-20T00:08:24.086 DEBUG:teuthology.orchestra.run.smithi084:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-02-20T00:08:24.155 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-02-20T00:08:24.155 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd of=/usr/bin/adjust-ulimits 2024-02-20T00:08:24.182 DEBUG:teuthology.orchestra.run.smithi132:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-02-20T00:08:24.248 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-02-20T00:08:24.249 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-02-20T00:08:24.249 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd of=/usr/bin/stdin-killer 2024-02-20T00:08:24.277 DEBUG:teuthology.orchestra.run.smithi084:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-02-20T00:08:24.345 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-02-20T00:08:24.345 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd of=/usr/bin/stdin-killer 2024-02-20T00:08:24.371 DEBUG:teuthology.orchestra.run.smithi132:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-02-20T00:08:24.438 INFO:teuthology.run_tasks:Running task print... 2024-02-20T00:08:24.451 INFO:teuthology.task.print:**** done install task... 2024-02-20T00:08:24.451 INFO:teuthology.run_tasks:Running task cephadm... 2024-02-20T00:08:24.537 INFO:tasks.cephadm:Config: {'avoid_pacific_features': True, 'cephadm_branch': 'v16.2.4', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'image': 'docker.io/ceph/ceph:v16.2.4', 'roleless': True, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'mon pg warn min per osd': 0}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(PG_DEGRADED\\)', 'Degraded data redundancy', '\\(PG_', 'MDS_INSUFFICIENT_STANDBY', 'deprecated feature inline_data', 'compat changed unexpectedly', 'overall HEALTH_', 'OSD_DOWN', 'OSD_', 'but it is still running', 'is not responding'], 'sha1': '19854089e18d4f65dda2b6cd74e737367c2514bd'} 2024-02-20T00:08:24.537 INFO:tasks.cephadm:Cluster image is docker.io/ceph/ceph:v16.2.4 2024-02-20T00:08:24.538 INFO:tasks.cephadm:Cluster fsid is 2a2e6c42-cf84-11ee-95bb-87774f69a715 2024-02-20T00:08:24.538 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-02-20T00:08:24.538 INFO:tasks.cephadm:No mon roles; fabricating mons 2024-02-20T00:08:24.538 INFO:tasks.cephadm:Monitor IPs: {'mon.smithi084': '172.21.15.84', 'mon.smithi132': '172.21.15.132'} 2024-02-20T00:08:24.538 INFO:tasks.cephadm:Normalizing hostnames... 2024-02-20T00:08:24.538 DEBUG:teuthology.orchestra.run.smithi084:> sudo hostname $(hostname -s) 2024-02-20T00:08:24.568 DEBUG:teuthology.orchestra.run.smithi132:> sudo hostname $(hostname -s) 2024-02-20T00:08:24.595 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref v16.2.4)... 2024-02-20T00:08:24.596 DEBUG:teuthology.orchestra.run.smithi084:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v16.2.4/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-02-20T00:08:24.749 INFO:teuthology.orchestra.run.smithi084.stdout:-rw-rw-r--. 1 ubuntu ubuntu 292005 Feb 20 00:08 /home/ubuntu/cephtest/cephadm 2024-02-20T00:08:24.749 DEBUG:teuthology.orchestra.run.smithi132:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v16.2.4/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-02-20T00:08:24.880 INFO:teuthology.orchestra.run.smithi132.stdout:-rw-rw-r--. 1 ubuntu ubuntu 292005 Feb 20 00:08 /home/ubuntu/cephtest/cephadm 2024-02-20T00:08:24.881 DEBUG:teuthology.orchestra.run.smithi084:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-02-20T00:08:24.901 DEBUG:teuthology.orchestra.run.smithi132:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-02-20T00:08:24.929 DEBUG:teuthology.orchestra.run.smithi084:> sudo mkdir -p /etc/ceph 2024-02-20T00:08:24.970 DEBUG:teuthology.orchestra.run.smithi132:> sudo mkdir -p /etc/ceph 2024-02-20T00:08:24.997 DEBUG:teuthology.orchestra.run.smithi084:> sudo chmod 777 /etc/ceph 2024-02-20T00:08:25.038 DEBUG:teuthology.orchestra.run.smithi132:> sudo chmod 777 /etc/ceph 2024-02-20T00:08:25.065 INFO:tasks.cephadm:Writing seed config... 2024-02-20T00:08:25.066 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-02-20T00:08:25.066 INFO:tasks.cephadm: override: [client] debug client = 20 2024-02-20T00:08:25.066 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-02-20T00:08:25.066 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-02-20T00:08:25.066 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-02-20T00:08:25.066 INFO:tasks.cephadm: override: [global] mon pg warn min per osd = 0 2024-02-20T00:08:25.066 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-02-20T00:08:25.066 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-02-20T00:08:25.066 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-02-20T00:08:25.066 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-02-20T00:08:25.066 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-02-20T00:08:25.066 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-02-20T00:08:25.067 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-02-20T00:08:25.067 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-02-20T00:08:25.067 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-02-20T00:08:25.067 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-02-20T00:08:25.067 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-02-20T00:08:25.067 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-02-20T00:08:25.067 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-02-20T00:08:25.067 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-02-20T00:08:25.067 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-02-20T00:08:25.067 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-02-20T00:08:25.067 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2024-02-20T00:08:25.067 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2024-02-20T00:08:25.067 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2024-02-20T00:08:25.067 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-02-20T00:08:25.067 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-02-20T00:08:25.068 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2024-02-20T00:08:25.068 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2024-02-20T00:08:25.068 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-02-20T00:08:25.068 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-02-20T00:08:25.068 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2024-02-20T00:08:25.068 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-02-20T00:08:25.068 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-02-20T00:08:25.068 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-02-20T00:08:25.068 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-02-20T00:08:25.068 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-02-20T00:08:25.068 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-02-20T00:08:25.069 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-02-20T00:08:25.069 DEBUG:teuthology.orchestra.run.smithi084:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-02-20T00:08:25.096 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 2a2e6c42-cf84-11ee-95bb-87774f69a715 mon pg warn min per osd = 0 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-02-20T00:08:25.096 DEBUG:teuthology.orchestra.run.smithi084:mon.smithi084> sudo journalctl -f -n 0 -u ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715@mon.smithi084.service 2024-02-20T00:08:25.138 INFO:tasks.cephadm:Bootstrapping... 2024-02-20T00:08:25.139 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 -v bootstrap --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-ip 172.21.15.84 && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-02-20T00:08:25.165 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:-- Logs begin at Mon 2024-02-19 23:58:49 UTC. -- 2024-02-20T00:08:25.405 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2024-02-20T00:08:26.386 INFO:teuthology.orchestra.run.smithi084.stderr:/bin/podman: 4.9.3-dev 2024-02-20T00:08:26.396 INFO:teuthology.orchestra.run.smithi084.stderr:Verifying podman|docker is present... 2024-02-20T00:08:26.397 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2024-02-20T00:08:26.419 INFO:teuthology.orchestra.run.smithi084.stderr:/bin/podman: 4.9.3-dev 2024-02-20T00:08:26.433 INFO:teuthology.orchestra.run.smithi084.stderr:Verifying lvm2 is present... 2024-02-20T00:08:26.433 INFO:teuthology.orchestra.run.smithi084.stderr:Verifying time synchronization is in place... 2024-02-20T00:08:26.433 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: systemctl is-enabled chrony.service 2024-02-20T00:08:26.440 INFO:teuthology.orchestra.run.smithi084.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-02-20T00:08:26.440 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: systemctl is-active chrony.service 2024-02-20T00:08:26.446 INFO:teuthology.orchestra.run.smithi084.stderr:systemctl: inactive 2024-02-20T00:08:26.447 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: systemctl is-enabled chronyd.service 2024-02-20T00:08:26.453 INFO:teuthology.orchestra.run.smithi084.stderr:systemctl: enabled 2024-02-20T00:08:26.453 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: systemctl is-active chronyd.service 2024-02-20T00:08:26.458 INFO:teuthology.orchestra.run.smithi084.stderr:systemctl: active 2024-02-20T00:08:26.459 INFO:teuthology.orchestra.run.smithi084.stderr:Unit chronyd.service is enabled and running 2024-02-20T00:08:26.459 INFO:teuthology.orchestra.run.smithi084.stderr:Repeating the final host check... 2024-02-20T00:08:26.459 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2024-02-20T00:08:26.480 INFO:teuthology.orchestra.run.smithi084.stderr:/bin/podman: 4.9.3-dev 2024-02-20T00:08:26.495 INFO:teuthology.orchestra.run.smithi084.stderr:podman|docker (/bin/podman) is present 2024-02-20T00:08:26.495 INFO:teuthology.orchestra.run.smithi084.stderr:systemctl is present 2024-02-20T00:08:26.495 INFO:teuthology.orchestra.run.smithi084.stderr:lvcreate is present 2024-02-20T00:08:26.495 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: systemctl is-enabled chrony.service 2024-02-20T00:08:26.501 INFO:teuthology.orchestra.run.smithi084.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-02-20T00:08:26.501 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: systemctl is-active chrony.service 2024-02-20T00:08:26.507 INFO:teuthology.orchestra.run.smithi084.stderr:systemctl: inactive 2024-02-20T00:08:26.507 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: systemctl is-enabled chronyd.service 2024-02-20T00:08:26.512 INFO:teuthology.orchestra.run.smithi084.stderr:systemctl: enabled 2024-02-20T00:08:26.513 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: systemctl is-active chronyd.service 2024-02-20T00:08:26.518 INFO:teuthology.orchestra.run.smithi084.stderr:systemctl: active 2024-02-20T00:08:26.519 INFO:teuthology.orchestra.run.smithi084.stderr:Unit chronyd.service is enabled and running 2024-02-20T00:08:26.519 INFO:teuthology.orchestra.run.smithi084.stderr:Host looks OK 2024-02-20T00:08:26.519 INFO:teuthology.orchestra.run.smithi084.stderr:Cluster fsid: 2a2e6c42-cf84-11ee-95bb-87774f69a715 2024-02-20T00:08:26.519 INFO:teuthology.orchestra.run.smithi084.stderr:Acquiring lock 140012450322920 on /run/cephadm/2a2e6c42-cf84-11ee-95bb-87774f69a715.lock 2024-02-20T00:08:26.519 INFO:teuthology.orchestra.run.smithi084.stderr:Lock 140012450322920 acquired on /run/cephadm/2a2e6c42-cf84-11ee-95bb-87774f69a715.lock 2024-02-20T00:08:26.520 INFO:teuthology.orchestra.run.smithi084.stderr:Verifying IP 172.21.15.84 port 3300 ... 2024-02-20T00:08:26.520 INFO:teuthology.orchestra.run.smithi084.stderr:Verifying IP 172.21.15.84 port 6789 ... 2024-02-20T00:08:26.520 INFO:teuthology.orchestra.run.smithi084.stderr:Base mon IP is 172.21.15.84, final addrv is [v2:172.21.15.84:3300,v1:172.21.15.84:6789] 2024-02-20T00:08:26.520 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: /sbin/ip route ls 2024-02-20T00:08:26.522 INFO:teuthology.orchestra.run.smithi084.stderr:/sbin/ip: default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.84 metric 100 2024-02-20T00:08:26.522 INFO:teuthology.orchestra.run.smithi084.stderr:/sbin/ip: 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.84 metric 100 2024-02-20T00:08:26.523 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: /sbin/ip -6 route ls 2024-02-20T00:08:26.525 INFO:teuthology.orchestra.run.smithi084.stderr:/sbin/ip: ::1 dev lo proto kernel metric 256 pref medium 2024-02-20T00:08:26.526 INFO:teuthology.orchestra.run.smithi084.stderr:/sbin/ip: fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-02-20T00:08:26.526 INFO:teuthology.orchestra.run.smithi084.stderr:/sbin/ip: default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1790sec hoplimit 64 pref medium 2024-02-20T00:08:26.526 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: /sbin/ip -6 addr ls 2024-02-20T00:08:26.528 INFO:teuthology.orchestra.run.smithi084.stderr:/sbin/ip: 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-02-20T00:08:26.528 INFO:teuthology.orchestra.run.smithi084.stderr:/sbin/ip: inet6 ::1/128 scope host 2024-02-20T00:08:26.528 INFO:teuthology.orchestra.run.smithi084.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-02-20T00:08:26.528 INFO:teuthology.orchestra.run.smithi084.stderr:/sbin/ip: 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-02-20T00:08:26.528 INFO:teuthology.orchestra.run.smithi084.stderr:/sbin/ip: inet6 fe80::ec4:7aff:fe8f:d751/64 scope link 2024-02-20T00:08:26.529 INFO:teuthology.orchestra.run.smithi084.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-02-20T00:08:26.529 INFO:teuthology.orchestra.run.smithi084.stderr:Mon IP 172.21.15.84 is in CIDR network 172.21.0.0/20 2024-02-20T00:08:26.529 INFO:teuthology.orchestra.run.smithi084.stderr:- internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-02-20T00:08:26.530 INFO:teuthology.orchestra.run.smithi084.stderr:Pulling container image docker.io/ceph/ceph:v16.2.4... 2024-02-20T00:08:26.531 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: /bin/podman pull docker.io/ceph/ceph:v16.2.4 2024-02-20T00:08:26.621 INFO:teuthology.orchestra.run.smithi084.stderr:/bin/podman: Trying to pull docker.io/ceph/ceph:v16.2.4... 2024-02-20T00:08:34.550 INFO:teuthology.orchestra.run.smithi084.stderr:/bin/podman: Getting image source signatures 2024-02-20T00:08:34.557 INFO:teuthology.orchestra.run.smithi084.stderr:/bin/podman: Copying blob sha256:551101690791055467b1840be133b2e6b029d263771bc6c46a96bb7acdacac39 2024-02-20T00:08:34.557 INFO:teuthology.orchestra.run.smithi084.stderr:/bin/podman: Copying blob sha256:7a0437f04f83f084b7ed68ad9c4a4947e12fc4e1b006b38129bac89114ec3621 2024-02-20T00:08:46.340 INFO:teuthology.orchestra.run.smithi084.stderr:/bin/podman: Copying config sha256:8d91d370c2b86c07de46146aba8d36718eaefa69b1880c77fa312fda6efd7d29 2024-02-20T00:08:46.508 INFO:teuthology.orchestra.run.smithi084.stderr:/bin/podman: Writing manifest to image destination 2024-02-20T00:08:47.234 INFO:teuthology.orchestra.run.smithi084.stderr:/bin/podman: 8d91d370c2b86c07de46146aba8d36718eaefa69b1880c77fa312fda6efd7d29 2024-02-20T00:08:47.242 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi084 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --version 2024-02-20T00:08:47.875 INFO:teuthology.orchestra.run.smithi084.stderr:ceph: ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable) 2024-02-20T00:08:48.454 INFO:teuthology.orchestra.run.smithi084.stderr:Ceph version: ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable) 2024-02-20T00:08:48.454 INFO:teuthology.orchestra.run.smithi084.stderr:Extracting ceph user uid/gid from container image... 2024-02-20T00:08:48.454 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi084 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 -c %u %g /var/lib/ceph 2024-02-20T00:08:49.028 INFO:teuthology.orchestra.run.smithi084.stderr:stat: 167 167 2024-02-20T00:08:49.588 INFO:teuthology.orchestra.run.smithi084.stderr:Creating initial keys... 2024-02-20T00:08:49.589 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi084 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2024-02-20T00:08:50.109 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-authtool: AQCR7dNlC+UNOhAAiAC/kzdu7LciGblbCegN3A== 2024-02-20T00:08:50.708 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi084 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2024-02-20T00:08:51.215 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-authtool: AQCT7dNlvqJ4BRAAt3/ofwwNpdCrZHNr+6SeLA== 2024-02-20T00:08:51.779 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi084 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2024-02-20T00:08:52.345 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-authtool: AQCU7dNlMd68DRAAFDxZxLsm+MiUA9LbUBpKPQ== 2024-02-20T00:08:52.828 INFO:teuthology.orchestra.run.smithi084.stderr:Creating initial monmap... 2024-02-20T00:08:52.828 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/monmaptool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi084 -e CEPH_USE_RANDOM_NONCE=1 -v /tmp/ceph-tmp5uyqhbp_:/tmp/monmap:z docker.io/ceph/ceph:v16.2.4 --create --clobber --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 --addv smithi084 [v2:172.21.15.84:3300,v1:172.21.15.84:6789] /tmp/monmap 2024-02-20T00:08:53.369 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: monmap file /tmp/monmap 2024-02-20T00:08:53.369 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: set fsid to 2a2e6c42-cf84-11ee-95bb-87774f69a715 2024-02-20T00:08:53.369 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-02-20T00:08:53.926 INFO:teuthology.orchestra.run.smithi084.stderr:monmaptool for smithi084 [v2:172.21.15.84:3300,v1:172.21.15.84:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-02-20T00:08:53.926 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/monmaptool: set fsid to 2a2e6c42-cf84-11ee-95bb-87774f69a715 2024-02-20T00:08:53.926 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-02-20T00:08:53.926 INFO:teuthology.orchestra.run.smithi084.stderr: 2024-02-20T00:08:53.926 INFO:teuthology.orchestra.run.smithi084.stderr:Creating mon... 2024-02-20T00:08:53.927 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-mon --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi084 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/2a2e6c42-cf84-11ee-95bb-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/2a2e6c42-cf84-11ee-95bb-87774f69a715/mon.smithi084:/var/lib/ceph/mon/ceph-smithi084:z -v /tmp/ceph-tmp8g2f_0qg:/tmp/keyring:z -v /tmp/ceph-tmp5uyqhbp_:/tmp/monmap:z docker.io/ceph/ceph:v16.2.4 --mkfs -i smithi084 --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -c /dev/null --monmap /tmp/monmap --keyring /tmp/keyring --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true 2024-02-20T00:08:54.461 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.383+0000 7f08f0113700 0 set uid:gid to 167:167 (ceph:ceph) 2024-02-20T00:08:54.461 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.383+0000 7f08f0113700 1 imported monmap: 2024-02-20T00:08:54.461 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: epoch 0 2024-02-20T00:08:54.461 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 2024-02-20T00:08:54.461 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: last_changed 2024-02-20T00:08:53.282221+0000 2024-02-20T00:08:54.462 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: created 2024-02-20T00:08:53.282221+0000 2024-02-20T00:08:54.462 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: min_mon_release 0 (unknown) 2024-02-20T00:08:54.462 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: election_strategy: 1 2024-02-20T00:08:54.462 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: 0: [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] mon.smithi084 2024-02-20T00:08:54.462 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: 2024-02-20T00:08:54.462 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.383+0000 7f08f0113700 0 /usr/bin/ceph-mon: set fsid to 2a2e6c42-cf84-11ee-95bb-87774f69a715 2024-02-20T00:08:54.462 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: RocksDB version: 6.8.1 2024-02-20T00:08:54.462 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: 2024-02-20T00:08:54.462 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-02-20T00:08:54.462 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Compile date May 13 2021 2024-02-20T00:08:54.462 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: DB SUMMARY 2024-02-20T00:08:54.462 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: 2024-02-20T00:08:54.462 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi084/store.db dir, Total Num: 0, files: 2024-02-20T00:08:54.462 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: 2024-02-20T00:08:54.463 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi084/store.db: 2024-02-20T00:08:54.463 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: 2024-02-20T00:08:54.463 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.error_if_exists: 0 2024-02-20T00:08:54.463 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.create_if_missing: 1 2024-02-20T00:08:54.463 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.paranoid_checks: 1 2024-02-20T00:08:54.463 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.env: 0x55edddf8a1c0 2024-02-20T00:08:54.463 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.fs: Posix File System 2024-02-20T00:08:54.463 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.info_log: 0x55eddeac4ec0 2024-02-20T00:08:54.463 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.max_file_opening_threads: 16 2024-02-20T00:08:54.463 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.statistics: (nil) 2024-02-20T00:08:54.463 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.use_fsync: 0 2024-02-20T00:08:54.463 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.max_log_file_size: 0 2024-02-20T00:08:54.463 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-20T00:08:54.463 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.log_file_time_to_roll: 0 2024-02-20T00:08:54.463 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.keep_log_file_num: 1000 2024-02-20T00:08:54.464 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.recycle_log_file_num: 0 2024-02-20T00:08:54.464 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.allow_fallocate: 1 2024-02-20T00:08:54.464 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.allow_mmap_reads: 0 2024-02-20T00:08:54.464 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.allow_mmap_writes: 0 2024-02-20T00:08:54.464 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.use_direct_reads: 0 2024-02-20T00:08:54.464 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-20T00:08:54.464 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.create_missing_column_families: 0 2024-02-20T00:08:54.464 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.db_log_dir: 2024-02-20T00:08:54.464 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi084/store.db 2024-02-20T00:08:54.464 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.table_cache_numshardbits: 6 2024-02-20T00:08:54.464 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.max_subcompactions: 1 2024-02-20T00:08:54.464 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.max_background_flushes: -1 2024-02-20T00:08:54.464 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.WAL_ttl_seconds: 0 2024-02-20T00:08:54.464 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.WAL_size_limit_MB: 0 2024-02-20T00:08:54.465 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-02-20T00:08:54.465 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-20T00:08:54.465 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.is_fd_close_on_exec: 1 2024-02-20T00:08:54.465 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.advise_random_on_open: 1 2024-02-20T00:08:54.465 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.db_write_buffer_size: 0 2024-02-20T00:08:54.465 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.write_buffer_manager: 0x55eddeacb530 2024-02-20T00:08:54.465 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-20T00:08:54.465 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-02-20T00:08:54.465 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-20T00:08:54.465 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.use_adaptive_mutex: 0 2024-02-20T00:08:54.465 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.rate_limiter: (nil) 2024-02-20T00:08:54.465 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-20T00:08:54.465 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.wal_recovery_mode: 2 2024-02-20T00:08:54.465 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.enable_thread_tracking: 0 2024-02-20T00:08:54.466 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.enable_pipelined_write: 0 2024-02-20T00:08:54.466 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.unordered_write: 0 2024-02-20T00:08:54.466 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-20T00:08:54.466 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-20T00:08:54.466 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-20T00:08:54.466 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-20T00:08:54.466 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.row_cache: None 2024-02-20T00:08:54.466 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.wal_filter: None 2024-02-20T00:08:54.466 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-20T00:08:54.466 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.allow_ingest_behind: 0 2024-02-20T00:08:54.466 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.preserve_deletes: 0 2024-02-20T00:08:54.466 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.two_write_queues: 0 2024-02-20T00:08:54.466 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.manual_wal_flush: 0 2024-02-20T00:08:54.466 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.atomic_flush: 0 2024-02-20T00:08:54.467 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-20T00:08:54.467 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.persist_stats_to_disk: 0 2024-02-20T00:08:54.467 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.write_dbid_to_manifest: 0 2024-02-20T00:08:54.467 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.log_readahead_size: 0 2024-02-20T00:08:54.467 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2024-02-20T00:08:54.467 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.max_background_jobs: 2 2024-02-20T00:08:54.467 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.max_background_compactions: -1 2024-02-20T00:08:54.467 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-20T00:08:54.467 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-20T00:08:54.467 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.delayed_write_rate : 16777216 2024-02-20T00:08:54.467 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.max_total_wal_size: 0 2024-02-20T00:08:54.467 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-20T00:08:54.467 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.stats_dump_period_sec: 600 2024-02-20T00:08:54.467 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.stats_persist_period_sec: 600 2024-02-20T00:08:54.468 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-20T00:08:54.468 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.max_open_files: -1 2024-02-20T00:08:54.468 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.bytes_per_sync: 0 2024-02-20T00:08:54.468 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.wal_bytes_per_sync: 0 2024-02-20T00:08:54.468 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.strict_bytes_per_sync: 0 2024-02-20T00:08:54.468 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.compaction_readahead_size: 0 2024-02-20T00:08:54.468 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Compression algorithms supported: 2024-02-20T00:08:54.468 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-20T00:08:54.468 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: kZSTD supported: 0 2024-02-20T00:08:54.468 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: kXpressCompression supported: 0 2024-02-20T00:08:54.468 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: kLZ4HCCompression supported: 1 2024-02-20T00:08:54.468 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: kLZ4Compression supported: 1 2024-02-20T00:08:54.468 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: kBZip2Compression supported: 0 2024-02-20T00:08:54.469 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: kZlibCompression supported: 1 2024-02-20T00:08:54.469 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: kSnappyCompression supported: 1 2024-02-20T00:08:54.469 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Fast CRC32 supported: Supported on x86 2024-02-20T00:08:54.469 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: [db_impl/db_impl_open.cc:273] Creating manifest 1 2024-02-20T00:08:54.469 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: 2024-02-20T00:08:54.469 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi084/store.db/MANIFEST-000001 2024-02-20T00:08:54.469 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: 2024-02-20T00:08:54.469 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2024-02-20T00:08:54.469 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: 2024-02-20T00:08:54.469 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-20T00:08:54.469 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.merge_operator: 2024-02-20T00:08:54.469 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.compaction_filter: None 2024-02-20T00:08:54.469 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.compaction_filter_factory: None 2024-02-20T00:08:54.469 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.memtable_factory: SkipListFactory 2024-02-20T00:08:54.469 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.table_factory: BlockBasedTable 2024-02-20T00:08:54.469 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55edde9ddcb8) 2024-02-20T00:08:54.470 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks: 1 2024-02-20T00:08:54.470 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks_with_high_priority: 0 2024-02-20T00:08:54.470 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-20T00:08:54.470 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: pin_top_level_index_and_filter: 1 2024-02-20T00:08:54.470 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: index_type: 0 2024-02-20T00:08:54.470 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: data_block_index_type: 0 2024-02-20T00:08:54.470 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: index_shortening: 1 2024-02-20T00:08:54.470 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: data_block_hash_table_util_ratio: 0.750000 2024-02-20T00:08:54.470 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: hash_index_allow_collision: 1 2024-02-20T00:08:54.470 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: checksum: 1 2024-02-20T00:08:54.470 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: no_block_cache: 0 2024-02-20T00:08:54.470 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: block_cache: 0x55eddea12d10 2024-02-20T00:08:54.470 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: block_cache_name: BinnedLRUCache 2024-02-20T00:08:54.470 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: block_cache_options: 2024-02-20T00:08:54.471 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: capacity : 536870912 2024-02-20T00:08:54.471 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: num_shard_bits : 4 2024-02-20T00:08:54.471 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: strict_capacity_limit : 0 2024-02-20T00:08:54.471 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: high_pri_pool_ratio: 0.000 2024-02-20T00:08:54.471 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: block_cache_compressed: (nil) 2024-02-20T00:08:54.471 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: persistent_cache: (nil) 2024-02-20T00:08:54.471 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: block_size: 4096 2024-02-20T00:08:54.471 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: block_size_deviation: 10 2024-02-20T00:08:54.471 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: block_restart_interval: 16 2024-02-20T00:08:54.471 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: index_block_restart_interval: 1 2024-02-20T00:08:54.471 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: metadata_block_size: 4096 2024-02-20T00:08:54.471 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: partition_filters: 0 2024-02-20T00:08:54.471 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: use_delta_encoding: 1 2024-02-20T00:08:54.471 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: filter_policy: rocksdb.BuiltinBloomFilter 2024-02-20T00:08:54.471 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: whole_key_filtering: 1 2024-02-20T00:08:54.472 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: verify_compression: 0 2024-02-20T00:08:54.472 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: read_amp_bytes_per_bit: 0 2024-02-20T00:08:54.472 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: format_version: 2 2024-02-20T00:08:54.472 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: enable_index_compression: 1 2024-02-20T00:08:54.472 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: block_align: 0 2024-02-20T00:08:54.472 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: 2024-02-20T00:08:54.472 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.write_buffer_size: 33554432 2024-02-20T00:08:54.472 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.max_write_buffer_number: 2 2024-02-20T00:08:54.472 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.compression: NoCompression 2024-02-20T00:08:54.472 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.bottommost_compression: Disabled 2024-02-20T00:08:54.472 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.prefix_extractor: nullptr 2024-02-20T00:08:54.472 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-20T00:08:54.472 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.num_levels: 7 2024-02-20T00:08:54.473 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-20T00:08:54.473 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-20T00:08:54.473 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-02-20T00:08:54.473 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-20T00:08:54.473 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-20T00:08:54.473 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-20T00:08:54.473 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-20T00:08:54.473 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-20T00:08:54.473 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-20T00:08:54.473 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.compression_opts.window_bits: -14 2024-02-20T00:08:54.473 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.compression_opts.level: 32767 2024-02-20T00:08:54.473 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.compression_opts.strategy: 0 2024-02-20T00:08:54.473 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-20T00:08:54.473 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-20T00:08:54.474 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.compression_opts.enabled: false 2024-02-20T00:08:54.474 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-20T00:08:54.474 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-20T00:08:54.474 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-20T00:08:54.474 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.target_file_size_base: 67108864 2024-02-20T00:08:54.474 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.target_file_size_multiplier: 1 2024-02-20T00:08:54.474 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-20T00:08:54.474 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-20T00:08:54.474 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-20T00:08:54.474 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-20T00:08:54.474 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-20T00:08:54.474 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-20T00:08:54.474 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-20T00:08:54.474 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-20T00:08:54.475 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-20T00:08:54.475 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-20T00:08:54.475 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-20T00:08:54.475 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-20T00:08:54.475 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.arena_block_size: 4194304 2024-02-20T00:08:54.475 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-20T00:08:54.475 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-20T00:08:54.475 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-02-20T00:08:54.475 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.disable_auto_compactions: 0 2024-02-20T00:08:54.475 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-20T00:08:54.475 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-20T00:08:54.475 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-20T00:08:54.475 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-20T00:08:54.475 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-20T00:08:54.476 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-20T00:08:54.476 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-20T00:08:54.476 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-20T00:08:54.476 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-20T00:08:54.476 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-20T00:08:54.476 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.table_properties_collectors: 2024-02-20T00:08:54.476 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.inplace_update_support: 0 2024-02-20T00:08:54.476 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.inplace_update_num_locks: 10000 2024-02-20T00:08:54.476 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-20T00:08:54.476 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-20T00:08:54.476 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.memtable_huge_page_size: 0 2024-02-20T00:08:54.476 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.bloom_locality: 0 2024-02-20T00:08:54.476 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.max_successive_merges: 0 2024-02-20T00:08:54.476 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.optimize_filters_for_hits: 0 2024-02-20T00:08:54.476 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.paranoid_file_checks: 0 2024-02-20T00:08:54.477 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.force_consistency_checks: 0 2024-02-20T00:08:54.477 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.report_bg_io_stats: 0 2024-02-20T00:08:54.477 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.ttl: 2592000 2024-02-20T00:08:54.477 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.384+0000 7f08f0113700 4 rocksdb: Options.periodic_compaction_seconds: 0 2024-02-20T00:08:54.477 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.385+0000 7f08f0113700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi084/store.db/MANIFEST-000001 succeeded,manifest_file_number is 1, next_file_number is 3, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-02-20T00:08:54.477 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: 2024-02-20T00:08:54.477 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.385+0000 7f08f0113700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 0 2024-02-20T00:08:54.477 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: 2024-02-20T00:08:54.477 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.386+0000 7f08f0113700 4 rocksdb: DB pointer 0x55eddead7800 2024-02-20T00:08:54.477 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.386+0000 7f08d8fac700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2024-02-20T00:08:54.477 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.386+0000 7f08d8fac700 4 rocksdb: [db_impl/db_impl.cc:851] 2024-02-20T00:08:54.477 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: ** DB Stats ** 2024-02-20T00:08:54.477 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2024-02-20T00:08:54.478 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-20T00:08:54.478 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-20T00:08:54.478 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-20T00:08:54.478 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-20T00:08:54.478 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-02-20T00:08:54.478 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-20T00:08:54.478 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: 2024-02-20T00:08:54.478 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-02-20T00:08:54.478 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-20T00:08:54.478 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-20T00:08:54.478 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-20T00:08:54.478 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-20T00:08:54.478 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: 2024-02-20T00:08:54.478 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-02-20T00:08:54.479 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-20T00:08:54.479 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-20T00:08:54.479 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2024-02-20T00:08:54.479 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2024-02-20T00:08:54.479 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-20T00:08:54.479 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2024-02-20T00:08:54.479 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2024-02-20T00:08:54.479 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2024-02-20T00:08:54.479 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-20T00:08:54.479 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-20T00:08:54.479 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-20T00:08:54.479 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: 2024-02-20T00:08:54.479 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2024-02-20T00:08:54.479 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: 2024-02-20T00:08:54.479 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-02-20T00:08:54.479 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-20T00:08:54.480 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-20T00:08:54.480 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-20T00:08:54.480 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-20T00:08:54.480 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: 2024-02-20T00:08:54.480 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-02-20T00:08:54.480 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-20T00:08:54.480 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-20T00:08:54.480 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2024-02-20T00:08:54.480 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2024-02-20T00:08:54.480 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-20T00:08:54.480 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2024-02-20T00:08:54.480 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2024-02-20T00:08:54.480 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2024-02-20T00:08:54.480 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-20T00:08:54.481 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-20T00:08:54.481 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-20T00:08:54.481 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: 2024-02-20T00:08:54.481 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2024-02-20T00:08:54.481 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: 2024-02-20T00:08:54.481 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.387+0000 7f08f0113700 4 rocksdb: [db_impl/db_impl.cc:397] Shutdown: canceling all background work 2024-02-20T00:08:54.481 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.387+0000 7f08f0113700 4 rocksdb: [db_impl/db_impl.cc:573] Shutdown complete 2024-02-20T00:08:54.481 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-02-20T00:08:54.387+0000 7f08f0113700 0 /usr/bin/ceph-mon: created monfs at /var/lib/ceph/mon/ceph-smithi084 for mon.smithi084 2024-02-20T00:08:55.035 INFO:teuthology.orchestra.run.smithi084.stderr:create mon.smithi084 on 2024-02-20T00:08:55.035 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: install -d -m0770 -o 167 -g 167 /var/run/ceph/2a2e6c42-cf84-11ee-95bb-87774f69a715 2024-02-20T00:08:55.040 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: systemctl disable ceph.target 2024-02-20T00:08:55.047 INFO:teuthology.orchestra.run.smithi084.stderr:systemctl: Removed /etc/systemd/system/multi-user.target.wants/ceph.target. 2024-02-20T00:08:55.129 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: systemctl enable ceph.target 2024-02-20T00:08:55.136 INFO:teuthology.orchestra.run.smithi084.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-02-20T00:08:55.214 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: systemctl start ceph.target 2024-02-20T00:08:55.221 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: systemctl enable ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715.target 2024-02-20T00:08:55.226 INFO:teuthology.orchestra.run.smithi084.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715.target → /etc/systemd/system/ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715.target. 2024-02-20T00:08:55.226 INFO:teuthology.orchestra.run.smithi084.stderr:systemctl: Created symlink /etc/systemd/system/ceph.target.wants/ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715.target → /etc/systemd/system/ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715.target. 2024-02-20T00:08:55.303 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: systemctl start ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715.target 2024-02-20T00:08:55.311 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: systemctl daemon-reload 2024-02-20T00:08:55.394 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: systemctl stop ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715@mon.smithi084 2024-02-20T00:08:55.424 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: systemctl reset-failed ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715@mon.smithi084 2024-02-20T00:08:55.430 INFO:teuthology.orchestra.run.smithi084.stderr:systemctl: Failed to reset failed state of unit ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715@mon.smithi084.service: Unit ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715@mon.smithi084.service not loaded. 2024-02-20T00:08:55.430 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: systemctl enable ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715@mon.smithi084 2024-02-20T00:08:55.439 INFO:teuthology.orchestra.run.smithi084.stderr:systemctl: Created symlink /etc/systemd/system/ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715.target.wants/ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715@mon.smithi084.service → /etc/systemd/system/ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715@.service. 2024-02-20T00:08:55.520 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: systemctl start ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715@mon.smithi084 2024-02-20T00:08:55.560 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:08:55 smithi084 systemd[1]: Starting Ceph mon.smithi084 for 2a2e6c42-cf84-11ee-95bb-87774f69a715... 2024-02-20T00:08:56.010 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: systemctl is-enabled firewalld.service 2024-02-20T00:08:56.018 INFO:teuthology.orchestra.run.smithi084.stderr:systemctl: disabled 2024-02-20T00:08:56.018 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: systemctl is-active firewalld.service 2024-02-20T00:08:56.025 INFO:teuthology.orchestra.run.smithi084.stderr:systemctl: inactive 2024-02-20T00:08:56.026 INFO:teuthology.orchestra.run.smithi084.stderr:firewalld.service is not enabled 2024-02-20T00:08:56.026 INFO:teuthology.orchestra.run.smithi084.stderr:Not possible to enable service . firewalld.service is not available 2024-02-20T00:08:56.026 INFO:teuthology.orchestra.run.smithi084.stderr:Not possible to open ports <[]>. firewalld.service is not available 2024-02-20T00:08:56.026 INFO:teuthology.orchestra.run.smithi084.stderr:Waiting for mon to start... 2024-02-20T00:08:56.026 INFO:teuthology.orchestra.run.smithi084.stderr:Waiting for mon... 2024-02-20T00:08:56.026 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi084 -e CEPH_USE_RANDOM_NONCE=1 -v /var/lib/ceph/2a2e6c42-cf84-11ee-95bb-87774f69a715/mon.smithi084:/var/lib/ceph/mon/ceph-smithi084:z -v /tmp/ceph-tmp3ot6j0xn:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp4xryone4:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status 2024-02-20T00:08:56.258 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:08:56 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[37700]: cluster 2024-02-20T00:08:55.999171+0000 mon.smithi084 (mon.0) 0 : [INF] mkfs 2a2e6c42-cf84-11ee-95bb-87774f69a715 2024-02-20T00:08:56.258 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:08:56 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[37700]: cluster 2024-02-20T00:08:55.982370+0000 mon.smithi084 (mon.0) 1 : cluster [INF] mon.smithi084 is new leader, mons smithi084 in quorum (ranks 0) 2024-02-20T00:08:56.258 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:08:56 smithi084 bash[37685]: 78c7df2988728082b5ea6c8c2af97e009760f57fc837d6bc86ebc9c2e0035f37 2024-02-20T00:08:56.259 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:08:56 smithi084 systemd[1]: Started Ceph mon.smithi084 for 2a2e6c42-cf84-11ee-95bb-87774f69a715. 2024-02-20T00:08:56.605 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:08:56.600+0000 7f0f28e6c700 1 Processor -- start 2024-02-20T00:08:56.605 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:08:56.600+0000 7f0f28e6c700 1 -- start start 2024-02-20T00:08:56.605 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:08:56.601+0000 7f0f28e6c700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0f24070430 0x7f0f240fea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-20T00:08:56.605 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:08:56.601+0000 7f0f28e6c700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f240fef40 con 0x7f0f24070430 2024-02-20T00:08:56.605 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:08:56.601+0000 7f0f2259c700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0f24070430 0x7f0f240fea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:08:56.606 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:08:56.601+0000 7f0f2259c700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0f24070430 0x7f0f240fea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:44112/0 (socket says 172.21.15.84:44112) 2024-02-20T00:08:56.606 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:08:56.601+0000 7f0f2259c700 1 -- 172.21.15.84:0/664317705 learned_addr learned my addr 172.21.15.84:0/664317705 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-02-20T00:08:56.606 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:08:56.601+0000 7f0f2259c700 1 -- 172.21.15.84:0/664317705 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f240ff080 con 0x7f0f24070430 2024-02-20T00:08:56.606 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:08:56.601+0000 7f0f2259c700 1 --2- 172.21.15.84:0/664317705 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0f24070430 0x7f0f240fea50 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f0f1800b5f0 tx=0x7f0f18006ee0).ready entity=mon.0 client_cookie=afb57e802a34d26c server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:08:56.606 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:08:56.602+0000 7f0f2159a700 1 -- 172.21.15.84:0/664317705 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0f1800d600 con 0x7f0f24070430 2024-02-20T00:08:56.606 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:08:56.602+0000 7f0f2159a700 1 -- 172.21.15.84:0/664317705 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f1800d760 con 0x7f0f24070430 2024-02-20T00:08:56.606 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:08:56.602+0000 7f0f2159a700 1 -- 172.21.15.84:0/664317705 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0f1800da30 con 0x7f0f24070430 2024-02-20T00:08:56.606 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:08:56.602+0000 7f0f28e6c700 1 -- 172.21.15.84:0/664317705 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0f24070430 msgr2=0x7f0f240fea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:08:56.606 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:08:56.602+0000 7f0f28e6c700 1 --2- 172.21.15.84:0/664317705 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0f24070430 0x7f0f240fea50 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f0f1800b5f0 tx=0x7f0f18006ee0).stop 2024-02-20T00:08:56.606 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:08:56.602+0000 7f0f28e6c700 1 -- 172.21.15.84:0/664317705 shutdown_connections 2024-02-20T00:08:56.606 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:08:56.602+0000 7f0f28e6c700 1 --2- 172.21.15.84:0/664317705 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0f24070430 0x7f0f240fea50 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:08:56.606 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:08:56.602+0000 7f0f28e6c700 1 -- 172.21.15.84:0/664317705 >> 172.21.15.84:0/664317705 conn(0x7f0f240f6530 msgr2=0x7f0f240f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:08:56.606 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:08:56.602+0000 7f0f28e6c700 1 -- 172.21.15.84:0/664317705 shutdown_connections 2024-02-20T00:08:56.606 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:08:56.602+0000 7f0f28e6c700 1 -- 172.21.15.84:0/664317705 wait complete. 2024-02-20T00:08:56.606 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:08:56.603+0000 7f0f28e6c700 1 Processor -- start 2024-02-20T00:08:56.607 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:08:56.603+0000 7f0f28e6c700 1 -- start start 2024-02-20T00:08:56.607 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:08:56.603+0000 7f0f28e6c700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0f24070430 0x7f0f2410a330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:08:56.607 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:08:56.603+0000 7f0f28e6c700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f2410a820 con 0x7f0f24070430 2024-02-20T00:08:56.607 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:08:56.603+0000 7f0f2259c700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0f24070430 0x7f0f2410a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:08:56.607 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:08:56.603+0000 7f0f2259c700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0f24070430 0x7f0f2410a330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:44118/0 (socket says 172.21.15.84:44118) 2024-02-20T00:08:56.607 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:08:56.603+0000 7f0f2259c700 1 -- 172.21.15.84:0/2035633100 learned_addr learned my addr 172.21.15.84:0/2035633100 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-02-20T00:08:56.607 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:08:56.603+0000 7f0f2259c700 1 -- 172.21.15.84:0/2035633100 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f1800b200 con 0x7f0f24070430 2024-02-20T00:08:56.607 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:08:56.603+0000 7f0f2259c700 1 --2- 172.21.15.84:0/2035633100 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0f24070430 0x7f0f2410a330 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f0f1800c010 tx=0x7f0f18016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:08:56.608 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:08:56.604+0000 7f0f12ffd700 1 -- 172.21.15.84:0/2035633100 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0f18011cc0 con 0x7f0f24070430 2024-02-20T00:08:56.608 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:08:56.604+0000 7f0f12ffd700 1 -- 172.21.15.84:0/2035633100 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f18011e20 con 0x7f0f24070430 2024-02-20T00:08:56.608 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:08:56.604+0000 7f0f28e6c700 1 -- 172.21.15.84:0/2035633100 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f2410aa20 con 0x7f0f24070430 2024-02-20T00:08:56.608 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:08:56.604+0000 7f0f12ffd700 1 -- 172.21.15.84:0/2035633100 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0f1801d590 con 0x7f0f24070430 2024-02-20T00:08:56.608 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:08:56.604+0000 7f0f28e6c700 1 -- 172.21.15.84:0/2035633100 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f2410aeb0 con 0x7f0f24070430 2024-02-20T00:08:56.608 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:08:56.604+0000 7f0f12ffd700 1 -- 172.21.15.84:0/2035633100 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f0f1800f070 con 0x7f0f24070430 2024-02-20T00:08:56.608 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:08:56.604+0000 7f0f12ffd700 1 -- 172.21.15.84:0/2035633100 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f0f18027460 con 0x7f0f24070430 2024-02-20T00:08:56.608 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:08:56.605+0000 7f0f28e6c700 1 -- 172.21.15.84:0/2035633100 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f24045ad0 con 0x7f0f24070430 2024-02-20T00:08:56.608 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:08:56.607+0000 7f0f12ffd700 1 -- 172.21.15.84:0/2035633100 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f0f18016070 con 0x7f0f24070430 2024-02-20T00:08:56.677 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:08:56.673+0000 7f0f28e6c700 1 -- 172.21.15.84:0/2035633100 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7f0f2410c3c0 con 0x7f0f24070430 2024-02-20T00:08:56.677 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:08:56.673+0000 7f0f12ffd700 1 -- 172.21.15.84:0/2035633100 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+325 (secure 0 0 0) 0x7f0f18036070 con 0x7f0f24070430 2024-02-20T00:08:56.677 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: cluster: 2024-02-20T00:08:56.677 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: id: 2a2e6c42-cf84-11ee-95bb-87774f69a715 2024-02-20T00:08:56.677 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: health: HEALTH_OK 2024-02-20T00:08:56.678 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:08:56.678 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: services: 2024-02-20T00:08:56.678 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: mon: 1 daemons, quorum smithi084 (age 0.676188s) 2024-02-20T00:08:56.678 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: mgr: no daemons active 2024-02-20T00:08:56.678 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: osd: 0 osds: 0 up, 0 in 2024-02-20T00:08:56.678 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:08:56.678 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: data: 2024-02-20T00:08:56.678 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: pools: 0 pools, 0 pgs 2024-02-20T00:08:56.678 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: objects: 0 objects, 0 B 2024-02-20T00:08:56.678 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: usage: 0 B used, 0 B / 0 B avail 2024-02-20T00:08:56.678 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: pgs: 2024-02-20T00:08:56.678 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:08:56.678 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:08:56.674+0000 7f0f28e6c700 1 -- 172.21.15.84:0/2035633100 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0f24070430 msgr2=0x7f0f2410a330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:08:56.678 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:08:56.674+0000 7f0f28e6c700 1 --2- 172.21.15.84:0/2035633100 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0f24070430 0x7f0f2410a330 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f0f1800c010 tx=0x7f0f18016000).stop 2024-02-20T00:08:56.678 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:08:56.674+0000 7f0f28e6c700 1 -- 172.21.15.84:0/2035633100 shutdown_connections 2024-02-20T00:08:56.679 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:08:56.674+0000 7f0f28e6c700 1 --2- 172.21.15.84:0/2035633100 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0f24070430 0x7f0f2410a330 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:08:56.679 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:08:56.674+0000 7f0f28e6c700 1 -- 172.21.15.84:0/2035633100 >> 172.21.15.84:0/2035633100 conn(0x7f0f240f6530 msgr2=0x7f0f240f81a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:08:56.679 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:08:56.674+0000 7f0f28e6c700 1 -- 172.21.15.84:0/2035633100 shutdown_connections 2024-02-20T00:08:56.679 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:08:56.674+0000 7f0f28e6c700 1 -- 172.21.15.84:0/2035633100 wait complete. 2024-02-20T00:08:57.281 INFO:teuthology.orchestra.run.smithi084.stderr:mon is available 2024-02-20T00:08:57.282 INFO:teuthology.orchestra.run.smithi084.stderr:Assimilating anything we can from ceph.conf... 2024-02-20T00:08:57.282 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi084 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/2a2e6c42-cf84-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp3ot6j0xn:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp4xryone4:/etc/ceph/ceph.conf:z -v /var/lib/ceph/2a2e6c42-cf84-11ee-95bb-87774f69a715/mon.smithi084:/var/lib/ceph/mon/ceph-smithi084:z docker.io/ceph/ceph:v16.2.4 config assimilate-conf -i /var/lib/ceph/mon/ceph-smithi084/config 2024-02-20T00:08:57.284 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:08:57 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[37700]: cluster 2024-02-20T00:08:55.997937+0000 mon.smithi084 (mon.0) 2 : cluster [INF] mon.smithi084 is new leader, mons smithi084 in quorum (ranks 0) 2024-02-20T00:08:57.284 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:08:57 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[37700]: cluster 2024-02-20T00:08:55.998104+0000 mon.smithi084 (mon.0) 3 : cluster [DBG] monmap e1: 1 mons at {smithi084=[v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0]} 2024-02-20T00:08:57.284 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:08:57 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[37700]: cluster 2024-02-20T00:08:56.001014+0000 mon.smithi084 (mon.0) 4 : cluster [DBG] fsmap 2024-02-20T00:08:57.284 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:08:57 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[37700]: cluster 2024-02-20T00:08:56.003793+0000 mon.smithi084 (mon.0) 5 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2024-02-20T00:08:57.284 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:08:57 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[37700]: cluster 2024-02-20T00:08:56.004173+0000 mon.smithi084 (mon.0) 6 : cluster [DBG] mgrmap e1: no daemons active 2024-02-20T00:08:57.284 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:08:57 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[37700]: audit 2024-02-20T00:08:56.674080+0000 mon.smithi084 (mon.0) 7 : audit [DBG] from='client.? 172.21.15.84:0/2035633100' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-02-20T00:08:57.785 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:08:57.782+0000 7f1dc06f6700 1 Processor -- start 2024-02-20T00:08:57.785 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:08:57.782+0000 7f1dc06f6700 1 -- start start 2024-02-20T00:08:57.785 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:08:57.782+0000 7f1dc06f6700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1db8070430 0x7f1db80fea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-20T00:08:57.785 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:08:57.782+0000 7f1dc06f6700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1db80fef40 con 0x7f1db8070430 2024-02-20T00:08:57.785 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:08:57.783+0000 7f1dbe492700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1db8070430 0x7f1db80fea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:08:57.786 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:08:57.783+0000 7f1dbe492700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1db8070430 0x7f1db80fea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:44132/0 (socket says 172.21.15.84:44132) 2024-02-20T00:08:57.786 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:08:57.783+0000 7f1dbe492700 1 -- 172.21.15.84:0/1483326449 learned_addr learned my addr 172.21.15.84:0/1483326449 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-02-20T00:08:57.786 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:08:57.783+0000 7f1dbe492700 1 -- 172.21.15.84:0/1483326449 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1db80ff080 con 0x7f1db8070430 2024-02-20T00:08:57.786 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:08:57.783+0000 7f1dbe492700 1 --2- 172.21.15.84:0/1483326449 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1db8070430 0x7f1db80fea50 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7f1da800b5f0 tx=0x7f1da8006ee0).ready entity=mon.0 client_cookie=fee002e6ce40e359 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:08:57.786 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:08:57.783+0000 7f1dbd490700 1 -- 172.21.15.84:0/1483326449 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1da800d600 con 0x7f1db8070430 2024-02-20T00:08:57.786 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:08:57.783+0000 7f1dbd490700 1 -- 172.21.15.84:0/1483326449 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1da800d760 con 0x7f1db8070430 2024-02-20T00:08:57.786 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:08:57.783+0000 7f1dbd490700 1 -- 172.21.15.84:0/1483326449 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1da800da30 con 0x7f1db8070430 2024-02-20T00:08:57.788 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:08:57.784+0000 7f1dc06f6700 1 -- 172.21.15.84:0/1483326449 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1db8070430 msgr2=0x7f1db80fea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:08:57.789 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:08:57.784+0000 7f1dc06f6700 1 --2- 172.21.15.84:0/1483326449 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1db8070430 0x7f1db80fea50 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7f1da800b5f0 tx=0x7f1da8006ee0).stop 2024-02-20T00:08:57.789 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:08:57.784+0000 7f1dc06f6700 1 -- 172.21.15.84:0/1483326449 shutdown_connections 2024-02-20T00:08:57.789 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:08:57.784+0000 7f1dc06f6700 1 --2- 172.21.15.84:0/1483326449 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1db8070430 0x7f1db80fea50 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:08:57.789 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:08:57.784+0000 7f1dc06f6700 1 -- 172.21.15.84:0/1483326449 >> 172.21.15.84:0/1483326449 conn(0x7f1db80f6530 msgr2=0x7f1db80f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:08:57.789 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:08:57.784+0000 7f1dc06f6700 1 -- 172.21.15.84:0/1483326449 shutdown_connections 2024-02-20T00:08:57.789 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:08:57.784+0000 7f1dc06f6700 1 -- 172.21.15.84:0/1483326449 wait complete. 2024-02-20T00:08:57.789 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:08:57.785+0000 7f1dc06f6700 1 Processor -- start 2024-02-20T00:08:57.789 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:08:57.785+0000 7f1dc06f6700 1 -- start start 2024-02-20T00:08:57.789 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:08:57.785+0000 7f1dc06f6700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1db8070430 0x7f1db80fc4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:08:57.789 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:08:57.785+0000 7f1dc06f6700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1db80fa950 con 0x7f1db8070430 2024-02-20T00:08:57.789 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:08:57.785+0000 7f1dbe492700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1db8070430 0x7f1db80fc4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:08:57.789 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:08:57.785+0000 7f1dbe492700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1db8070430 0x7f1db80fc4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:44144/0 (socket says 172.21.15.84:44144) 2024-02-20T00:08:57.789 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:08:57.785+0000 7f1dbe492700 1 -- 172.21.15.84:0/178197532 learned_addr learned my addr 172.21.15.84:0/178197532 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-02-20T00:08:57.789 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:08:57.785+0000 7f1dbe492700 1 -- 172.21.15.84:0/178197532 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1da800b200 con 0x7f1db8070430 2024-02-20T00:08:57.789 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:08:57.785+0000 7f1dbe492700 1 --2- 172.21.15.84:0/178197532 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1db8070430 0x7f1db80fc4d0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7f1da8006c70 tx=0x7f1da8016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:08:57.789 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:08:57.785+0000 7f1da6ffd700 1 -- 172.21.15.84:0/178197532 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1da8011b40 con 0x7f1db8070430 2024-02-20T00:08:57.790 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:08:57.785+0000 7f1da6ffd700 1 -- 172.21.15.84:0/178197532 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1da8011ca0 con 0x7f1db8070430 2024-02-20T00:08:57.790 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:08:57.786+0000 7f1dc06f6700 1 -- 172.21.15.84:0/178197532 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1db80faaf0 con 0x7f1db8070430 2024-02-20T00:08:57.790 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:08:57.786+0000 7f1da6ffd700 1 -- 172.21.15.84:0/178197532 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1da801d430 con 0x7f1db8070430 2024-02-20T00:08:57.790 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:08:57.786+0000 7f1dc06f6700 1 -- 172.21.15.84:0/178197532 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1db80faf80 con 0x7f1db8070430 2024-02-20T00:08:57.790 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:08:57.786+0000 7f1da6ffd700 1 -- 172.21.15.84:0/178197532 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f1da800f070 con 0x7f1db8070430 2024-02-20T00:08:57.790 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:08:57.786+0000 7f1da6ffd700 1 -- 172.21.15.84:0/178197532 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f1da8027460 con 0x7f1db8070430 2024-02-20T00:08:57.790 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:08:57.787+0000 7f1dc06f6700 1 -- 172.21.15.84:0/178197532 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1db8045ad0 con 0x7f1db8070430 2024-02-20T00:08:57.790 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:08:57.789+0000 7f1da6ffd700 1 -- 172.21.15.84:0/178197532 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f1da8016070 con 0x7f1db8070430 2024-02-20T00:08:57.854 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:08:57.853+0000 7f1dc06f6700 1 -- 172.21.15.84:0/178197532 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f1db80008d0 con 0x7f1db8070430 2024-02-20T00:08:57.860 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:08:57.856+0000 7f1da6ffd700 1 -- 172.21.15.84:0/178197532 <== mon.0 v2:172.21.15.84:3300/0 7 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f1da801d5e0 con 0x7f1db8070430 2024-02-20T00:08:57.860 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:08:57.856+0000 7f1da6ffd700 1 -- 172.21.15.84:0/178197532 <== mon.0 v2:172.21.15.84:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+489 (secure 0 0 0) 0x7f1da8036020 con 0x7f1db8070430 2024-02-20T00:08:57.860 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:08:57.860 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: [global] 2024-02-20T00:08:57.860 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: fsid = 2a2e6c42-cf84-11ee-95bb-87774f69a715 2024-02-20T00:08:57.860 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: log_to_journald = false 2024-02-20T00:08:57.860 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: mon_host = [v2:172.21.15.84:3300,v1:172.21.15.84:6789] 2024-02-20T00:08:57.860 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-02-20T00:08:57.860 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-02-20T00:08:57.861 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-02-20T00:08:57.861 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-02-20T00:08:57.861 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:08:57.861 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: [mgr] 2024-02-20T00:08:57.861 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-02-20T00:08:57.861 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:08:57.861 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: [osd] 2024-02-20T00:08:57.861 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: mon_osd_backfillfull_ratio = 0.85 2024-02-20T00:08:57.861 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: mon_osd_full_ratio = 0.9 2024-02-20T00:08:57.861 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: mon_osd_nearfull_ratio = 0.8 2024-02-20T00:08:57.861 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-02-20T00:08:57.861 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: osd_memory_target_autotune = true 2024-02-20T00:08:57.861 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-02-20T00:08:57.861 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:08:57.857+0000 7f1dc06f6700 1 -- 172.21.15.84:0/178197532 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1db8070430 msgr2=0x7f1db80fc4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:08:57.861 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:08:57.857+0000 7f1dc06f6700 1 --2- 172.21.15.84:0/178197532 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1db8070430 0x7f1db80fc4d0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7f1da8006c70 tx=0x7f1da8016000).stop 2024-02-20T00:08:57.861 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:08:57.857+0000 7f1dc06f6700 1 -- 172.21.15.84:0/178197532 shutdown_connections 2024-02-20T00:08:57.861 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:08:57.857+0000 7f1dc06f6700 1 --2- 172.21.15.84:0/178197532 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1db8070430 0x7f1db80fc4d0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:08:57.862 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:08:57.857+0000 7f1dc06f6700 1 -- 172.21.15.84:0/178197532 >> 172.21.15.84:0/178197532 conn(0x7f1db80f6530 msgr2=0x7f1db80f81e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:08:57.862 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:08:57.857+0000 7f1dc06f6700 1 -- 172.21.15.84:0/178197532 shutdown_connections 2024-02-20T00:08:57.862 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:08:57.857+0000 7f1dc06f6700 1 -- 172.21.15.84:0/178197532 wait complete. 2024-02-20T00:08:58.168 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:08:58 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[37700]: audit 2024-02-20T00:08:57.853928+0000 mon.smithi084 (mon.0) 8 : audit [INF] from='client.? 172.21.15.84:0/178197532' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-02-20T00:08:58.169 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:08:58 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[37700]: audit 2024-02-20T00:08:57.856913+0000 mon.smithi084 (mon.0) 9 : audit [INF] from='client.? 172.21.15.84:0/178197532' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-02-20T00:08:58.430 INFO:teuthology.orchestra.run.smithi084.stderr:Generating new minimal ceph.conf... 2024-02-20T00:08:58.430 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi084 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/2a2e6c42-cf84-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp3ot6j0xn:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp4xryone4:/etc/ceph/ceph.conf:z -v /var/lib/ceph/2a2e6c42-cf84-11ee-95bb-87774f69a715/mon.smithi084:/var/lib/ceph/mon/ceph-smithi084:z docker.io/ceph/ceph:v16.2.4 config generate-minimal-conf -o /var/lib/ceph/mon/ceph-smithi084/config 2024-02-20T00:08:58.920 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:08:58.918+0000 7f519e065700 1 Processor -- start 2024-02-20T00:08:58.924 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:08:58.918+0000 7f519e065700 1 -- start start 2024-02-20T00:08:58.924 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:08:58.918+0000 7f519e065700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f51980703b0 0x7f5198100ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-20T00:08:58.924 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:08:58.918+0000 7f519e065700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51980fc8c0 con 0x7f51980703b0 2024-02-20T00:08:58.924 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:08:58.918+0000 7f51977fe700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f51980703b0 0x7f5198100ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:08:58.925 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:08:58.918+0000 7f51977fe700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f51980703b0 0x7f5198100ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:54916/0 (socket says 172.21.15.84:54916) 2024-02-20T00:08:58.925 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:08:58.918+0000 7f51977fe700 1 -- 172.21.15.84:0/2951201606 learned_addr learned my addr 172.21.15.84:0/2951201606 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-02-20T00:08:58.925 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:08:58.919+0000 7f51977fe700 1 -- 172.21.15.84:0/2951201606 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51981013c0 con 0x7f51980703b0 2024-02-20T00:08:58.925 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:08:58.919+0000 7f51977fe700 1 --2- 172.21.15.84:0/2951201606 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f51980703b0 0x7f5198100ed0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f518800b5f0 tx=0x7f5188006ee0).ready entity=mon.0 client_cookie=b7cf81c05163cce1 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:08:58.925 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:08:58.919+0000 7f51967fc700 1 -- 172.21.15.84:0/2951201606 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f518800d600 con 0x7f51980703b0 2024-02-20T00:08:58.925 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:08:58.919+0000 7f51967fc700 1 -- 172.21.15.84:0/2951201606 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f518800dbf0 con 0x7f51980703b0 2024-02-20T00:08:58.925 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:08:58.919+0000 7f51967fc700 1 -- 172.21.15.84:0/2951201606 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5188011cb0 con 0x7f51980703b0 2024-02-20T00:08:58.925 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:08:58.920+0000 7f519e065700 1 -- 172.21.15.84:0/2951201606 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f51980703b0 msgr2=0x7f5198100ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:08:58.925 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:08:58.920+0000 7f519e065700 1 --2- 172.21.15.84:0/2951201606 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f51980703b0 0x7f5198100ed0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f518800b5f0 tx=0x7f5188006ee0).stop 2024-02-20T00:08:58.925 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:08:58.920+0000 7f519e065700 1 -- 172.21.15.84:0/2951201606 shutdown_connections 2024-02-20T00:08:58.925 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:08:58.920+0000 7f519e065700 1 --2- 172.21.15.84:0/2951201606 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f51980703b0 0x7f5198100ed0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:08:58.925 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:08:58.920+0000 7f519e065700 1 -- 172.21.15.84:0/2951201606 >> 172.21.15.84:0/2951201606 conn(0x7f51980f6530 msgr2=0x7f51980f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:08:58.925 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:08:58.920+0000 7f519e065700 1 -- 172.21.15.84:0/2951201606 shutdown_connections 2024-02-20T00:08:58.926 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:08:58.920+0000 7f519e065700 1 -- 172.21.15.84:0/2951201606 wait complete. 2024-02-20T00:08:58.926 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:08:58.920+0000 7f519e065700 1 Processor -- start 2024-02-20T00:08:58.926 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:08:58.920+0000 7f519e065700 1 -- start start 2024-02-20T00:08:58.926 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:08:58.921+0000 7f519e065700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f51980703b0 0x7f519810cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:08:58.926 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:08:58.921+0000 7f519e065700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f519810d270 con 0x7f51980703b0 2024-02-20T00:08:58.926 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:08:58.921+0000 7f51977fe700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f51980703b0 0x7f519810cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:08:58.926 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:08:58.921+0000 7f51977fe700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f51980703b0 0x7f519810cd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:54928/0 (socket says 172.21.15.84:54928) 2024-02-20T00:08:58.926 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:08:58.921+0000 7f51977fe700 1 -- 172.21.15.84:0/64531182 learned_addr learned my addr 172.21.15.84:0/64531182 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-02-20T00:08:58.926 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:08:58.921+0000 7f51977fe700 1 -- 172.21.15.84:0/64531182 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f518800b200 con 0x7f51980703b0 2024-02-20T00:08:58.926 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:08:58.921+0000 7f51977fe700 1 --2- 172.21.15.84:0/64531182 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f51980703b0 0x7f519810cd80 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f5188006800 tx=0x7f5188006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:08:58.926 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:08:58.921+0000 7f517ffff700 1 -- 172.21.15.84:0/64531182 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f518801b570 con 0x7f51980703b0 2024-02-20T00:08:58.926 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:08:58.921+0000 7f519e065700 1 -- 172.21.15.84:0/64531182 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f519810d470 con 0x7f51980703b0 2024-02-20T00:08:58.926 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:08:58.921+0000 7f517ffff700 1 -- 172.21.15.84:0/64531182 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f518801bb60 con 0x7f51980703b0 2024-02-20T00:08:58.926 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:08:58.921+0000 7f517ffff700 1 -- 172.21.15.84:0/64531182 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f518801a9f0 con 0x7f51980703b0 2024-02-20T00:08:58.927 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:08:58.921+0000 7f519e065700 1 -- 172.21.15.84:0/64531182 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f519810e080 con 0x7f51980703b0 2024-02-20T00:08:58.927 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:08:58.922+0000 7f517ffff700 1 -- 172.21.15.84:0/64531182 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f518800f070 con 0x7f51980703b0 2024-02-20T00:08:58.927 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:08:58.922+0000 7f517ffff700 1 -- 172.21.15.84:0/64531182 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f5188025d80 con 0x7f51980703b0 2024-02-20T00:08:58.927 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:08:58.923+0000 7f519e065700 1 -- 172.21.15.84:0/64531182 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5198059100 con 0x7f51980703b0 2024-02-20T00:08:58.927 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:08:58.925+0000 7f517ffff700 1 -- 172.21.15.84:0/64531182 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f5188016070 con 0x7f51980703b0 2024-02-20T00:08:58.990 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:08:58.989+0000 7f519e065700 1 -- 172.21.15.84:0/64531182 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f51981020e0 con 0x7f51980703b0 2024-02-20T00:08:58.990 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:08:58.989+0000 7f517ffff700 1 -- 172.21.15.84:0/64531182 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+175 (secure 0 0 0) 0x7f5188025460 con 0x7f51980703b0 2024-02-20T00:08:58.991 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:08:58.990+0000 7f519e065700 1 -- 172.21.15.84:0/64531182 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f51980703b0 msgr2=0x7f519810cd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:08:58.992 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:08:58.990+0000 7f519e065700 1 --2- 172.21.15.84:0/64531182 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f51980703b0 0x7f519810cd80 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f5188006800 tx=0x7f5188006ee0).stop 2024-02-20T00:08:58.992 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:08:58.990+0000 7f519e065700 1 -- 172.21.15.84:0/64531182 shutdown_connections 2024-02-20T00:08:58.992 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:08:58.990+0000 7f519e065700 1 --2- 172.21.15.84:0/64531182 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f51980703b0 0x7f519810cd80 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:08:58.992 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:08:58.990+0000 7f519e065700 1 -- 172.21.15.84:0/64531182 >> 172.21.15.84:0/64531182 conn(0x7f51980f6530 msgr2=0x7f51980f7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:08:58.992 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:08:58.990+0000 7f519e065700 1 -- 172.21.15.84:0/64531182 shutdown_connections 2024-02-20T00:08:58.992 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:08:58.990+0000 7f519e065700 1 -- 172.21.15.84:0/64531182 wait complete. 2024-02-20T00:08:59.306 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:08:59 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[37700]: audit 2024-02-20T00:08:58.989836+0000 mon.smithi084 (mon.0) 10 : audit [DBG] from='client.? 172.21.15.84:0/64531182' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T00:08:59.528 INFO:teuthology.orchestra.run.smithi084.stderr:Restarting the monitor... 2024-02-20T00:08:59.529 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: systemctl restart ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715@mon.smithi084 2024-02-20T00:08:59.607 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:08:59 smithi084 systemd[1]: Stopping Ceph mon.smithi084 for 2a2e6c42-cf84-11ee-95bb-87774f69a715... 2024-02-20T00:09:00.046 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:08:59 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[37700]: debug 2024-02-20T00:08:59.666+0000 7f6621a94700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi084 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-02-20T00:09:00.046 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:08:59 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[37700]: debug 2024-02-20T00:08:59.666+0000 7f6621a94700 -1 mon.smithi084@0(leader) e1 *** Got Signal Terminated *** 2024-02-20T00:09:00.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 podman[38029]: ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084 2024-02-20T00:09:00.690 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 systemd[1]: ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715@mon.smithi084.service: Succeeded. 2024-02-20T00:09:00.690 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 systemd[1]: Stopped Ceph mon.smithi084 for 2a2e6c42-cf84-11ee-95bb-87774f69a715. 2024-02-20T00:09:00.691 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 systemd[1]: Starting Ceph mon.smithi084 for 2a2e6c42-cf84-11ee-95bb-87774f69a715... 2024-02-20T00:09:00.869 INFO:teuthology.orchestra.run.smithi084.stderr:Setting mon public_network to 172.21.0.0/20 2024-02-20T00:09:00.869 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi084 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/2a2e6c42-cf84-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp3ot6j0xn:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp4xryone4:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mon public_network 172.21.0.0/20 2024-02-20T00:09:01.110 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.808+0000 7fb5a430b700 0 set uid:gid to 167:167 (ceph:ceph) 2024-02-20T00:09:01.110 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.808+0000 7fb5a430b700 0 ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable), process ceph-mon, pid 7 2024-02-20T00:09:01.110 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.808+0000 7fb5a430b700 0 pidfile_write: ignore empty --pid-file 2024-02-20T00:09:01.110 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.813+0000 7fb5a430b700 0 load: jerasure load: lrc load: isa 2024-02-20T00:09:01.110 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: RocksDB version: 6.8.1 2024-02-20T00:09:01.110 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: 2024-02-20T00:09:01.110 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-02-20T00:09:01.110 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Compile date May 13 2021 2024-02-20T00:09:01.110 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: DB SUMMARY 2024-02-20T00:09:01.110 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: 2024-02-20T00:09:01.111 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: CURRENT file: CURRENT 2024-02-20T00:09:01.111 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: 2024-02-20T00:09:01.111 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: IDENTITY file: IDENTITY 2024-02-20T00:09:01.111 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: 2024-02-20T00:09:01.111 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: MANIFEST file: MANIFEST-000005 size: 131 Bytes 2024-02-20T00:09:01.111 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: 2024-02-20T00:09:01.111 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi084/store.db dir, Total Num: 1, files: 000004.sst 2024-02-20T00:09:01.111 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: 2024-02-20T00:09:01.111 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi084/store.db: 000006.log size: 98696 ; 2024-02-20T00:09:01.111 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: 2024-02-20T00:09:01.111 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.error_if_exists: 0 2024-02-20T00:09:01.111 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.create_if_missing: 0 2024-02-20T00:09:01.111 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.paranoid_checks: 1 2024-02-20T00:09:01.111 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.env: 0x55d70f21f1c0 2024-02-20T00:09:01.112 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.fs: Posix File System 2024-02-20T00:09:01.112 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.info_log: 0x55d710353960 2024-02-20T00:09:01.112 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.max_file_opening_threads: 16 2024-02-20T00:09:01.112 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.statistics: (nil) 2024-02-20T00:09:01.112 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.use_fsync: 0 2024-02-20T00:09:01.112 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.max_log_file_size: 0 2024-02-20T00:09:01.112 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-20T00:09:01.112 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.log_file_time_to_roll: 0 2024-02-20T00:09:01.112 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.keep_log_file_num: 1000 2024-02-20T00:09:01.112 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.recycle_log_file_num: 0 2024-02-20T00:09:01.112 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.allow_fallocate: 1 2024-02-20T00:09:01.112 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.allow_mmap_reads: 0 2024-02-20T00:09:01.112 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.allow_mmap_writes: 0 2024-02-20T00:09:01.113 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.use_direct_reads: 0 2024-02-20T00:09:01.113 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-20T00:09:01.113 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.create_missing_column_families: 0 2024-02-20T00:09:01.113 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.db_log_dir: 2024-02-20T00:09:01.113 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi084/store.db 2024-02-20T00:09:01.113 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.table_cache_numshardbits: 6 2024-02-20T00:09:01.113 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.max_subcompactions: 1 2024-02-20T00:09:01.113 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.max_background_flushes: -1 2024-02-20T00:09:01.113 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.WAL_ttl_seconds: 0 2024-02-20T00:09:01.113 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.WAL_size_limit_MB: 0 2024-02-20T00:09:01.113 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-02-20T00:09:01.113 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-20T00:09:01.113 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.is_fd_close_on_exec: 1 2024-02-20T00:09:01.113 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.advise_random_on_open: 1 2024-02-20T00:09:01.113 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.db_write_buffer_size: 0 2024-02-20T00:09:01.114 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.write_buffer_manager: 0x55d710fc8030 2024-02-20T00:09:01.114 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-20T00:09:01.114 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-02-20T00:09:01.114 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-20T00:09:01.114 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.use_adaptive_mutex: 0 2024-02-20T00:09:01.114 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.rate_limiter: (nil) 2024-02-20T00:09:01.114 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-20T00:09:01.114 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.wal_recovery_mode: 2 2024-02-20T00:09:01.114 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.enable_thread_tracking: 0 2024-02-20T00:09:01.114 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.enable_pipelined_write: 0 2024-02-20T00:09:01.114 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.unordered_write: 0 2024-02-20T00:09:01.114 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-20T00:09:01.115 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-20T00:09:01.115 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-20T00:09:01.115 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-20T00:09:01.115 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.row_cache: None 2024-02-20T00:09:01.115 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.wal_filter: None 2024-02-20T00:09:01.115 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-20T00:09:01.115 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.allow_ingest_behind: 0 2024-02-20T00:09:01.115 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.preserve_deletes: 0 2024-02-20T00:09:01.115 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.two_write_queues: 0 2024-02-20T00:09:01.115 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.manual_wal_flush: 0 2024-02-20T00:09:01.115 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.atomic_flush: 0 2024-02-20T00:09:01.115 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-20T00:09:01.115 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.persist_stats_to_disk: 0 2024-02-20T00:09:01.115 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.write_dbid_to_manifest: 0 2024-02-20T00:09:01.115 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.log_readahead_size: 0 2024-02-20T00:09:01.116 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2024-02-20T00:09:01.116 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.max_background_jobs: 2 2024-02-20T00:09:01.116 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.max_background_compactions: -1 2024-02-20T00:09:01.116 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-20T00:09:01.116 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-20T00:09:01.116 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.delayed_write_rate : 16777216 2024-02-20T00:09:01.116 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.max_total_wal_size: 0 2024-02-20T00:09:01.116 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-20T00:09:01.116 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.stats_dump_period_sec: 600 2024-02-20T00:09:01.116 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.stats_persist_period_sec: 600 2024-02-20T00:09:01.116 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-20T00:09:01.116 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.max_open_files: -1 2024-02-20T00:09:01.116 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.bytes_per_sync: 0 2024-02-20T00:09:01.117 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.wal_bytes_per_sync: 0 2024-02-20T00:09:01.117 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.strict_bytes_per_sync: 0 2024-02-20T00:09:01.117 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.compaction_readahead_size: 0 2024-02-20T00:09:01.117 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Compression algorithms supported: 2024-02-20T00:09:01.117 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-20T00:09:01.117 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: kZSTD supported: 0 2024-02-20T00:09:01.117 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: kXpressCompression supported: 0 2024-02-20T00:09:01.117 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: kLZ4HCCompression supported: 1 2024-02-20T00:09:01.117 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: kLZ4Compression supported: 1 2024-02-20T00:09:01.117 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: kBZip2Compression supported: 0 2024-02-20T00:09:01.117 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: kZlibCompression supported: 1 2024-02-20T00:09:01.117 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: kSnappyCompression supported: 1 2024-02-20T00:09:01.117 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Fast CRC32 supported: Supported on x86 2024-02-20T00:09:01.117 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi084/store.db/MANIFEST-000005 2024-02-20T00:09:01.117 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: 2024-02-20T00:09:01.118 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2024-02-20T00:09:01.118 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: 2024-02-20T00:09:01.118 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-20T00:09:01.118 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.merge_operator: 2024-02-20T00:09:01.118 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.compaction_filter: None 2024-02-20T00:09:01.118 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.compaction_filter_factory: None 2024-02-20T00:09:01.118 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.memtable_factory: SkipListFactory 2024-02-20T00:09:01.118 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.table_factory: BlockBasedTable 2024-02-20T00:09:01.118 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55d71026bcb0) 2024-02-20T00:09:01.118 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cache_index_and_filter_blocks: 1 2024-02-20T00:09:01.118 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cache_index_and_filter_blocks_with_high_priority: 0 2024-02-20T00:09:01.118 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-20T00:09:01.118 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: pin_top_level_index_and_filter: 1 2024-02-20T00:09:01.118 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: index_type: 0 2024-02-20T00:09:01.118 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: data_block_index_type: 0 2024-02-20T00:09:01.118 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: index_shortening: 1 2024-02-20T00:09:01.119 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: data_block_hash_table_util_ratio: 0.750000 2024-02-20T00:09:01.119 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: hash_index_allow_collision: 1 2024-02-20T00:09:01.119 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: checksum: 1 2024-02-20T00:09:01.119 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: no_block_cache: 0 2024-02-20T00:09:01.119 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: block_cache: 0x55d7102a0f10 2024-02-20T00:09:01.119 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: block_cache_name: BinnedLRUCache 2024-02-20T00:09:01.119 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: block_cache_options: 2024-02-20T00:09:01.119 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: capacity : 536870912 2024-02-20T00:09:01.119 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: num_shard_bits : 4 2024-02-20T00:09:01.119 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: strict_capacity_limit : 0 2024-02-20T00:09:01.119 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: high_pri_pool_ratio: 0.000 2024-02-20T00:09:01.119 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: block_cache_compressed: (nil) 2024-02-20T00:09:01.119 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: persistent_cache: (nil) 2024-02-20T00:09:01.119 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: block_size: 4096 2024-02-20T00:09:01.119 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: block_size_deviation: 10 2024-02-20T00:09:01.120 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: block_restart_interval: 16 2024-02-20T00:09:01.120 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: index_block_restart_interval: 1 2024-02-20T00:09:01.120 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: metadata_block_size: 4096 2024-02-20T00:09:01.120 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: partition_filters: 0 2024-02-20T00:09:01.120 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: use_delta_encoding: 1 2024-02-20T00:09:01.120 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: filter_policy: rocksdb.BuiltinBloomFilter 2024-02-20T00:09:01.120 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: whole_key_filtering: 1 2024-02-20T00:09:01.120 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: verify_compression: 0 2024-02-20T00:09:01.120 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: read_amp_bytes_per_bit: 0 2024-02-20T00:09:01.120 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: format_version: 2 2024-02-20T00:09:01.120 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: enable_index_compression: 1 2024-02-20T00:09:01.120 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: block_align: 0 2024-02-20T00:09:01.120 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: 2024-02-20T00:09:01.120 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.write_buffer_size: 33554432 2024-02-20T00:09:01.120 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.max_write_buffer_number: 2 2024-02-20T00:09:01.121 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.compression: NoCompression 2024-02-20T00:09:01.121 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.bottommost_compression: Disabled 2024-02-20T00:09:01.121 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.prefix_extractor: nullptr 2024-02-20T00:09:01.121 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-20T00:09:01.121 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.num_levels: 7 2024-02-20T00:09:01.121 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-20T00:09:01.121 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-20T00:09:01.121 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-02-20T00:09:01.121 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-20T00:09:01.121 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-20T00:09:01.121 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-20T00:09:01.121 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-20T00:09:01.121 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-20T00:09:01.121 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-20T00:09:01.122 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.compression_opts.window_bits: -14 2024-02-20T00:09:01.122 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.compression_opts.level: 32767 2024-02-20T00:09:01.122 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.compression_opts.strategy: 0 2024-02-20T00:09:01.122 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-20T00:09:01.122 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-20T00:09:01.122 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.compression_opts.enabled: false 2024-02-20T00:09:01.122 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-20T00:09:01.122 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-20T00:09:01.122 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-20T00:09:01.122 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.target_file_size_base: 67108864 2024-02-20T00:09:01.122 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.target_file_size_multiplier: 1 2024-02-20T00:09:01.122 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-20T00:09:01.122 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-20T00:09:01.122 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-20T00:09:01.122 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-20T00:09:01.123 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-20T00:09:01.123 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-20T00:09:01.123 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-20T00:09:01.123 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-20T00:09:01.123 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-20T00:09:01.123 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-20T00:09:01.123 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-20T00:09:01.123 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-20T00:09:01.123 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.arena_block_size: 4194304 2024-02-20T00:09:01.123 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-20T00:09:01.123 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-20T00:09:01.123 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-02-20T00:09:01.123 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.disable_auto_compactions: 0 2024-02-20T00:09:01.123 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-20T00:09:01.123 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-20T00:09:01.124 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-20T00:09:01.124 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-20T00:09:01.124 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-20T00:09:01.124 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-20T00:09:01.124 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-20T00:09:01.124 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-20T00:09:01.124 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-20T00:09:01.124 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-20T00:09:01.124 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.table_properties_collectors: 2024-02-20T00:09:01.124 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.inplace_update_support: 0 2024-02-20T00:09:01.124 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.inplace_update_num_locks: 10000 2024-02-20T00:09:01.124 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-20T00:09:01.124 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-20T00:09:01.125 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.memtable_huge_page_size: 0 2024-02-20T00:09:01.125 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.bloom_locality: 0 2024-02-20T00:09:01.125 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.max_successive_merges: 0 2024-02-20T00:09:01.125 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.optimize_filters_for_hits: 0 2024-02-20T00:09:01.125 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.paranoid_file_checks: 0 2024-02-20T00:09:01.125 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.force_consistency_checks: 0 2024-02-20T00:09:01.125 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.report_bg_io_stats: 0 2024-02-20T00:09:01.125 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.ttl: 2592000 2024-02-20T00:09:01.125 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: Options.periodic_compaction_seconds: 0 2024-02-20T00:09:01.125 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi084/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 5, log_number is 4,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-02-20T00:09:01.125 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: 2024-02-20T00:09:01.125 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.814+0000 7fb5a430b700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 4 2024-02-20T00:09:01.125 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: 2024-02-20T00:09:01.126 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.815+0000 7fb5a430b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1708387740815555, "job": 1, "event": "recovery_started", "log_files": [6]} 2024-02-20T00:09:01.126 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.815+0000 7fb5a430b700 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #6 mode 2 2024-02-20T00:09:01.126 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.815+0000 7fb5a430b700 3 rocksdb: [le/block_based/filter_policy.cc:584] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-02-20T00:09:01.126 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.815+0000 7fb5a430b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1708387740816403, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 7, "file_size": 94270, "table_properties": {"data_size": 92358, "index_size": 389, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 709, "raw_key_size": 12563, "raw_average_key_size": 49, "raw_value_size": 85389, "raw_average_value_size": 336, "num_data_blocks": 12, "num_entries": 254, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1708387740, "oldest_key_time": 3, "file_creation_time": 0}} 2024-02-20T00:09:01.126 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.815+0000 7fb5a430b700 4 rocksdb: [version_set.cc:3826] Creating manifest 8 2024-02-20T00:09:01.126 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: 2024-02-20T00:09:01.126 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.816+0000 7fb5a430b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1708387740816806, "job": 1, "event": "recovery_finished"} 2024-02-20T00:09:01.126 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.816+0000 7fb5a430b700 4 rocksdb: DB pointer 0x55d710365800 2024-02-20T00:09:01.126 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.817+0000 7fb58b123700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2024-02-20T00:09:01.126 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.817+0000 7fb58b123700 4 rocksdb: [db_impl/db_impl.cc:851] 2024-02-20T00:09:01.126 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: ** DB Stats ** 2024-02-20T00:09:01.126 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: Uptime(secs): 0.0 total, 0.0 interval 2024-02-20T00:09:01.126 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-20T00:09:01.126 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-20T00:09:01.126 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-20T00:09:01.127 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-20T00:09:01.127 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-02-20T00:09:01.127 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-20T00:09:01.127 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: 2024-02-20T00:09:01.127 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: ** Compaction Stats [default] ** 2024-02-20T00:09:01.127 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-20T00:09:01.127 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-20T00:09:01.127 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: L0 2/0 93.75 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 156.9 0.00 0.00 1 0.001 0 0 2024-02-20T00:09:01.127 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: Sum 2/0 93.75 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 156.9 0.00 0.00 1 0.001 0 0 2024-02-20T00:09:01.127 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 156.9 0.00 0.00 1 0.001 0 0 2024-02-20T00:09:01.127 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: 2024-02-20T00:09:01.127 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: ** Compaction Stats [default] ** 2024-02-20T00:09:01.127 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-20T00:09:01.127 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-20T00:09:01.128 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 156.9 0.00 0.00 1 0.001 0 0 2024-02-20T00:09:01.128 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: Uptime(secs): 0.0 total, 0.0 interval 2024-02-20T00:09:01.128 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: Flush(GB): cumulative 0.000, interval 0.000 2024-02-20T00:09:01.128 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-20T00:09:01.128 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: AddFile(Total Files): cumulative 0, interval 0 2024-02-20T00:09:01.128 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: AddFile(L0 Files): cumulative 0, interval 0 2024-02-20T00:09:01.128 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: AddFile(Keys): cumulative 0, interval 0 2024-02-20T00:09:01.128 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: Cumulative compaction: 0.00 GB write, 31.76 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-20T00:09:01.128 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: Interval compaction: 0.00 GB write, 31.76 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-20T00:09:01.128 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-20T00:09:01.128 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: 2024-02-20T00:09:01.128 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: ** File Read Latency Histogram By Level [default] ** 2024-02-20T00:09:01.128 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: 2024-02-20T00:09:01.129 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: ** Compaction Stats [default] ** 2024-02-20T00:09:01.129 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-20T00:09:01.129 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-20T00:09:01.129 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: L0 2/0 93.75 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 156.9 0.00 0.00 1 0.001 0 0 2024-02-20T00:09:01.129 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: Sum 2/0 93.75 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 156.9 0.00 0.00 1 0.001 0 0 2024-02-20T00:09:01.129 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-20T00:09:01.129 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: 2024-02-20T00:09:01.129 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: ** Compaction Stats [default] ** 2024-02-20T00:09:01.129 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-20T00:09:01.129 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-20T00:09:01.129 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 156.9 0.00 0.00 1 0.001 0 0 2024-02-20T00:09:01.129 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: Uptime(secs): 0.0 total, 0.0 interval 2024-02-20T00:09:01.129 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: Flush(GB): cumulative 0.000, interval 0.000 2024-02-20T00:09:01.129 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-20T00:09:01.129 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: AddFile(Total Files): cumulative 0, interval 0 2024-02-20T00:09:01.130 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: AddFile(L0 Files): cumulative 0, interval 0 2024-02-20T00:09:01.130 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: AddFile(Keys): cumulative 0, interval 0 2024-02-20T00:09:01.130 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: Cumulative compaction: 0.00 GB write, 31.43 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-20T00:09:01.130 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-20T00:09:01.130 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-20T00:09:01.130 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: 2024-02-20T00:09:01.130 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: ** File Read Latency Histogram By Level [default] ** 2024-02-20T00:09:01.130 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: 2024-02-20T00:09:01.130 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.817+0000 7fb5a430b700 0 starting mon.smithi084 rank 0 at public addrs [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] at bind addrs [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi084 fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 2024-02-20T00:09:01.130 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.818+0000 7fb5a430b700 1 mon.smithi084@-1(???) e1 preinit fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 2024-02-20T00:09:01.130 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.818+0000 7fb5a430b700 0 mon.smithi084@-1(???).mds e1 new map 2024-02-20T00:09:01.130 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.818+0000 7fb5a430b700 0 mon.smithi084@-1(???).mds e1 print_map 2024-02-20T00:09:01.130 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: e1 2024-02-20T00:09:01.130 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: enable_multiple, ever_enabled_multiple: 1,1 2024-02-20T00:09:01.131 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-20T00:09:01.131 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: legacy client fscid: -1 2024-02-20T00:09:01.131 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: 2024-02-20T00:09:01.131 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: No filesystems configured 2024-02-20T00:09:01.131 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: 2024-02-20T00:09:01.131 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.818+0000 7fb5a430b700 0 mon.smithi084@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-02-20T00:09:01.131 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.818+0000 7fb5a430b700 0 mon.smithi084@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-20T00:09:01.131 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.818+0000 7fb5a430b700 0 mon.smithi084@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-20T00:09:01.131 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.818+0000 7fb5a430b700 0 mon.smithi084@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-20T00:09:01.131 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:08:58.989836+0000 mon.smithi084 (mon.0) 10 : audit [DBG] from='client.? 172.21.15.84:0/64531182' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T00:09:01.132 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.818+0000 7fb5a430b700 1 mon.smithi084@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-02-20T00:09:01.132 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.819+0000 7fb5a430b700 20 expand_channel_meta expand map: {default=false} 2024-02-20T00:09:01.132 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.819+0000 7fb5a430b700 20 expand_channel_meta from 'false' to 'false' 2024-02-20T00:09:01.132 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.819+0000 7fb5a430b700 20 expand_channel_meta expanded map: {default=false} 2024-02-20T00:09:01.132 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.819+0000 7fb5a430b700 20 expand_channel_meta expand map: {default=info} 2024-02-20T00:09:01.132 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.819+0000 7fb5a430b700 20 expand_channel_meta from 'info' to 'info' 2024-02-20T00:09:01.132 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.819+0000 7fb5a430b700 20 expand_channel_meta expanded map: {default=info} 2024-02-20T00:09:01.132 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.819+0000 7fb5a430b700 20 expand_channel_meta expand map: {default=daemon} 2024-02-20T00:09:01.132 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.819+0000 7fb5a430b700 20 expand_channel_meta from 'daemon' to 'daemon' 2024-02-20T00:09:01.132 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.819+0000 7fb5a430b700 20 expand_channel_meta expanded map: {default=daemon} 2024-02-20T00:09:01.132 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.819+0000 7fb5a430b700 20 expand_channel_meta expand map: {default=debug} 2024-02-20T00:09:01.132 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.819+0000 7fb5a430b700 20 expand_channel_meta from 'debug' to 'debug' 2024-02-20T00:09:01.132 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:09:00.819+0000 7fb5a430b700 20 expand_channel_meta expanded map: {default=debug} 2024-02-20T00:09:01.132 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 bash[38100]: 8cb1a74f96d9c09134fa105836cd1325ad31f6c897f121c39fde9aa211cd6e2e 2024-02-20T00:09:01.133 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 systemd[1]: Started Ceph mon.smithi084 for 2a2e6c42-cf84-11ee-95bb-87774f69a715. 2024-02-20T00:09:01.133 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:09:00.835266+0000 mon.smithi084 (mon.0) 1 : cluster [INF] mon.smithi084 is new leader, mons smithi084 in quorum (ranks 0) 2024-02-20T00:09:01.133 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:09:00.835328+0000 mon.smithi084 (mon.0) 2 : cluster [DBG] monmap e1: 1 mons at {smithi084=[v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0]} 2024-02-20T00:09:01.133 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:09:00.835362+0000 mon.smithi084 (mon.0) 3 : cluster [DBG] fsmap 2024-02-20T00:09:01.133 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:09:00.835386+0000 mon.smithi084 (mon.0) 4 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2024-02-20T00:09:01.133 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:09:00.835748+0000 mon.smithi084 (mon.0) 5 : cluster [DBG] mgrmap e1: no daemons active 2024-02-20T00:09:01.460 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:01.458+0000 7f84b5f47700 1 Processor -- start 2024-02-20T00:09:01.464 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:01.458+0000 7f84b5f47700 1 -- start start 2024-02-20T00:09:01.464 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:01.458+0000 7f84b5f47700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f84b00fad90 0x7f84b00fb1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-20T00:09:01.464 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:01.458+0000 7f84b5f47700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84b00fb6a0 con 0x7f84b00fad90 2024-02-20T00:09:01.464 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:01.458+0000 7f84af7fe700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f84b00fad90 0x7f84b00fb1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:09:01.464 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:01.458+0000 7f84af7fe700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f84b00fad90 0x7f84b00fb1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:54936/0 (socket says 172.21.15.84:54936) 2024-02-20T00:09:01.464 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:01.459+0000 7f84af7fe700 1 -- 172.21.15.84:0/345802220 learned_addr learned my addr 172.21.15.84:0/345802220 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-02-20T00:09:01.465 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:01.459+0000 7f84af7fe700 1 -- 172.21.15.84:0/345802220 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84b00fb7e0 con 0x7f84b00fad90 2024-02-20T00:09:01.465 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:01.459+0000 7f84af7fe700 1 --2- 172.21.15.84:0/345802220 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f84b00fad90 0x7f84b00fb1b0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f84a000b870 tx=0x7f84a0006f70).ready entity=mon.0 client_cookie=b06998698786cd24 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:09:01.465 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:01.459+0000 7f84ae7fc700 1 -- 172.21.15.84:0/345802220 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f84a000da20 con 0x7f84b00fad90 2024-02-20T00:09:01.465 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:01.459+0000 7f84ae7fc700 1 -- 172.21.15.84:0/345802220 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f84a0010470 con 0x7f84b00fad90 2024-02-20T00:09:01.465 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:01.459+0000 7f84ae7fc700 1 -- 172.21.15.84:0/345802220 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f84a000f5b0 con 0x7f84b00fad90 2024-02-20T00:09:01.465 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:01.460+0000 7f84b5f47700 1 -- 172.21.15.84:0/345802220 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f84b00fad90 msgr2=0x7f84b00fb1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:09:01.465 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:01.460+0000 7f84b5f47700 1 --2- 172.21.15.84:0/345802220 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f84b00fad90 0x7f84b00fb1b0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f84a000b870 tx=0x7f84a0006f70).stop 2024-02-20T00:09:01.465 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:01.460+0000 7f84b5f47700 1 -- 172.21.15.84:0/345802220 shutdown_connections 2024-02-20T00:09:01.465 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:01.460+0000 7f84b5f47700 1 --2- 172.21.15.84:0/345802220 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f84b00fad90 0x7f84b00fb1b0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:09:01.465 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:01.460+0000 7f84b5f47700 1 -- 172.21.15.84:0/345802220 >> 172.21.15.84:0/345802220 conn(0x7f84b00f6530 msgr2=0x7f84b00f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:09:01.465 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:01.460+0000 7f84b5f47700 1 -- 172.21.15.84:0/345802220 shutdown_connections 2024-02-20T00:09:01.465 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:01.460+0000 7f84b5f47700 1 -- 172.21.15.84:0/345802220 wait complete. 2024-02-20T00:09:01.465 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:01.460+0000 7f84b5f47700 1 Processor -- start 2024-02-20T00:09:01.466 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:01.461+0000 7f84b5f47700 1 -- start start 2024-02-20T00:09:01.466 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:01.461+0000 7f84b5f47700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f84b00fad90 0x7f84b010cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:09:01.466 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:01.461+0000 7f84b5f47700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84b010d270 con 0x7f84b00fad90 2024-02-20T00:09:01.466 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:01.461+0000 7f84af7fe700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f84b00fad90 0x7f84b010cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:09:01.466 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:01.461+0000 7f84af7fe700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f84b00fad90 0x7f84b010cd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:54946/0 (socket says 172.21.15.84:54946) 2024-02-20T00:09:01.466 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:01.461+0000 7f84af7fe700 1 -- 172.21.15.84:0/1667647723 learned_addr learned my addr 172.21.15.84:0/1667647723 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-02-20T00:09:01.466 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:01.461+0000 7f84af7fe700 1 -- 172.21.15.84:0/1667647723 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84a000b480 con 0x7f84b00fad90 2024-02-20T00:09:01.466 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:01.461+0000 7f84af7fe700 1 --2- 172.21.15.84:0/1667647723 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f84b00fad90 0x7f84b010cd80 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f84a0006800 tx=0x7f84a0006f70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:09:01.466 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:01.461+0000 7f8497fff700 1 -- 172.21.15.84:0/1667647723 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f84a000f870 con 0x7f84b00fad90 2024-02-20T00:09:01.466 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:01.462+0000 7f8497fff700 1 -- 172.21.15.84:0/1667647723 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f84a000fe60 con 0x7f84b00fad90 2024-02-20T00:09:01.466 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:01.462+0000 7f8497fff700 1 -- 172.21.15.84:0/1667647723 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f84a0019e40 con 0x7f84b00fad90 2024-02-20T00:09:01.466 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:01.462+0000 7f84b5f47700 1 -- 172.21.15.84:0/1667647723 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84b010d470 con 0x7f84b00fad90 2024-02-20T00:09:01.466 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:01.462+0000 7f84b5f47700 1 -- 172.21.15.84:0/1667647723 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84b010e080 con 0x7f84b00fad90 2024-02-20T00:09:01.466 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:01.462+0000 7f8497fff700 1 -- 172.21.15.84:0/1667647723 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f84a002a070 con 0x7f84b00fad90 2024-02-20T00:09:01.467 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:01.462+0000 7f8497fff700 1 -- 172.21.15.84:0/1667647723 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f84a0023760 con 0x7f84b00fad90 2024-02-20T00:09:01.467 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:01.463+0000 7f84b5f47700 1 -- 172.21.15.84:0/1667647723 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84b0059100 con 0x7f84b00fad90 2024-02-20T00:09:01.467 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:01.465+0000 7f8497fff700 1 -- 172.21.15.84:0/1667647723 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f84a0028060 con 0x7f84b00fad90 2024-02-20T00:09:01.529 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:01.527+0000 7f84b5f47700 1 -- 172.21.15.84:0/1667647723 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7f84b00fcc20 con 0x7f84b00fad90 2024-02-20T00:09:01.531 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:01.530+0000 7f8497fff700 1 -- 172.21.15.84:0/1667647723 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7f84a0014070 con 0x7f84b00fad90 2024-02-20T00:09:01.532 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:01.530+0000 7f84b5f47700 1 -- 172.21.15.84:0/1667647723 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f84b00fad90 msgr2=0x7f84b010cd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:09:01.532 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:01.530+0000 7f84b5f47700 1 --2- 172.21.15.84:0/1667647723 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f84b00fad90 0x7f84b010cd80 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f84a0006800 tx=0x7f84a0006f70).stop 2024-02-20T00:09:01.533 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:01.530+0000 7f84b5f47700 1 -- 172.21.15.84:0/1667647723 shutdown_connections 2024-02-20T00:09:01.533 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:01.530+0000 7f84b5f47700 1 --2- 172.21.15.84:0/1667647723 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f84b00fad90 0x7f84b010cd80 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:09:01.533 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:01.530+0000 7f84b5f47700 1 -- 172.21.15.84:0/1667647723 >> 172.21.15.84:0/1667647723 conn(0x7f84b00f6530 msgr2=0x7f84b00ff9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:09:01.533 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:01.530+0000 7f84b5f47700 1 -- 172.21.15.84:0/1667647723 shutdown_connections 2024-02-20T00:09:01.533 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:01.530+0000 7f84b5f47700 1 -- 172.21.15.84:0/1667647723 wait complete. 2024-02-20T00:09:02.108 INFO:teuthology.orchestra.run.smithi084.stderr:Wrote config to /etc/ceph/ceph.conf 2024-02-20T00:09:02.108 INFO:teuthology.orchestra.run.smithi084.stderr:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-02-20T00:09:02.108 INFO:teuthology.orchestra.run.smithi084.stderr:Creating mgr... 2024-02-20T00:09:02.108 INFO:teuthology.orchestra.run.smithi084.stderr:Verifying port 9283 ... 2024-02-20T00:09:02.110 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: systemctl daemon-reload 2024-02-20T00:09:02.208 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: systemctl stop ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715@mgr.smithi084.bmqrnf 2024-02-20T00:09:02.230 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: systemctl reset-failed ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715@mgr.smithi084.bmqrnf 2024-02-20T00:09:02.236 INFO:teuthology.orchestra.run.smithi084.stderr:systemctl: Failed to reset failed state of unit ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715@mgr.smithi084.bmqrnf.service: Unit ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715@mgr.smithi084.bmqrnf.service not loaded. 2024-02-20T00:09:02.236 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: systemctl enable ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715@mgr.smithi084.bmqrnf 2024-02-20T00:09:02.242 INFO:teuthology.orchestra.run.smithi084.stderr:systemctl: Created symlink /etc/systemd/system/ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715.target.wants/ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715@mgr.smithi084.bmqrnf.service → /etc/systemd/system/ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715@.service. 2024-02-20T00:09:02.329 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: systemctl start ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715@mgr.smithi084.bmqrnf 2024-02-20T00:09:02.858 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:02 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:09:01.530478+0000 mon.smithi084 (mon.0) 6 : audit [INF] from='client.? 172.21.15.84:0/1667647723' entity='client.admin' 2024-02-20T00:09:02.904 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: systemctl is-enabled firewalld.service 2024-02-20T00:09:02.913 INFO:teuthology.orchestra.run.smithi084.stderr:systemctl: disabled 2024-02-20T00:09:02.913 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: systemctl is-active firewalld.service 2024-02-20T00:09:02.920 INFO:teuthology.orchestra.run.smithi084.stderr:systemctl: inactive 2024-02-20T00:09:02.921 INFO:teuthology.orchestra.run.smithi084.stderr:firewalld.service is not enabled 2024-02-20T00:09:02.921 INFO:teuthology.orchestra.run.smithi084.stderr:Not possible to enable service . firewalld.service is not available 2024-02-20T00:09:02.921 INFO:teuthology.orchestra.run.smithi084.stderr:Not possible to open ports <[]>. firewalld.service is not available 2024-02-20T00:09:02.921 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: systemctl is-enabled firewalld.service 2024-02-20T00:09:02.928 INFO:teuthology.orchestra.run.smithi084.stderr:systemctl: disabled 2024-02-20T00:09:02.928 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: systemctl is-active firewalld.service 2024-02-20T00:09:02.935 INFO:teuthology.orchestra.run.smithi084.stderr:systemctl: inactive 2024-02-20T00:09:02.936 INFO:teuthology.orchestra.run.smithi084.stderr:firewalld.service is not enabled 2024-02-20T00:09:02.936 INFO:teuthology.orchestra.run.smithi084.stderr:Not possible to open ports <[9283]>. firewalld.service is not available 2024-02-20T00:09:02.936 INFO:teuthology.orchestra.run.smithi084.stderr:Waiting for mgr to start... 2024-02-20T00:09:02.936 INFO:teuthology.orchestra.run.smithi084.stderr:Waiting for mgr... 2024-02-20T00:09:02.936 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi084 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/2a2e6c42-cf84-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp3ot6j0xn:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp4xryone4:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2024-02-20T00:09:03.526 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:03.522+0000 7fe84b0d2700 1 Processor -- start 2024-02-20T00:09:03.527 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:03.522+0000 7fe84b0d2700 1 -- start start 2024-02-20T00:09:03.527 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:03.522+0000 7fe84b0d2700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe844070430 0x7fe8440fea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-20T00:09:03.527 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:03.522+0000 7fe84b0d2700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8440fef40 con 0x7fe844070430 2024-02-20T00:09:03.527 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:03.522+0000 7fe848e6e700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe844070430 0x7fe8440fea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:09:03.527 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:03.522+0000 7fe848e6e700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe844070430 0x7fe8440fea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:54976/0 (socket says 172.21.15.84:54976) 2024-02-20T00:09:03.527 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:03.522+0000 7fe848e6e700 1 -- 172.21.15.84:0/1293312639 learned_addr learned my addr 172.21.15.84:0/1293312639 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-02-20T00:09:03.527 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:03.523+0000 7fe848e6e700 1 -- 172.21.15.84:0/1293312639 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8440ff080 con 0x7fe844070430 2024-02-20T00:09:03.527 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:03.523+0000 7fe848e6e700 1 --2- 172.21.15.84:0/1293312639 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe844070430 0x7fe8440fea50 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7fe83800b5f0 tx=0x7fe838006ee0).ready entity=mon.0 client_cookie=d3a93bd8c88ec154 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:09:03.527 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:03.523+0000 7fe8437fe700 1 -- 172.21.15.84:0/1293312639 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe83800d600 con 0x7fe844070430 2024-02-20T00:09:03.527 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:03.523+0000 7fe8437fe700 1 -- 172.21.15.84:0/1293312639 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fe83800dbf0 con 0x7fe844070430 2024-02-20T00:09:03.527 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:03.523+0000 7fe8437fe700 1 -- 172.21.15.84:0/1293312639 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe838011cb0 con 0x7fe844070430 2024-02-20T00:09:03.527 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:03.524+0000 7fe84b0d2700 1 -- 172.21.15.84:0/1293312639 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe844070430 msgr2=0x7fe8440fea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:09:03.527 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:03.524+0000 7fe84b0d2700 1 --2- 172.21.15.84:0/1293312639 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe844070430 0x7fe8440fea50 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7fe83800b5f0 tx=0x7fe838006ee0).stop 2024-02-20T00:09:03.528 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:03.524+0000 7fe84b0d2700 1 -- 172.21.15.84:0/1293312639 shutdown_connections 2024-02-20T00:09:03.528 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:03.524+0000 7fe84b0d2700 1 --2- 172.21.15.84:0/1293312639 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe844070430 0x7fe8440fea50 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:09:03.528 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:03.524+0000 7fe84b0d2700 1 -- 172.21.15.84:0/1293312639 >> 172.21.15.84:0/1293312639 conn(0x7fe8440f6530 msgr2=0x7fe8440f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:09:03.528 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:03.524+0000 7fe84b0d2700 1 -- 172.21.15.84:0/1293312639 shutdown_connections 2024-02-20T00:09:03.528 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:03.524+0000 7fe84b0d2700 1 -- 172.21.15.84:0/1293312639 wait complete. 2024-02-20T00:09:03.528 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:03.525+0000 7fe84b0d2700 1 Processor -- start 2024-02-20T00:09:03.528 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:03.525+0000 7fe84b0d2700 1 -- start start 2024-02-20T00:09:03.528 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:03.525+0000 7fe84b0d2700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe844070430 0x7fe8440fc4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:09:03.528 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:03.525+0000 7fe84b0d2700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8440fa950 con 0x7fe844070430 2024-02-20T00:09:03.528 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:03.525+0000 7fe848e6e700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe844070430 0x7fe8440fc4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:09:03.530 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:03.525+0000 7fe848e6e700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe844070430 0x7fe8440fc4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:54988/0 (socket says 172.21.15.84:54988) 2024-02-20T00:09:03.530 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:03.525+0000 7fe848e6e700 1 -- 172.21.15.84:0/3821002738 learned_addr learned my addr 172.21.15.84:0/3821002738 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-02-20T00:09:03.530 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:03.526+0000 7fe848e6e700 1 -- 172.21.15.84:0/3821002738 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe83800b200 con 0x7fe844070430 2024-02-20T00:09:03.530 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:03.526+0000 7fe848e6e700 1 --2- 172.21.15.84:0/3821002738 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe844070430 0x7fe8440fc4d0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fe838006800 tx=0x7fe838006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:09:03.530 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:03.526+0000 7fe8417fa700 1 -- 172.21.15.84:0/3821002738 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe83801b570 con 0x7fe844070430 2024-02-20T00:09:03.530 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:03.526+0000 7fe84b0d2700 1 -- 172.21.15.84:0/3821002738 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8440faaf0 con 0x7fe844070430 2024-02-20T00:09:03.530 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:03.526+0000 7fe8417fa700 1 -- 172.21.15.84:0/3821002738 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fe83801bb60 con 0x7fe844070430 2024-02-20T00:09:03.530 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:03.526+0000 7fe8417fa700 1 -- 172.21.15.84:0/3821002738 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe83801a9f0 con 0x7fe844070430 2024-02-20T00:09:03.530 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:03.526+0000 7fe84b0d2700 1 -- 172.21.15.84:0/3821002738 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8440faf80 con 0x7fe844070430 2024-02-20T00:09:03.530 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:03.526+0000 7fe8417fa700 1 -- 172.21.15.84:0/3821002738 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7fe83800f070 con 0x7fe844070430 2024-02-20T00:09:03.530 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:03.526+0000 7fe8417fa700 1 -- 172.21.15.84:0/3821002738 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fe838025d40 con 0x7fe844070430 2024-02-20T00:09:03.530 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:03.527+0000 7fe84b0d2700 1 -- 172.21.15.84:0/3821002738 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe844045ad0 con 0x7fe844070430 2024-02-20T00:09:03.531 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:03.529+0000 7fe8417fa700 1 -- 172.21.15.84:0/3821002738 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7fe838016070 con 0x7fe844070430 2024-02-20T00:09:03.607 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:03.598+0000 7fe84b0d2700 1 -- 172.21.15.84:0/3821002738 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fe844104af0 con 0x7fe844070430 2024-02-20T00:09:03.607 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:03.598+0000 7fe8417fa700 1 -- 172.21.15.84:0/3821002738 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1230 (secure 0 0 0) 0x7fe838025460 con 0x7fe844070430 2024-02-20T00:09:03.607 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:03.607 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: { 2024-02-20T00:09:03.607 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "fsid": "2a2e6c42-cf84-11ee-95bb-87774f69a715", 2024-02-20T00:09:03.607 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "health": { 2024-02-20T00:09:03.607 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-02-20T00:09:03.607 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "checks": {}, 2024-02-20T00:09:03.607 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "mutes": [] 2024-02-20T00:09:03.607 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: }, 2024-02-20T00:09:03.607 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-02-20T00:09:03.608 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "quorum": [ 2024-02-20T00:09:03.608 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 0 2024-02-20T00:09:03.608 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: ], 2024-02-20T00:09:03.608 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "quorum_names": [ 2024-02-20T00:09:03.608 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "smithi084" 2024-02-20T00:09:03.608 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: ], 2024-02-20T00:09:03.608 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "quorum_age": 2, 2024-02-20T00:09:03.608 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "monmap": { 2024-02-20T00:09:03.608 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-20T00:09:03.608 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2024-02-20T00:09:03.608 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "num_mons": 1 2024-02-20T00:09:03.608 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: }, 2024-02-20T00:09:03.608 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "osdmap": { 2024-02-20T00:09:03.608 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-20T00:09:03.609 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "num_osds": 0, 2024-02-20T00:09:03.609 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-02-20T00:09:03.609 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-02-20T00:09:03.609 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-02-20T00:09:03.609 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-02-20T00:09:03.609 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-02-20T00:09:03.609 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: }, 2024-02-20T00:09:03.609 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "pgmap": { 2024-02-20T00:09:03.609 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-02-20T00:09:03.609 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-02-20T00:09:03.609 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "num_pools": 0, 2024-02-20T00:09:03.609 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "num_objects": 0, 2024-02-20T00:09:03.609 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-02-20T00:09:03.610 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-02-20T00:09:03.610 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-02-20T00:09:03.610 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "bytes_total": 0 2024-02-20T00:09:03.610 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: }, 2024-02-20T00:09:03.610 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "fsmap": { 2024-02-20T00:09:03.610 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-20T00:09:03.610 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "by_rank": [], 2024-02-20T00:09:03.610 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "up:standby": 0 2024-02-20T00:09:03.610 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: }, 2024-02-20T00:09:03.610 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "mgrmap": { 2024-02-20T00:09:03.610 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "available": false, 2024-02-20T00:09:03.610 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-02-20T00:09:03.610 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "modules": [ 2024-02-20T00:09:03.610 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "iostat", 2024-02-20T00:09:03.610 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "restful" 2024-02-20T00:09:03.610 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: ], 2024-02-20T00:09:03.611 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "services": {} 2024-02-20T00:09:03.611 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: }, 2024-02-20T00:09:03.611 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "servicemap": { 2024-02-20T00:09:03.611 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-20T00:09:03.611 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "modified": "2024-02-20T00:08:55.999532+0000", 2024-02-20T00:09:03.611 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "services": {} 2024-02-20T00:09:03.611 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: }, 2024-02-20T00:09:03.611 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "progress_events": {} 2024-02-20T00:09:03.611 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: } 2024-02-20T00:09:03.611 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:03.599+0000 7fe84b0d2700 1 -- 172.21.15.84:0/3821002738 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe844070430 msgr2=0x7fe8440fc4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:09:03.611 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:03.599+0000 7fe84b0d2700 1 --2- 172.21.15.84:0/3821002738 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe844070430 0x7fe8440fc4d0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fe838006800 tx=0x7fe838006ee0).stop 2024-02-20T00:09:03.611 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:03.599+0000 7fe84b0d2700 1 -- 172.21.15.84:0/3821002738 shutdown_connections 2024-02-20T00:09:03.611 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:03.599+0000 7fe84b0d2700 1 --2- 172.21.15.84:0/3821002738 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe844070430 0x7fe8440fc4d0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:09:03.611 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:03.599+0000 7fe84b0d2700 1 -- 172.21.15.84:0/3821002738 >> 172.21.15.84:0/3821002738 conn(0x7fe8440f6530 msgr2=0x7fe8440f7f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:09:03.611 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:03.599+0000 7fe84b0d2700 1 -- 172.21.15.84:0/3821002738 shutdown_connections 2024-02-20T00:09:03.612 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:03.599+0000 7fe84b0d2700 1 -- 172.21.15.84:0/3821002738 wait complete. 2024-02-20T00:09:03.931 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:03 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:09:03.598747+0000 mon.smithi084 (mon.0) 7 : audit [DBG] from='client.? 172.21.15.84:0/3821002738' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-20T00:09:04.234 INFO:teuthology.orchestra.run.smithi084.stderr:mgr not available, waiting (1/15)... 2024-02-20T00:09:06.236 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi084 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/2a2e6c42-cf84-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp3ot6j0xn:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp4xryone4:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2024-02-20T00:09:06.786 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:06.757+0000 7fed18745700 1 Processor -- start 2024-02-20T00:09:06.786 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:06.757+0000 7fed18745700 1 -- start start 2024-02-20T00:09:06.786 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:06.757+0000 7fed18745700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fed100703d0 0x7fed10100eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-20T00:09:06.786 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:06.757+0000 7fed18745700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed100fc8a0 con 0x7fed100703d0 2024-02-20T00:09:06.786 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:06.757+0000 7fed164e1700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fed100703d0 0x7fed10100eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:09:06.786 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:06.758+0000 7fed164e1700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fed100703d0 0x7fed10100eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:55002/0 (socket says 172.21.15.84:55002) 2024-02-20T00:09:06.786 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:06.758+0000 7fed164e1700 1 -- 172.21.15.84:0/3479406164 learned_addr learned my addr 172.21.15.84:0/3479406164 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-02-20T00:09:06.786 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:06.758+0000 7fed164e1700 1 -- 172.21.15.84:0/3479406164 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed101013a0 con 0x7fed100703d0 2024-02-20T00:09:06.786 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:06.758+0000 7fed164e1700 1 --2- 172.21.15.84:0/3479406164 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fed100703d0 0x7fed10100eb0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7fed0c00b5f0 tx=0x7fed0c006ee0).ready entity=mon.0 client_cookie=e09f4a9371388008 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:09:06.786 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:06.758+0000 7fed154df700 1 -- 172.21.15.84:0/3479406164 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fed0c00d600 con 0x7fed100703d0 2024-02-20T00:09:06.787 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:06.758+0000 7fed154df700 1 -- 172.21.15.84:0/3479406164 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fed0c00dbf0 con 0x7fed100703d0 2024-02-20T00:09:06.787 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:06.758+0000 7fed154df700 1 -- 172.21.15.84:0/3479406164 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fed0c011cb0 con 0x7fed100703d0 2024-02-20T00:09:06.787 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:06.759+0000 7fed18745700 1 -- 172.21.15.84:0/3479406164 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fed100703d0 msgr2=0x7fed10100eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:09:06.787 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:06.759+0000 7fed18745700 1 --2- 172.21.15.84:0/3479406164 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fed100703d0 0x7fed10100eb0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7fed0c00b5f0 tx=0x7fed0c006ee0).stop 2024-02-20T00:09:06.787 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:06.759+0000 7fed18745700 1 -- 172.21.15.84:0/3479406164 shutdown_connections 2024-02-20T00:09:06.787 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:06.759+0000 7fed18745700 1 --2- 172.21.15.84:0/3479406164 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fed100703d0 0x7fed10100eb0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:09:06.787 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:06.759+0000 7fed18745700 1 -- 172.21.15.84:0/3479406164 >> 172.21.15.84:0/3479406164 conn(0x7fed100f6530 msgr2=0x7fed100f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:09:06.787 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:06.759+0000 7fed18745700 1 -- 172.21.15.84:0/3479406164 shutdown_connections 2024-02-20T00:09:06.787 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:06.759+0000 7fed18745700 1 -- 172.21.15.84:0/3479406164 wait complete. 2024-02-20T00:09:06.787 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:06.759+0000 7fed18745700 1 Processor -- start 2024-02-20T00:09:06.787 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:06.759+0000 7fed18745700 1 -- start start 2024-02-20T00:09:06.787 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:06.759+0000 7fed18745700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fed100703d0 0x7fed1010ce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:09:06.787 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:06.759+0000 7fed18745700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed1010d310 con 0x7fed100703d0 2024-02-20T00:09:06.787 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:06.759+0000 7fed164e1700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fed100703d0 0x7fed1010ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:09:06.787 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:06.759+0000 7fed164e1700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fed100703d0 0x7fed1010ce20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:55004/0 (socket says 172.21.15.84:55004) 2024-02-20T00:09:06.788 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:06.759+0000 7fed164e1700 1 -- 172.21.15.84:0/2807550052 learned_addr learned my addr 172.21.15.84:0/2807550052 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-02-20T00:09:06.788 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:06.760+0000 7fed164e1700 1 -- 172.21.15.84:0/2807550052 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed0c00b200 con 0x7fed100703d0 2024-02-20T00:09:06.788 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:06.760+0000 7fed164e1700 1 --2- 172.21.15.84:0/2807550052 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fed100703d0 0x7fed1010ce20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7fed0c006c70 tx=0x7fed0c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:09:06.788 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:06.760+0000 7fecfeffd700 1 -- 172.21.15.84:0/2807550052 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fed0c011400 con 0x7fed100703d0 2024-02-20T00:09:06.788 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:06.760+0000 7fed18745700 1 -- 172.21.15.84:0/2807550052 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed1010d510 con 0x7fed100703d0 2024-02-20T00:09:06.788 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:06.760+0000 7fecfeffd700 1 -- 172.21.15.84:0/2807550052 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fed0c01b8a0 con 0x7fed100703d0 2024-02-20T00:09:06.788 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:06.760+0000 7fecfeffd700 1 -- 172.21.15.84:0/2807550052 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fed0c01a7c0 con 0x7fed100703d0 2024-02-20T00:09:06.788 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:06.760+0000 7fed18745700 1 -- 172.21.15.84:0/2807550052 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed1010e120 con 0x7fed100703d0 2024-02-20T00:09:06.788 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:06.760+0000 7fecfeffd700 1 -- 172.21.15.84:0/2807550052 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7fed0c00f070 con 0x7fed100703d0 2024-02-20T00:09:06.788 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:06.760+0000 7fecfeffd700 1 -- 172.21.15.84:0/2807550052 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fed0c025be0 con 0x7fed100703d0 2024-02-20T00:09:06.788 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:06.761+0000 7fed18745700 1 -- 172.21.15.84:0/2807550052 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed10059100 con 0x7fed100703d0 2024-02-20T00:09:06.788 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:06.763+0000 7fecfeffd700 1 -- 172.21.15.84:0/2807550052 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7fed0c016070 con 0x7fed100703d0 2024-02-20T00:09:06.840 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:06.832+0000 7fed18745700 1 -- 172.21.15.84:0/2807550052 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fed10102020 con 0x7fed100703d0 2024-02-20T00:09:06.840 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:06.832+0000 7fecfeffd700 1 -- 172.21.15.84:0/2807550052 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1230 (secure 0 0 0) 0x7fed0c01ae80 con 0x7fed100703d0 2024-02-20T00:09:06.840 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:06.840 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: { 2024-02-20T00:09:06.840 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "fsid": "2a2e6c42-cf84-11ee-95bb-87774f69a715", 2024-02-20T00:09:06.840 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "health": { 2024-02-20T00:09:06.840 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-02-20T00:09:06.840 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "checks": {}, 2024-02-20T00:09:06.841 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "mutes": [] 2024-02-20T00:09:06.841 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: }, 2024-02-20T00:09:06.841 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-02-20T00:09:06.841 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "quorum": [ 2024-02-20T00:09:06.841 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 0 2024-02-20T00:09:06.841 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: ], 2024-02-20T00:09:06.841 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "quorum_names": [ 2024-02-20T00:09:06.841 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "smithi084" 2024-02-20T00:09:06.841 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: ], 2024-02-20T00:09:06.841 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "quorum_age": 5, 2024-02-20T00:09:06.841 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "monmap": { 2024-02-20T00:09:06.841 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-20T00:09:06.841 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2024-02-20T00:09:06.842 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "num_mons": 1 2024-02-20T00:09:06.842 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: }, 2024-02-20T00:09:06.842 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "osdmap": { 2024-02-20T00:09:06.842 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-20T00:09:06.842 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "num_osds": 0, 2024-02-20T00:09:06.842 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-02-20T00:09:06.842 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-02-20T00:09:06.842 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-02-20T00:09:06.842 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-02-20T00:09:06.842 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-02-20T00:09:06.842 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: }, 2024-02-20T00:09:06.842 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "pgmap": { 2024-02-20T00:09:06.842 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-02-20T00:09:06.842 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-02-20T00:09:06.842 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "num_pools": 0, 2024-02-20T00:09:06.843 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "num_objects": 0, 2024-02-20T00:09:06.843 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-02-20T00:09:06.843 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-02-20T00:09:06.843 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-02-20T00:09:06.843 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "bytes_total": 0 2024-02-20T00:09:06.843 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: }, 2024-02-20T00:09:06.843 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "fsmap": { 2024-02-20T00:09:06.843 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-20T00:09:06.843 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "by_rank": [], 2024-02-20T00:09:06.843 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "up:standby": 0 2024-02-20T00:09:06.843 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: }, 2024-02-20T00:09:06.843 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "mgrmap": { 2024-02-20T00:09:06.843 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "available": false, 2024-02-20T00:09:06.843 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-02-20T00:09:06.843 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "modules": [ 2024-02-20T00:09:06.844 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "iostat", 2024-02-20T00:09:06.844 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "restful" 2024-02-20T00:09:06.844 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: ], 2024-02-20T00:09:06.844 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "services": {} 2024-02-20T00:09:06.844 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: }, 2024-02-20T00:09:06.844 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "servicemap": { 2024-02-20T00:09:06.844 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-20T00:09:06.844 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "modified": "2024-02-20T00:08:55.999532+0000", 2024-02-20T00:09:06.844 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "services": {} 2024-02-20T00:09:06.844 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: }, 2024-02-20T00:09:06.844 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "progress_events": {} 2024-02-20T00:09:06.844 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: } 2024-02-20T00:09:06.844 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:06.833+0000 7fed18745700 1 -- 172.21.15.84:0/2807550052 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fed100703d0 msgr2=0x7fed1010ce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:09:06.844 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:06.833+0000 7fed18745700 1 --2- 172.21.15.84:0/2807550052 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fed100703d0 0x7fed1010ce20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7fed0c006c70 tx=0x7fed0c006ee0).stop 2024-02-20T00:09:06.845 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:06.833+0000 7fed18745700 1 -- 172.21.15.84:0/2807550052 shutdown_connections 2024-02-20T00:09:06.845 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:06.833+0000 7fed18745700 1 --2- 172.21.15.84:0/2807550052 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fed100703d0 0x7fed1010ce20 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:09:06.845 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:06.833+0000 7fed18745700 1 -- 172.21.15.84:0/2807550052 >> 172.21.15.84:0/2807550052 conn(0x7fed100f6530 msgr2=0x7fed100f7f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:09:06.845 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:06.833+0000 7fed18745700 1 -- 172.21.15.84:0/2807550052 shutdown_connections 2024-02-20T00:09:06.845 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:06.833+0000 7fed18745700 1 -- 172.21.15.84:0/2807550052 wait complete. 2024-02-20T00:09:07.107 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:06 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:09:06.832964+0000 mon.smithi084 (mon.0) 8 : audit [DBG] from='client.? 172.21.15.84:0/2807550052' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-20T00:09:07.362 INFO:teuthology.orchestra.run.smithi084.stderr:mgr not available, waiting (2/15)... 2024-02-20T00:09:08.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:07 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:09:07.732400+0000 mon.smithi084 (mon.0) 9 : cluster [INF] Activating manager daemon smithi084.bmqrnf 2024-02-20T00:09:08.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:07 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:09:07.734923+0000 mon.smithi084 (mon.0) 10 : cluster [DBG] mgrmap e2: smithi084.bmqrnf(active, starting, since 0.00261025s) 2024-02-20T00:09:08.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:07 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:09:07.735872+0000 mon.smithi084 (mon.0) 11 : audit [DBG] from='mgr.14100 172.21.15.84:0/309762878' entity='mgr.smithi084.bmqrnf' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-20T00:09:08.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:07 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:09:07.735998+0000 mon.smithi084 (mon.0) 12 : audit [DBG] from='mgr.14100 172.21.15.84:0/309762878' entity='mgr.smithi084.bmqrnf' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-20T00:09:08.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:07 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:09:07.736105+0000 mon.smithi084 (mon.0) 13 : audit [DBG] from='mgr.14100 172.21.15.84:0/309762878' entity='mgr.smithi084.bmqrnf' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-20T00:09:08.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:07 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:09:07.736222+0000 mon.smithi084 (mon.0) 14 : audit [DBG] from='mgr.14100 172.21.15.84:0/309762878' entity='mgr.smithi084.bmqrnf' cmd=[{"prefix": "mon metadata", "id": "smithi084"}]: dispatch 2024-02-20T00:09:08.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:07 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:09:07.736544+0000 mon.smithi084 (mon.0) 15 : audit [DBG] from='mgr.14100 172.21.15.84:0/309762878' entity='mgr.smithi084.bmqrnf' cmd=[{"prefix": "mgr metadata", "who": "smithi084.bmqrnf", "id": "smithi084.bmqrnf"}]: dispatch 2024-02-20T00:09:08.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:07 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:09:07.738383+0000 mon.smithi084 (mon.0) 16 : cluster [INF] Manager daemon smithi084.bmqrnf is now available 2024-02-20T00:09:08.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:07 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:09:07.745842+0000 mon.smithi084 (mon.0) 17 : audit [INF] from='mgr.14100 172.21.15.84:0/309762878' entity='mgr.smithi084.bmqrnf' 2024-02-20T00:09:08.359 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:07 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:09:07.749479+0000 mon.smithi084 (mon.0) 18 : audit [INF] from='mgr.14100 172.21.15.84:0/309762878' entity='mgr.smithi084.bmqrnf' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi084.bmqrnf/mirror_snapshot_schedule"}]: dispatch 2024-02-20T00:09:08.359 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:07 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:09:07.750502+0000 mon.smithi084 (mon.0) 19 : audit [INF] from='mgr.14100 172.21.15.84:0/309762878' entity='mgr.smithi084.bmqrnf' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi084.bmqrnf/trash_purge_schedule"}]: dispatch 2024-02-20T00:09:08.359 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:07 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:09:07.754881+0000 mon.smithi084 (mon.0) 20 : audit [INF] from='mgr.14100 172.21.15.84:0/309762878' entity='mgr.smithi084.bmqrnf' 2024-02-20T00:09:08.359 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:07 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:09:07.756648+0000 mon.smithi084 (mon.0) 21 : audit [INF] from='mgr.14100 172.21.15.84:0/309762878' entity='mgr.smithi084.bmqrnf' 2024-02-20T00:09:09.365 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi084 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/2a2e6c42-cf84-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp3ot6j0xn:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp4xryone4:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2024-02-20T00:09:09.963 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:09.958+0000 7efed539f700 1 Processor -- start 2024-02-20T00:09:09.963 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:09.958+0000 7efed539f700 1 -- start start 2024-02-20T00:09:09.963 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:09.958+0000 7efed539f700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7efed00704a0 0x7efed00fea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-20T00:09:09.963 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:09.958+0000 7efed539f700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efed00fef40 con 0x7efed00704a0 2024-02-20T00:09:09.963 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:09.959+0000 7efeceffd700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7efed00704a0 0x7efed00fea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:09:09.964 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:09.959+0000 7efeceffd700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7efed00704a0 0x7efed00fea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:36442/0 (socket says 172.21.15.84:36442) 2024-02-20T00:09:09.964 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:09.959+0000 7efeceffd700 1 -- 172.21.15.84:0/2972010142 learned_addr learned my addr 172.21.15.84:0/2972010142 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-02-20T00:09:09.964 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:09.959+0000 7efeceffd700 1 -- 172.21.15.84:0/2972010142 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efed00ff080 con 0x7efed00704a0 2024-02-20T00:09:09.964 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:09.959+0000 7efeceffd700 1 --2- 172.21.15.84:0/2972010142 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7efed00704a0 0x7efed00fea50 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7efeb800b5f0 tx=0x7efeb8006ee0).ready entity=mon.0 client_cookie=c02534471484c71b server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:09:09.964 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:09.959+0000 7efecdffb700 1 -- 172.21.15.84:0/2972010142 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efeb800d600 con 0x7efed00704a0 2024-02-20T00:09:09.964 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:09.959+0000 7efecdffb700 1 -- 172.21.15.84:0/2972010142 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7efeb800dbf0 con 0x7efed00704a0 2024-02-20T00:09:09.964 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:09.959+0000 7efecdffb700 1 -- 172.21.15.84:0/2972010142 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efeb8011cb0 con 0x7efed00704a0 2024-02-20T00:09:09.964 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:09.960+0000 7efed539f700 1 -- 172.21.15.84:0/2972010142 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7efed00704a0 msgr2=0x7efed00fea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:09:09.964 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:09.960+0000 7efed539f700 1 --2- 172.21.15.84:0/2972010142 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7efed00704a0 0x7efed00fea50 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7efeb800b5f0 tx=0x7efeb8006ee0).stop 2024-02-20T00:09:09.964 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:09.960+0000 7efed539f700 1 -- 172.21.15.84:0/2972010142 shutdown_connections 2024-02-20T00:09:09.964 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:09.960+0000 7efed539f700 1 --2- 172.21.15.84:0/2972010142 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7efed00704a0 0x7efed00fea50 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:09:09.964 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:09.960+0000 7efed539f700 1 -- 172.21.15.84:0/2972010142 >> 172.21.15.84:0/2972010142 conn(0x7efed00f6530 msgr2=0x7efed00f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:09:09.964 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:09.960+0000 7efed539f700 1 -- 172.21.15.84:0/2972010142 shutdown_connections 2024-02-20T00:09:09.965 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:09.960+0000 7efed539f700 1 -- 172.21.15.84:0/2972010142 wait complete. 2024-02-20T00:09:09.967 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:09.961+0000 7efed539f700 1 Processor -- start 2024-02-20T00:09:09.967 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:09.961+0000 7efed539f700 1 -- start start 2024-02-20T00:09:09.967 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:09.961+0000 7efed539f700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7efed00704a0 0x7efed010ab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:09:09.967 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:09.961+0000 7efed539f700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efed010b040 con 0x7efed00704a0 2024-02-20T00:09:09.967 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:09.961+0000 7efeceffd700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7efed00704a0 0x7efed010ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:09:09.967 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:09.961+0000 7efeceffd700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7efed00704a0 0x7efed010ab50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:36452/0 (socket says 172.21.15.84:36452) 2024-02-20T00:09:09.967 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:09.962+0000 7efeceffd700 1 -- 172.21.15.84:0/1242827431 learned_addr learned my addr 172.21.15.84:0/1242827431 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-02-20T00:09:09.967 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:09.962+0000 7efeceffd700 1 -- 172.21.15.84:0/1242827431 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efeb800b200 con 0x7efed00704a0 2024-02-20T00:09:09.967 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:09.962+0000 7efeceffd700 1 --2- 172.21.15.84:0/1242827431 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7efed00704a0 0x7efed010ab50 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7efeb8006800 tx=0x7efeb8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:09:09.967 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:09.962+0000 7efec77fe700 1 -- 172.21.15.84:0/1242827431 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efeb801b570 con 0x7efed00704a0 2024-02-20T00:09:09.968 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:09.962+0000 7efec77fe700 1 -- 172.21.15.84:0/1242827431 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7efeb801bb60 con 0x7efed00704a0 2024-02-20T00:09:09.968 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:09.962+0000 7efed539f700 1 -- 172.21.15.84:0/1242827431 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efed010b240 con 0x7efed00704a0 2024-02-20T00:09:09.968 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:09.962+0000 7efec77fe700 1 -- 172.21.15.84:0/1242827431 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efeb801a9f0 con 0x7efed00704a0 2024-02-20T00:09:09.968 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:09.962+0000 7efed539f700 1 -- 172.21.15.84:0/1242827431 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efed010b6d0 con 0x7efed00704a0 2024-02-20T00:09:09.968 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:09.963+0000 7efec77fe700 1 -- 172.21.15.84:0/1242827431 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 3) v1 ==== 36029+0+0 (secure 0 0 0) 0x7efeb800f070 con 0x7efed00704a0 2024-02-20T00:09:09.968 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:09.963+0000 7efed539f700 1 -- 172.21.15.84:0/1242827431 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efed0045ad0 con 0x7efed00704a0 2024-02-20T00:09:09.968 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:09.963+0000 7efec77fe700 1 --2- 172.21.15.84:0/1242827431 >> [v2:172.21.15.84:6800/2888547837,v1:172.21.15.84:6801/2888547837] conn(0x7efeb002fd60 0x7efeb0032220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:09:09.968 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:09.963+0000 7efec77fe700 1 -- 172.21.15.84:0/1242827431 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7efeb80411a0 con 0x7efed00704a0 2024-02-20T00:09:09.968 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:09.963+0000 7efece7fc700 1 --2- 172.21.15.84:0/1242827431 >> [v2:172.21.15.84:6800/2888547837,v1:172.21.15.84:6801/2888547837] conn(0x7efeb002fd60 0x7efeb0032220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:09:09.968 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:09.964+0000 7efece7fc700 1 --2- 172.21.15.84:0/1242827431 >> [v2:172.21.15.84:6800/2888547837,v1:172.21.15.84:6801/2888547837] conn(0x7efeb002fd60 0x7efeb0032220 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7efec00096a0 tx=0x7efec0006b40).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:09:09.970 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:09.968+0000 7efec77fe700 1 -- 172.21.15.84:0/1242827431 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7efeb8019cd0 con 0x7efed00704a0 2024-02-20T00:09:10.107 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:09 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:09:08.738584+0000 mon.smithi084 (mon.0) 22 : cluster [DBG] mgrmap e3: smithi084.bmqrnf(active, since 1.00627s) 2024-02-20T00:09:10.178 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:10.169+0000 7efed539f700 1 -- 172.21.15.84:0/1242827431 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7efed00ffdc0 con 0x7efed00704a0 2024-02-20T00:09:10.178 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:10.170+0000 7efec77fe700 1 -- 172.21.15.84:0/1242827431 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1229 (secure 0 0 0) 0x7efeb8016070 con 0x7efed00704a0 2024-02-20T00:09:10.178 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:10.179 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: { 2024-02-20T00:09:10.179 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "fsid": "2a2e6c42-cf84-11ee-95bb-87774f69a715", 2024-02-20T00:09:10.179 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "health": { 2024-02-20T00:09:10.179 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-02-20T00:09:10.179 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "checks": {}, 2024-02-20T00:09:10.179 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "mutes": [] 2024-02-20T00:09:10.179 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: }, 2024-02-20T00:09:10.179 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-02-20T00:09:10.179 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "quorum": [ 2024-02-20T00:09:10.179 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 0 2024-02-20T00:09:10.179 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: ], 2024-02-20T00:09:10.179 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "quorum_names": [ 2024-02-20T00:09:10.179 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "smithi084" 2024-02-20T00:09:10.179 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: ], 2024-02-20T00:09:10.179 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "quorum_age": 9, 2024-02-20T00:09:10.180 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "monmap": { 2024-02-20T00:09:10.180 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-20T00:09:10.180 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2024-02-20T00:09:10.180 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "num_mons": 1 2024-02-20T00:09:10.180 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: }, 2024-02-20T00:09:10.180 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "osdmap": { 2024-02-20T00:09:10.180 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-20T00:09:10.180 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "num_osds": 0, 2024-02-20T00:09:10.180 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-02-20T00:09:10.180 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-02-20T00:09:10.180 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-02-20T00:09:10.180 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-02-20T00:09:10.180 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-02-20T00:09:10.180 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: }, 2024-02-20T00:09:10.180 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "pgmap": { 2024-02-20T00:09:10.181 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-02-20T00:09:10.181 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-02-20T00:09:10.181 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "num_pools": 0, 2024-02-20T00:09:10.181 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "num_objects": 0, 2024-02-20T00:09:10.181 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-02-20T00:09:10.181 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-02-20T00:09:10.181 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-02-20T00:09:10.181 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "bytes_total": 0 2024-02-20T00:09:10.181 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: }, 2024-02-20T00:09:10.181 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "fsmap": { 2024-02-20T00:09:10.181 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-20T00:09:10.181 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "by_rank": [], 2024-02-20T00:09:10.181 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "up:standby": 0 2024-02-20T00:09:10.181 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: }, 2024-02-20T00:09:10.181 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "mgrmap": { 2024-02-20T00:09:10.182 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "available": true, 2024-02-20T00:09:10.182 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-02-20T00:09:10.182 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "modules": [ 2024-02-20T00:09:10.182 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "iostat", 2024-02-20T00:09:10.182 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "restful" 2024-02-20T00:09:10.182 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: ], 2024-02-20T00:09:10.182 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "services": {} 2024-02-20T00:09:10.182 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: }, 2024-02-20T00:09:10.182 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "servicemap": { 2024-02-20T00:09:10.182 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-20T00:09:10.182 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "modified": "2024-02-20T00:08:55.999532+0000", 2024-02-20T00:09:10.182 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "services": {} 2024-02-20T00:09:10.182 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: }, 2024-02-20T00:09:10.182 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "progress_events": {} 2024-02-20T00:09:10.182 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: } 2024-02-20T00:09:10.182 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:10.171+0000 7efed539f700 1 -- 172.21.15.84:0/1242827431 >> [v2:172.21.15.84:6800/2888547837,v1:172.21.15.84:6801/2888547837] conn(0x7efeb002fd60 msgr2=0x7efeb0032220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:09:10.183 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:10.171+0000 7efed539f700 1 --2- 172.21.15.84:0/1242827431 >> [v2:172.21.15.84:6800/2888547837,v1:172.21.15.84:6801/2888547837] conn(0x7efeb002fd60 0x7efeb0032220 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7efec00096a0 tx=0x7efec0006b40).stop 2024-02-20T00:09:10.183 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:10.171+0000 7efed539f700 1 -- 172.21.15.84:0/1242827431 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7efed00704a0 msgr2=0x7efed010ab50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:09:10.183 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:10.171+0000 7efed539f700 1 --2- 172.21.15.84:0/1242827431 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7efed00704a0 0x7efed010ab50 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7efeb8006800 tx=0x7efeb8006ee0).stop 2024-02-20T00:09:10.183 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:10.171+0000 7efed539f700 1 -- 172.21.15.84:0/1242827431 shutdown_connections 2024-02-20T00:09:10.183 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:10.171+0000 7efed539f700 1 --2- 172.21.15.84:0/1242827431 >> [v2:172.21.15.84:6800/2888547837,v1:172.21.15.84:6801/2888547837] conn(0x7efeb002fd60 0x7efeb0032220 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:09:10.183 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:10.171+0000 7efed539f700 1 --2- 172.21.15.84:0/1242827431 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7efed00704a0 0x7efed010ab50 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:09:10.183 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:10.171+0000 7efed539f700 1 -- 172.21.15.84:0/1242827431 >> 172.21.15.84:0/1242827431 conn(0x7efed00f6530 msgr2=0x7efed00f7ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:09:10.183 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:10.172+0000 7efed539f700 1 -- 172.21.15.84:0/1242827431 shutdown_connections 2024-02-20T00:09:10.183 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:10.172+0000 7efed539f700 1 -- 172.21.15.84:0/1242827431 wait complete. 2024-02-20T00:09:10.825 INFO:teuthology.orchestra.run.smithi084.stderr:mgr is available 2024-02-20T00:09:10.825 INFO:teuthology.orchestra.run.smithi084.stderr:Enabling cephadm module... 2024-02-20T00:09:10.825 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi084 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/2a2e6c42-cf84-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp3ot6j0xn:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp4xryone4:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable cephadm 2024-02-20T00:09:10.827 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:10 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:09:10.171587+0000 mon.smithi084 (mon.0) 23 : audit [DBG] from='client.? 172.21.15.84:0/1242827431' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-20T00:09:11.355 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:11.350+0000 7fdde98bf700 1 Processor -- start 2024-02-20T00:09:11.355 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:11.350+0000 7fdde98bf700 1 -- start start 2024-02-20T00:09:11.355 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:11.350+0000 7fdde98bf700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdde40fbd90 0x7fdde40fc1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-20T00:09:11.355 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:11.350+0000 7fdde98bf700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdde40fc6a0 con 0x7fdde40fbd90 2024-02-20T00:09:11.355 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:11.351+0000 7fdde88bd700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdde40fbd90 0x7fdde40fc1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:09:11.356 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:11.351+0000 7fdde88bd700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdde40fbd90 0x7fdde40fc1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:36458/0 (socket says 172.21.15.84:36458) 2024-02-20T00:09:11.356 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:11.351+0000 7fdde88bd700 1 -- 172.21.15.84:0/1435554986 learned_addr learned my addr 172.21.15.84:0/1435554986 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-02-20T00:09:11.356 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:11.351+0000 7fdde88bd700 1 -- 172.21.15.84:0/1435554986 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdde40fc7e0 con 0x7fdde40fbd90 2024-02-20T00:09:11.356 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:11.351+0000 7fdde88bd700 1 --2- 172.21.15.84:0/1435554986 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdde40fbd90 0x7fdde40fc1b0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7fdddc009960 tx=0x7fdddc0092d0).ready entity=mon.0 client_cookie=c67659746dffd53d server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:09:11.356 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:11.351+0000 7fdde37fe700 1 -- 172.21.15.84:0/1435554986 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdddc00b9c0 con 0x7fdde40fbd90 2024-02-20T00:09:11.356 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:11.351+0000 7fdde37fe700 1 -- 172.21.15.84:0/1435554986 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fdddc011400 con 0x7fdde40fbd90 2024-02-20T00:09:11.356 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:11.351+0000 7fdde37fe700 1 -- 172.21.15.84:0/1435554986 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdddc010580 con 0x7fdde40fbd90 2024-02-20T00:09:11.356 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:11.352+0000 7fdde98bf700 1 -- 172.21.15.84:0/1435554986 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdde40fbd90 msgr2=0x7fdde40fc1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:09:11.356 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:11.352+0000 7fdde98bf700 1 --2- 172.21.15.84:0/1435554986 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdde40fbd90 0x7fdde40fc1b0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7fdddc009960 tx=0x7fdddc0092d0).stop 2024-02-20T00:09:11.356 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:11.352+0000 7fdde98bf700 1 -- 172.21.15.84:0/1435554986 shutdown_connections 2024-02-20T00:09:11.356 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:11.352+0000 7fdde98bf700 1 --2- 172.21.15.84:0/1435554986 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdde40fbd90 0x7fdde40fc1b0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:09:11.356 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:11.352+0000 7fdde98bf700 1 -- 172.21.15.84:0/1435554986 >> 172.21.15.84:0/1435554986 conn(0x7fdde40f7510 msgr2=0x7fdde40f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:09:11.356 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:11.352+0000 7fdde98bf700 1 -- 172.21.15.84:0/1435554986 shutdown_connections 2024-02-20T00:09:11.356 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:11.352+0000 7fdde98bf700 1 -- 172.21.15.84:0/1435554986 wait complete. 2024-02-20T00:09:11.357 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:11.353+0000 7fdde98bf700 1 Processor -- start 2024-02-20T00:09:11.357 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:11.353+0000 7fdde98bf700 1 -- start start 2024-02-20T00:09:11.359 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:11.353+0000 7fdde98bf700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdde40fbd90 0x7fdde410a930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:09:11.359 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:11.353+0000 7fdde98bf700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdde410ae20 con 0x7fdde40fbd90 2024-02-20T00:09:11.359 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:11.353+0000 7fdde88bd700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdde40fbd90 0x7fdde410a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:09:11.359 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:11.353+0000 7fdde88bd700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdde40fbd90 0x7fdde410a930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:36466/0 (socket says 172.21.15.84:36466) 2024-02-20T00:09:11.359 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:11.353+0000 7fdde88bd700 1 -- 172.21.15.84:0/2102280813 learned_addr learned my addr 172.21.15.84:0/2102280813 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-02-20T00:09:11.359 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:11.353+0000 7fdde88bd700 1 -- 172.21.15.84:0/2102280813 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdddc009570 con 0x7fdde40fbd90 2024-02-20T00:09:11.359 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:11.354+0000 7fdde88bd700 1 --2- 172.21.15.84:0/2102280813 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdde40fbd90 0x7fdde410a930 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7fdddc00b5c0 tx=0x7fdddc011f90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:09:11.359 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:11.354+0000 7fdde17fa700 1 -- 172.21.15.84:0/2102280813 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdddc0107f0 con 0x7fdde40fbd90 2024-02-20T00:09:11.359 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:11.354+0000 7fdde17fa700 1 -- 172.21.15.84:0/2102280813 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fdddc010de0 con 0x7fdde40fbd90 2024-02-20T00:09:11.359 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:11.354+0000 7fdde17fa700 1 -- 172.21.15.84:0/2102280813 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdddc01ad50 con 0x7fdde40fbd90 2024-02-20T00:09:11.359 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:11.354+0000 7fdde98bf700 1 -- 172.21.15.84:0/2102280813 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdde410b020 con 0x7fdde40fbd90 2024-02-20T00:09:11.359 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:11.354+0000 7fdde98bf700 1 -- 172.21.15.84:0/2102280813 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdde410b4b0 con 0x7fdde40fbd90 2024-02-20T00:09:11.359 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:11.354+0000 7fdde17fa700 1 -- 172.21.15.84:0/2102280813 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 4) v1 ==== 36109+0+0 (secure 0 0 0) 0x7fdddc00f070 con 0x7fdde40fbd90 2024-02-20T00:09:11.359 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:11.355+0000 7fdde17fa700 1 --2- 172.21.15.84:0/2102280813 >> [v2:172.21.15.84:6800/2888547837,v1:172.21.15.84:6801/2888547837] conn(0x7fddd002fe10 0x7fddd00322d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:09:11.360 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:11.355+0000 7fdde17fa700 1 -- 172.21.15.84:0/2102280813 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fdddc042920 con 0x7fdde40fbd90 2024-02-20T00:09:11.360 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:11.355+0000 7fdde3fff700 1 --2- 172.21.15.84:0/2102280813 >> [v2:172.21.15.84:6800/2888547837,v1:172.21.15.84:6801/2888547837] conn(0x7fddd002fe10 0x7fddd00322d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:09:11.360 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:11.355+0000 7fdde98bf700 1 -- 172.21.15.84:0/2102280813 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdde4045ad0 con 0x7fdde40fbd90 2024-02-20T00:09:11.360 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:11.355+0000 7fdde3fff700 1 --2- 172.21.15.84:0/2102280813 >> [v2:172.21.15.84:6800/2888547837,v1:172.21.15.84:6801/2888547837] conn(0x7fddd002fe10 0x7fddd00322d0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fddd80096a0 tx=0x7fddd8006b40).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:09:11.361 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:11.359+0000 7fdde17fa700 1 -- 172.21.15.84:0/2102280813 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdddc01e070 con 0x7fdde40fbd90 2024-02-20T00:09:11.545 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:11.543+0000 7fdde98bf700 1 -- 172.21.15.84:0/2102280813 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7fdde41049c0 con 0x7fdde40fbd90 2024-02-20T00:09:11.826 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:11.823+0000 7fdde17fa700 1 -- 172.21.15.84:0/2102280813 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mgrmap(e 5) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fdddc023c60 con 0x7fdde40fbd90 2024-02-20T00:09:11.827 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:11.824+0000 7fdde17fa700 1 -- 172.21.15.84:0/2102280813 <== mon.0 v2:172.21.15.84:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7fdddc006030 con 0x7fdde40fbd90 2024-02-20T00:09:11.829 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:11.826+0000 7fdde98bf700 1 -- 172.21.15.84:0/2102280813 >> [v2:172.21.15.84:6800/2888547837,v1:172.21.15.84:6801/2888547837] conn(0x7fddd002fe10 msgr2=0x7fddd00322d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:09:11.829 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:11.826+0000 7fdde98bf700 1 --2- 172.21.15.84:0/2102280813 >> [v2:172.21.15.84:6800/2888547837,v1:172.21.15.84:6801/2888547837] conn(0x7fddd002fe10 0x7fddd00322d0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fddd80096a0 tx=0x7fddd8006b40).stop 2024-02-20T00:09:11.830 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:11.826+0000 7fdde98bf700 1 -- 172.21.15.84:0/2102280813 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdde40fbd90 msgr2=0x7fdde410a930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:09:11.830 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:11.826+0000 7fdde98bf700 1 --2- 172.21.15.84:0/2102280813 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdde40fbd90 0x7fdde410a930 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7fdddc00b5c0 tx=0x7fdddc011f90).stop 2024-02-20T00:09:11.830 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:11.826+0000 7fdde98bf700 1 -- 172.21.15.84:0/2102280813 shutdown_connections 2024-02-20T00:09:11.830 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:11.826+0000 7fdde98bf700 1 --2- 172.21.15.84:0/2102280813 >> [v2:172.21.15.84:6800/2888547837,v1:172.21.15.84:6801/2888547837] conn(0x7fddd002fe10 0x7fddd00322d0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:09:11.830 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:11.826+0000 7fdde98bf700 1 --2- 172.21.15.84:0/2102280813 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdde40fbd90 0x7fdde410a930 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:09:11.830 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:11.826+0000 7fdde98bf700 1 -- 172.21.15.84:0/2102280813 >> 172.21.15.84:0/2102280813 conn(0x7fdde40f7510 msgr2=0x7fdde40f8e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:09:11.830 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:11.826+0000 7fdde98bf700 1 -- 172.21.15.84:0/2102280813 shutdown_connections 2024-02-20T00:09:11.830 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:11.826+0000 7fdde98bf700 1 -- 172.21.15.84:0/2102280813 wait complete. 2024-02-20T00:09:12.107 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:11 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:09:10.737997+0000 mon.smithi084 (mon.0) 24 : cluster [DBG] mgrmap e4: smithi084.bmqrnf(active, since 3s) 2024-02-20T00:09:12.108 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:11 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:09:11.545125+0000 mon.smithi084 (mon.0) 25 : audit [INF] from='client.? 172.21.15.84:0/2102280813' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-02-20T00:09:12.548 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi084 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/2a2e6c42-cf84-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp3ot6j0xn:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp4xryone4:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr stat 2024-02-20T00:09:13.108 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:12 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:09:11.826028+0000 mon.smithi084 (mon.0) 26 : audit [INF] from='client.? 172.21.15.84:0/2102280813' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-02-20T00:09:13.108 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:12 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:09:11.826109+0000 mon.smithi084 (mon.0) 27 : cluster [DBG] mgrmap e5: smithi084.bmqrnf(active, since 4s) 2024-02-20T00:09:13.169 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:13.165+0000 7f09b9b59700 1 Processor -- start 2024-02-20T00:09:13.169 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:13.165+0000 7f09b9b59700 1 -- start start 2024-02-20T00:09:13.169 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:13.165+0000 7f09b9b59700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f09b40703b0 0x7f09b4100ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-20T00:09:13.169 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:13.165+0000 7f09b9b59700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09b40fc8c0 con 0x7f09b40703b0 2024-02-20T00:09:13.169 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:13.166+0000 7f09b37fe700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f09b40703b0 0x7f09b4100ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:09:13.170 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:13.166+0000 7f09b37fe700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f09b40703b0 0x7f09b4100ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:36488/0 (socket says 172.21.15.84:36488) 2024-02-20T00:09:13.170 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:13.166+0000 7f09b37fe700 1 -- 172.21.15.84:0/2788463600 learned_addr learned my addr 172.21.15.84:0/2788463600 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-02-20T00:09:13.170 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:13.166+0000 7f09b37fe700 1 -- 172.21.15.84:0/2788463600 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09b41013c0 con 0x7f09b40703b0 2024-02-20T00:09:13.170 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:13.166+0000 7f09b37fe700 1 --2- 172.21.15.84:0/2788463600 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f09b40703b0 0x7f09b4100ed0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f09a400b5f0 tx=0x7f09a4006ee0).ready entity=mon.0 client_cookie=e046e25b9a55f923 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:09:13.170 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:13.166+0000 7f09b27fc700 1 -- 172.21.15.84:0/2788463600 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f09a400d600 con 0x7f09b40703b0 2024-02-20T00:09:13.172 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:13.166+0000 7f09b27fc700 1 -- 172.21.15.84:0/2788463600 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f09a400dbf0 con 0x7f09b40703b0 2024-02-20T00:09:13.172 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:13.166+0000 7f09b27fc700 1 -- 172.21.15.84:0/2788463600 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f09a4011cb0 con 0x7f09b40703b0 2024-02-20T00:09:13.172 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:13.167+0000 7f09b9b59700 1 -- 172.21.15.84:0/2788463600 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f09b40703b0 msgr2=0x7f09b4100ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:09:13.172 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:13.167+0000 7f09b9b59700 1 --2- 172.21.15.84:0/2788463600 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f09b40703b0 0x7f09b4100ed0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f09a400b5f0 tx=0x7f09a4006ee0).stop 2024-02-20T00:09:13.172 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:13.167+0000 7f09b9b59700 1 -- 172.21.15.84:0/2788463600 shutdown_connections 2024-02-20T00:09:13.172 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:13.167+0000 7f09b9b59700 1 --2- 172.21.15.84:0/2788463600 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f09b40703b0 0x7f09b4100ed0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:09:13.172 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:13.167+0000 7f09b9b59700 1 -- 172.21.15.84:0/2788463600 >> 172.21.15.84:0/2788463600 conn(0x7f09b40f6530 msgr2=0x7f09b40f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:09:13.172 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:13.167+0000 7f09b9b59700 1 -- 172.21.15.84:0/2788463600 shutdown_connections 2024-02-20T00:09:13.172 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:13.168+0000 7f09b9b59700 1 -- 172.21.15.84:0/2788463600 wait complete. 2024-02-20T00:09:13.172 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:13.168+0000 7f09b9b59700 1 Processor -- start 2024-02-20T00:09:13.172 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:13.168+0000 7f09b9b59700 1 -- start start 2024-02-20T00:09:13.172 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:13.168+0000 7f09b9b59700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f09b40703b0 0x7f09b410cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:09:13.173 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:13.168+0000 7f09b9b59700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09b410d2c0 con 0x7f09b40703b0 2024-02-20T00:09:13.173 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:13.168+0000 7f09b37fe700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f09b40703b0 0x7f09b410cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:09:13.173 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:13.168+0000 7f09b37fe700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f09b40703b0 0x7f09b410cdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:36498/0 (socket says 172.21.15.84:36498) 2024-02-20T00:09:13.173 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:13.169+0000 7f09b37fe700 1 -- 172.21.15.84:0/1803582166 learned_addr learned my addr 172.21.15.84:0/1803582166 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-02-20T00:09:13.173 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:13.169+0000 7f09b37fe700 1 -- 172.21.15.84:0/1803582166 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09a400b200 con 0x7f09b40703b0 2024-02-20T00:09:13.173 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:13.169+0000 7f09b37fe700 1 --2- 172.21.15.84:0/1803582166 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f09b40703b0 0x7f09b410cdd0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f09a400df90 tx=0x7f09a4006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:09:13.173 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:13.169+0000 7f099bfff700 1 -- 172.21.15.84:0/1803582166 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f09a401b410 con 0x7f09b40703b0 2024-02-20T00:09:13.173 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:13.169+0000 7f09b9b59700 1 -- 172.21.15.84:0/1803582166 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09b410d4c0 con 0x7f09b40703b0 2024-02-20T00:09:13.174 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:13.169+0000 7f099bfff700 1 -- 172.21.15.84:0/1803582166 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f09a401ba00 con 0x7f09b40703b0 2024-02-20T00:09:13.174 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:13.169+0000 7f099bfff700 1 -- 172.21.15.84:0/1803582166 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f09a401aa30 con 0x7f09b40703b0 2024-02-20T00:09:13.174 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:13.169+0000 7f09b9b59700 1 -- 172.21.15.84:0/1803582166 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09b410e100 con 0x7f09b40703b0 2024-02-20T00:09:13.174 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:13.170+0000 7f099bfff700 1 -- 172.21.15.84:0/1803582166 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 5) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f09a400f070 con 0x7f09b40703b0 2024-02-20T00:09:13.174 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:13.170+0000 7f099bfff700 1 --2- 172.21.15.84:0/1803582166 >> [v2:172.21.15.84:6800/2888547837,v1:172.21.15.84:6801/2888547837] conn(0x7f09a002fe10 0x7f09a00322d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:09:13.174 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:13.170+0000 7f099bfff700 1 -- 172.21.15.84:0/1803582166 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f09a4041260 con 0x7f09b40703b0 2024-02-20T00:09:13.175 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:13.170+0000 7f09b2ffd700 1 -- 172.21.15.84:0/1803582166 >> [v2:172.21.15.84:6800/2888547837,v1:172.21.15.84:6801/2888547837] conn(0x7f09a002fe10 msgr2=0x7f09a00322d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.84:6800/2888547837 2024-02-20T00:09:13.175 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:13.170+0000 7f09b2ffd700 1 --2- 172.21.15.84:0/1803582166 >> [v2:172.21.15.84:6800/2888547837,v1:172.21.15.84:6801/2888547837] conn(0x7f09a002fe10 0x7f09a00322d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-02-20T00:09:13.175 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:13.170+0000 7f09b9b59700 1 -- 172.21.15.84:0/1803582166 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09b410d950 con 0x7f09b40703b0 2024-02-20T00:09:13.177 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:13.175+0000 7f099bfff700 1 -- 172.21.15.84:0/1803582166 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f09a4016070 con 0x7f09b40703b0 2024-02-20T00:09:13.373 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:13.370+0000 7f09b9b59700 1 -- 172.21.15.84:0/1803582166 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f09b41020e0 con 0x7f09b40703b0 2024-02-20T00:09:13.374 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:13.370+0000 7f099bfff700 1 -- 172.21.15.84:0/1803582166 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+103 (secure 0 0 0) 0x7f09a4022d20 con 0x7f09b40703b0 2024-02-20T00:09:13.374 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:13.371+0000 7f09b2ffd700 1 -- 172.21.15.84:0/1803582166 >> [v2:172.21.15.84:6800/2888547837,v1:172.21.15.84:6801/2888547837] conn(0x7f09a002fe10 msgr2=0x7f09a00322d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.84:6800/2888547837 2024-02-20T00:09:13.374 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:13.371+0000 7f09b2ffd700 1 --2- 172.21.15.84:0/1803582166 >> [v2:172.21.15.84:6800/2888547837,v1:172.21.15.84:6801/2888547837] conn(0x7f09a002fe10 0x7f09a00322d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2024-02-20T00:09:13.374 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: { 2024-02-20T00:09:13.374 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "epoch": 5, 2024-02-20T00:09:13.374 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "available": true, 2024-02-20T00:09:13.374 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "active_name": "smithi084.bmqrnf", 2024-02-20T00:09:13.374 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "num_standby": 0 2024-02-20T00:09:13.374 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: } 2024-02-20T00:09:13.375 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:13.372+0000 7f09b9b59700 1 -- 172.21.15.84:0/1803582166 >> [v2:172.21.15.84:6800/2888547837,v1:172.21.15.84:6801/2888547837] conn(0x7f09a002fe10 msgr2=0x7f09a00322d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-20T00:09:13.375 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:13.372+0000 7f09b9b59700 1 --2- 172.21.15.84:0/1803582166 >> [v2:172.21.15.84:6800/2888547837,v1:172.21.15.84:6801/2888547837] conn(0x7f09a002fe10 0x7f09a00322d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-20T00:09:13.376 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:13.372+0000 7f09b9b59700 1 -- 172.21.15.84:0/1803582166 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f09b40703b0 msgr2=0x7f09b410cdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:09:13.376 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:13.372+0000 7f09b9b59700 1 --2- 172.21.15.84:0/1803582166 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f09b40703b0 0x7f09b410cdd0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f09a400df90 tx=0x7f09a4006ee0).stop 2024-02-20T00:09:13.376 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:13.372+0000 7f09b9b59700 1 -- 172.21.15.84:0/1803582166 shutdown_connections 2024-02-20T00:09:13.376 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:13.372+0000 7f09b9b59700 1 --2- 172.21.15.84:0/1803582166 >> [v2:172.21.15.84:6800/2888547837,v1:172.21.15.84:6801/2888547837] conn(0x7f09a002fe10 0x7f09a00322d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-20T00:09:13.376 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:13.372+0000 7f09b9b59700 1 --2- 172.21.15.84:0/1803582166 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f09b40703b0 0x7f09b410cdd0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:09:13.376 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:13.372+0000 7f09b9b59700 1 -- 172.21.15.84:0/1803582166 >> 172.21.15.84:0/1803582166 conn(0x7f09b40f6530 msgr2=0x7f09b40f7fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:09:13.376 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:13.373+0000 7f09b9b59700 1 -- 172.21.15.84:0/1803582166 shutdown_connections 2024-02-20T00:09:13.376 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:13.373+0000 7f09b9b59700 1 -- 172.21.15.84:0/1803582166 wait complete. 2024-02-20T00:09:13.946 INFO:teuthology.orchestra.run.smithi084.stderr:Waiting for the mgr to restart... 2024-02-20T00:09:13.946 INFO:teuthology.orchestra.run.smithi084.stderr:Waiting for mgr epoch 5... 2024-02-20T00:09:13.946 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi084 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/2a2e6c42-cf84-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp3ot6j0xn:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp4xryone4:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 tell mgr mgr_status 2024-02-20T00:09:13.948 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:13 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:09:13.372345+0000 mon.smithi084 (mon.0) 28 : audit [DBG] from='client.? 172.21.15.84:0/1803582166' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-02-20T00:09:15.894 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:14.775+0000 7fafccf18700 1 Processor -- start 2024-02-20T00:09:15.895 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:14.776+0000 7fafccf18700 1 -- start start 2024-02-20T00:09:15.895 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:14.776+0000 7fafccf18700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fafc80fad90 0x7fafc80fb1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-20T00:09:15.895 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:14.776+0000 7fafccf18700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafc80fb6a0 con 0x7fafc80fad90 2024-02-20T00:09:15.895 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:14.776+0000 7fafc659c700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fafc80fad90 0x7fafc80fb1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:09:15.895 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:14.776+0000 7fafc659c700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fafc80fad90 0x7fafc80fb1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:36512/0 (socket says 172.21.15.84:36512) 2024-02-20T00:09:15.895 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:14.776+0000 7fafc659c700 1 -- 172.21.15.84:0/2948161286 learned_addr learned my addr 172.21.15.84:0/2948161286 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-02-20T00:09:15.895 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:14.776+0000 7fafc659c700 1 -- 172.21.15.84:0/2948161286 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafc80fb7e0 con 0x7fafc80fad90 2024-02-20T00:09:15.895 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:14.776+0000 7fafc659c700 1 --2- 172.21.15.84:0/2948161286 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fafc80fad90 0x7fafc80fb1b0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7fafb800b5f0 tx=0x7fafb8006ee0).ready entity=mon.0 client_cookie=574fd65197c10837 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:09:15.895 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:14.776+0000 7fafc559a700 1 -- 172.21.15.84:0/2948161286 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fafb800d600 con 0x7fafc80fad90 2024-02-20T00:09:15.895 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:14.777+0000 7fafc559a700 1 -- 172.21.15.84:0/2948161286 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fafb800dbf0 con 0x7fafc80fad90 2024-02-20T00:09:15.895 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:14.777+0000 7fafc559a700 1 -- 172.21.15.84:0/2948161286 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fafb8011cb0 con 0x7fafc80fad90 2024-02-20T00:09:15.895 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:14.777+0000 7fafccf18700 1 -- 172.21.15.84:0/2948161286 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fafc80fad90 msgr2=0x7fafc80fb1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:09:15.895 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:14.777+0000 7fafccf18700 1 --2- 172.21.15.84:0/2948161286 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fafc80fad90 0x7fafc80fb1b0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7fafb800b5f0 tx=0x7fafb8006ee0).stop 2024-02-20T00:09:15.895 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:14.777+0000 7fafccf18700 1 -- 172.21.15.84:0/2948161286 shutdown_connections 2024-02-20T00:09:15.896 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:14.777+0000 7fafccf18700 1 --2- 172.21.15.84:0/2948161286 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fafc80fad90 0x7fafc80fb1b0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:09:15.896 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:14.777+0000 7fafccf18700 1 -- 172.21.15.84:0/2948161286 >> 172.21.15.84:0/2948161286 conn(0x7fafc80f6530 msgr2=0x7fafc80f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:09:15.896 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:14.777+0000 7fafccf18700 1 -- 172.21.15.84:0/2948161286 shutdown_connections 2024-02-20T00:09:15.896 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:14.777+0000 7fafccf18700 1 -- 172.21.15.84:0/2948161286 wait complete. 2024-02-20T00:09:15.896 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:14.778+0000 7fafccf18700 1 Processor -- start 2024-02-20T00:09:15.896 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:14.778+0000 7fafccf18700 1 -- start start 2024-02-20T00:09:15.896 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:14.778+0000 7fafccf18700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fafc80fad90 0x7fafc8109a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:09:15.896 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:14.778+0000 7fafccf18700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafc8109f20 con 0x7fafc80fad90 2024-02-20T00:09:15.896 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:14.778+0000 7fafc659c700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fafc80fad90 0x7fafc8109a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:09:15.896 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:14.778+0000 7fafc659c700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fafc80fad90 0x7fafc8109a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:36514/0 (socket says 172.21.15.84:36514) 2024-02-20T00:09:15.896 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:14.778+0000 7fafc659c700 1 -- 172.21.15.84:0/3892269001 learned_addr learned my addr 172.21.15.84:0/3892269001 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-02-20T00:09:15.896 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:14.778+0000 7fafc659c700 1 -- 172.21.15.84:0/3892269001 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafb800b200 con 0x7fafc80fad90 2024-02-20T00:09:15.896 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:14.779+0000 7fafc659c700 1 --2- 172.21.15.84:0/3892269001 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fafc80fad90 0x7fafc8109a30 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7fafb8002f40 tx=0x7fafb8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:09:15.896 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:14.779+0000 7fafb6ffd700 1 -- 172.21.15.84:0/3892269001 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fafb801b410 con 0x7fafc80fad90 2024-02-20T00:09:15.897 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:14.779+0000 7fafb6ffd700 1 -- 172.21.15.84:0/3892269001 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fafb801ba00 con 0x7fafc80fad90 2024-02-20T00:09:15.897 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:14.779+0000 7fafccf18700 1 -- 172.21.15.84:0/3892269001 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafc810a120 con 0x7fafc80fad90 2024-02-20T00:09:15.897 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:14.779+0000 7fafb6ffd700 1 -- 172.21.15.84:0/3892269001 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fafb801a900 con 0x7fafc80fad90 2024-02-20T00:09:15.897 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:14.779+0000 7fafccf18700 1 -- 172.21.15.84:0/3892269001 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafc810a5b0 con 0x7fafc80fad90 2024-02-20T00:09:15.897 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:14.779+0000 7fafb6ffd700 1 -- 172.21.15.84:0/3892269001 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 5) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fafb800f070 con 0x7fafc80fad90 2024-02-20T00:09:15.897 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:14.780+0000 7fafb6ffd700 1 --2- 172.21.15.84:0/3892269001 >> [v2:172.21.15.84:6800/2888547837,v1:172.21.15.84:6801/2888547837] conn(0x7fafac02fe70 0x7fafac032330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:09:15.897 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:14.780+0000 7fafb6ffd700 1 -- 172.21.15.84:0/3892269001 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fafb8041250 con 0x7fafc80fad90 2024-02-20T00:09:15.897 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:14.780+0000 7fafc5d9b700 1 -- 172.21.15.84:0/3892269001 >> [v2:172.21.15.84:6800/2888547837,v1:172.21.15.84:6801/2888547837] conn(0x7fafac02fe70 msgr2=0x7fafac032330 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.84:6800/2888547837 2024-02-20T00:09:15.897 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:14.780+0000 7fafc5d9b700 1 --2- 172.21.15.84:0/3892269001 >> [v2:172.21.15.84:6800/2888547837,v1:172.21.15.84:6801/2888547837] conn(0x7fafac02fe70 0x7fafac032330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-02-20T00:09:15.897 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:14.780+0000 7fafccf18700 1 -- 172.21.15.84:0/3892269001 --> [v2:172.21.15.84:6800/2888547837,v1:172.21.15.84:6801/2888547837] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fafc8059100 con 0x7fafac02fe70 2024-02-20T00:09:15.897 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:14.981+0000 7fafc5d9b700 1 -- 172.21.15.84:0/3892269001 >> [v2:172.21.15.84:6800/2888547837,v1:172.21.15.84:6801/2888547837] conn(0x7fafac02fe70 msgr2=0x7fafac032330 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.84:6800/2888547837 2024-02-20T00:09:15.897 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:14.981+0000 7fafc5d9b700 1 --2- 172.21.15.84:0/3892269001 >> [v2:172.21.15.84:6800/2888547837,v1:172.21.15.84:6801/2888547837] conn(0x7fafac02fe70 0x7fafac032330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2024-02-20T00:09:15.897 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:15.382+0000 7fafc5d9b700 1 -- 172.21.15.84:0/3892269001 >> [v2:172.21.15.84:6800/2888547837,v1:172.21.15.84:6801/2888547837] conn(0x7fafac02fe70 msgr2=0x7fafac032330 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.84:6800/2888547837 2024-02-20T00:09:15.897 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:15.382+0000 7fafc5d9b700 1 --2- 172.21.15.84:0/3892269001 >> [v2:172.21.15.84:6800/2888547837,v1:172.21.15.84:6801/2888547837] conn(0x7fafac02fe70 0x7fafac032330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2024-02-20T00:09:16.185 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:16.183+0000 7fafc5d9b700 1 -- 172.21.15.84:0/3892269001 >> [v2:172.21.15.84:6800/2888547837,v1:172.21.15.84:6801/2888547837] conn(0x7fafac02fe70 msgr2=0x7fafac032330 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.84:6800/2888547837 2024-02-20T00:09:16.185 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:16.183+0000 7fafc5d9b700 1 --2- 172.21.15.84:0/3892269001 >> [v2:172.21.15.84:6800/2888547837,v1:172.21.15.84:6801/2888547837] conn(0x7fafac02fe70 0x7fafac032330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2024-02-20T00:09:16.829 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:16.826+0000 7fafb6ffd700 1 -- 172.21.15.84:0/3892269001 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mgrmap(e 6) v1 ==== 35970+0+0 (secure 0 0 0) 0x7fafb8025980 con 0x7fafc80fad90 2024-02-20T00:09:16.829 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:16.826+0000 7fafb6ffd700 1 -- 172.21.15.84:0/3892269001 >> [v2:172.21.15.84:6800/2888547837,v1:172.21.15.84:6801/2888547837] conn(0x7fafac02fe70 msgr2=0x7fafac032330 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-20T00:09:16.829 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:16.826+0000 7fafb6ffd700 1 --2- 172.21.15.84:0/3892269001 >> [v2:172.21.15.84:6800/2888547837,v1:172.21.15.84:6801/2888547837] conn(0x7fafac02fe70 0x7fafac032330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-20T00:09:17.107 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:16 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:09:16.780442+0000 mon.smithi084 (mon.0) 29 : cluster [INF] Active manager daemon smithi084.bmqrnf restarted 2024-02-20T00:09:17.108 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:16 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:09:16.780826+0000 mon.smithi084 (mon.0) 30 : cluster [INF] Activating manager daemon smithi084.bmqrnf 2024-02-20T00:09:17.108 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:16 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:09:16.783020+0000 mon.smithi084 (mon.0) 31 : cluster [DBG] osdmap e2: 0 total, 0 up, 0 in 2024-02-20T00:09:17.835 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:17.832+0000 7fafb6ffd700 1 -- 172.21.15.84:0/3892269001 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mgrmap(e 7) v1 ==== 36040+0+0 (secure 0 0 0) 0x7fafb8024ca0 con 0x7fafc80fad90 2024-02-20T00:09:17.835 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:17.832+0000 7fafb6ffd700 1 --2- 172.21.15.84:0/3892269001 >> [v2:172.21.15.84:6800/1412904880,v1:172.21.15.84:6801/1412904880] conn(0x7fafac035860 0x7fafac037c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:09:17.835 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:17.832+0000 7fafb6ffd700 1 -- 172.21.15.84:0/3892269001 --> [v2:172.21.15.84:6800/1412904880,v1:172.21.15.84:6801/1412904880] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fafc8059100 con 0x7fafac035860 2024-02-20T00:09:17.836 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:17.832+0000 7fafc5d9b700 1 --2- 172.21.15.84:0/3892269001 >> [v2:172.21.15.84:6800/1412904880,v1:172.21.15.84:6801/1412904880] conn(0x7fafac035860 0x7fafac037c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:09:17.836 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:17.833+0000 7fafc5d9b700 1 --2- 172.21.15.84:0/3892269001 >> [v2:172.21.15.84:6800/1412904880,v1:172.21.15.84:6801/1412904880] conn(0x7fafac035860 0x7fafac037c50 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7fafbc0036d0 tx=0x7fafbc006d20).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:09:17.836 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:17.834+0000 7fafb6ffd700 1 -- 172.21.15.84:0/3892269001 <== mgr.14114 v2:172.21.15.84:6800/1412904880 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6438 (secure 0 0 0) 0x7fafc8059100 con 0x7fafac035860 2024-02-20T00:09:17.843 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:17.840+0000 7fafccf18700 1 -- 172.21.15.84:0/3892269001 --> [v2:172.21.15.84:6800/1412904880,v1:172.21.15.84:6801/1412904880] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7fafc8045ad0 con 0x7fafac035860 2024-02-20T00:09:17.843 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:17.840+0000 7fafb6ffd700 1 -- 172.21.15.84:0/3892269001 <== mgr.14114 v2:172.21.15.84:6800/1412904880 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7fafc8045ad0 con 0x7fafac035860 2024-02-20T00:09:17.843 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: { 2024-02-20T00:09:17.843 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "mgrmap_epoch": 7, 2024-02-20T00:09:17.843 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "initialized": true 2024-02-20T00:09:17.845 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: } 2024-02-20T00:09:17.845 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:17.841+0000 7fafccf18700 1 -- 172.21.15.84:0/3892269001 >> [v2:172.21.15.84:6800/1412904880,v1:172.21.15.84:6801/1412904880] conn(0x7fafac035860 msgr2=0x7fafac037c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:09:17.845 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:17.841+0000 7fafccf18700 1 --2- 172.21.15.84:0/3892269001 >> [v2:172.21.15.84:6800/1412904880,v1:172.21.15.84:6801/1412904880] conn(0x7fafac035860 0x7fafac037c50 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7fafbc0036d0 tx=0x7fafbc006d20).stop 2024-02-20T00:09:17.845 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:17.841+0000 7fafccf18700 1 -- 172.21.15.84:0/3892269001 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fafc80fad90 msgr2=0x7fafc8109a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:09:17.845 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:17.841+0000 7fafccf18700 1 --2- 172.21.15.84:0/3892269001 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fafc80fad90 0x7fafc8109a30 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7fafb8002f40 tx=0x7fafb8006ee0).stop 2024-02-20T00:09:17.845 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:17.841+0000 7fafccf18700 1 -- 172.21.15.84:0/3892269001 shutdown_connections 2024-02-20T00:09:17.845 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:17.841+0000 7fafccf18700 1 --2- 172.21.15.84:0/3892269001 >> [v2:172.21.15.84:6800/1412904880,v1:172.21.15.84:6801/1412904880] conn(0x7fafac035860 0x7fafac037c50 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:09:17.846 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:17.841+0000 7fafccf18700 1 --2- 172.21.15.84:0/3892269001 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fafc80fad90 0x7fafc8109a30 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:09:17.846 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:17.841+0000 7fafccf18700 1 --2- 172.21.15.84:0/3892269001 >> [v2:172.21.15.84:6800/2888547837,v1:172.21.15.84:6801/2888547837] conn(0x7fafac02fe70 0x7fafac032330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-20T00:09:17.846 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:17.841+0000 7fafccf18700 1 -- 172.21.15.84:0/3892269001 >> 172.21.15.84:0/3892269001 conn(0x7fafc80f6530 msgr2=0x7fafc80f7f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:09:17.846 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:17.841+0000 7fafccf18700 1 -- 172.21.15.84:0/3892269001 shutdown_connections 2024-02-20T00:09:17.846 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:17.841+0000 7fafccf18700 1 -- 172.21.15.84:0/3892269001 wait complete. 2024-02-20T00:09:18.108 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:17 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:09:16.828618+0000 mon.smithi084 (mon.0) 32 : cluster [DBG] mgrmap e6: smithi084.bmqrnf(active, starting, since 0.0478598s) 2024-02-20T00:09:18.108 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:17 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:09:16.829741+0000 mon.smithi084 (mon.0) 33 : audit [DBG] from='mgr.14114 172.21.15.84:0/849529258' entity='mgr.smithi084.bmqrnf' cmd=[{"prefix": "mon metadata", "id": "smithi084"}]: dispatch 2024-02-20T00:09:18.108 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:17 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:09:16.830277+0000 mon.smithi084 (mon.0) 34 : audit [DBG] from='mgr.14114 172.21.15.84:0/849529258' entity='mgr.smithi084.bmqrnf' cmd=[{"prefix": "mgr metadata", "who": "smithi084.bmqrnf", "id": "smithi084.bmqrnf"}]: dispatch 2024-02-20T00:09:18.108 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:17 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:09:16.830923+0000 mon.smithi084 (mon.0) 35 : audit [DBG] from='mgr.14114 172.21.15.84:0/849529258' entity='mgr.smithi084.bmqrnf' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-20T00:09:18.108 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:17 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:09:16.831097+0000 mon.smithi084 (mon.0) 36 : audit [DBG] from='mgr.14114 172.21.15.84:0/849529258' entity='mgr.smithi084.bmqrnf' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-20T00:09:18.108 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:17 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:09:16.831200+0000 mon.smithi084 (mon.0) 37 : audit [DBG] from='mgr.14114 172.21.15.84:0/849529258' entity='mgr.smithi084.bmqrnf' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-20T00:09:18.108 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:17 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:09:16.846045+0000 mon.smithi084 (mon.0) 38 : cluster [INF] Manager daemon smithi084.bmqrnf is now available 2024-02-20T00:09:18.108 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:17 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:09:16.858798+0000 mon.smithi084 (mon.0) 39 : audit [INF] from='mgr.14114 172.21.15.84:0/849529258' entity='mgr.smithi084.bmqrnf' 2024-02-20T00:09:18.108 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:17 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:09:16.861548+0000 mon.smithi084 (mon.0) 40 : audit [INF] from='mgr.14114 172.21.15.84:0/849529258' entity='mgr.smithi084.bmqrnf' 2024-02-20T00:09:18.108 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:17 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:09:16.864052+0000 mon.smithi084 (mon.0) 41 : audit [INF] from='mgr.14114 172.21.15.84:0/849529258' entity='mgr.smithi084.bmqrnf' 2024-02-20T00:09:18.108 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:17 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:09:16.951679+0000 mon.smithi084 (mon.0) 42 : audit [INF] from='mgr.14114 172.21.15.84:0/849529258' entity='mgr.smithi084.bmqrnf' 2024-02-20T00:09:18.109 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:17 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:09:16.952297+0000 mon.smithi084 (mon.0) 43 : audit [DBG] from='mgr.14114 172.21.15.84:0/849529258' entity='mgr.smithi084.bmqrnf' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-20T00:09:18.109 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:17 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:09:16.953387+0000 mon.smithi084 (mon.0) 44 : audit [DBG] from='mgr.14114 172.21.15.84:0/849529258' entity='mgr.smithi084.bmqrnf' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-20T00:09:18.109 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:17 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:09:16.954672+0000 mon.smithi084 (mon.0) 45 : audit [DBG] from='mgr.14114 172.21.15.84:0/849529258' entity='mgr.smithi084.bmqrnf' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-20T00:09:18.109 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:17 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:09:16.965227+0000 mon.smithi084 (mon.0) 46 : audit [INF] from='mgr.14114 172.21.15.84:0/849529258' entity='mgr.smithi084.bmqrnf' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi084.bmqrnf/mirror_snapshot_schedule"}]: dispatch 2024-02-20T00:09:18.109 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:17 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:09:16.966383+0000 mon.smithi084 (mon.0) 47 : audit [INF] from='mgr.14114 172.21.15.84:0/849529258' entity='mgr.smithi084.bmqrnf' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi084.bmqrnf/trash_purge_schedule"}]: dispatch 2024-02-20T00:09:19.108 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:18 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:09:17.835324+0000 mgr.smithi084.bmqrnf (mgr.14114) 1 : audit [DBG] from='client.14118 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-02-20T00:09:19.108 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:18 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:09:17.835895+0000 mon.smithi084 (mon.0) 48 : cluster [DBG] mgrmap e7: smithi084.bmqrnf(active, since 1.05513s) 2024-02-20T00:09:19.108 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:18 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:09:17.842146+0000 mgr.smithi084.bmqrnf (mgr.14114) 2 : audit [DBG] from='client.14118 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-02-20T00:09:20.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:19 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:09:18.903445+0000 mon.smithi084 (mon.0) 49 : cluster [DBG] mgrmap e8: smithi084.bmqrnf(active, since 2s) 2024-02-20T00:09:23.215 INFO:teuthology.orchestra.run.smithi084.stderr:mgr epoch 5 is available 2024-02-20T00:09:23.216 INFO:teuthology.orchestra.run.smithi084.stderr:Setting orchestrator backend to cephadm... 2024-02-20T00:09:23.216 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi084 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/2a2e6c42-cf84-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp3ot6j0xn:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp4xryone4:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch set backend cephadm 2024-02-20T00:09:23.728 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:23.724+0000 7f1b5bdb9700 1 Processor -- start 2024-02-20T00:09:23.728 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:23.724+0000 7f1b5bdb9700 1 -- start start 2024-02-20T00:09:23.728 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:23.724+0000 7f1b5bdb9700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1b54070430 0x7f1b540fea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-20T00:09:23.728 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:23.724+0000 7f1b5bdb9700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b540fef40 con 0x7f1b54070430 2024-02-20T00:09:23.728 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:23.724+0000 7f1b59b55700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1b54070430 0x7f1b540fea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:09:23.729 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:23.725+0000 7f1b59b55700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1b54070430 0x7f1b540fea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:54656/0 (socket says 172.21.15.84:54656) 2024-02-20T00:09:23.729 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:23.725+0000 7f1b59b55700 1 -- 172.21.15.84:0/2573386140 learned_addr learned my addr 172.21.15.84:0/2573386140 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-02-20T00:09:23.729 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:23.725+0000 7f1b59b55700 1 -- 172.21.15.84:0/2573386140 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b540ff080 con 0x7f1b54070430 2024-02-20T00:09:23.729 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:23.725+0000 7f1b59b55700 1 --2- 172.21.15.84:0/2573386140 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1b54070430 0x7f1b540fea50 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f1b4400b5f0 tx=0x7f1b44006ee0).ready entity=mon.0 client_cookie=be3a97c9f89f0eb5 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:09:23.729 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:23.725+0000 7f1b58b53700 1 -- 172.21.15.84:0/2573386140 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1b4400d600 con 0x7f1b54070430 2024-02-20T00:09:23.729 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:23.725+0000 7f1b58b53700 1 -- 172.21.15.84:0/2573386140 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f1b4400dbf0 con 0x7f1b54070430 2024-02-20T00:09:23.729 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:23.725+0000 7f1b58b53700 1 -- 172.21.15.84:0/2573386140 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1b44011cb0 con 0x7f1b54070430 2024-02-20T00:09:23.729 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:23.726+0000 7f1b5bdb9700 1 -- 172.21.15.84:0/2573386140 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1b54070430 msgr2=0x7f1b540fea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:09:23.729 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:23.726+0000 7f1b5bdb9700 1 --2- 172.21.15.84:0/2573386140 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1b54070430 0x7f1b540fea50 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f1b4400b5f0 tx=0x7f1b44006ee0).stop 2024-02-20T00:09:23.729 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:23.726+0000 7f1b5bdb9700 1 -- 172.21.15.84:0/2573386140 shutdown_connections 2024-02-20T00:09:23.729 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:23.726+0000 7f1b5bdb9700 1 --2- 172.21.15.84:0/2573386140 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1b54070430 0x7f1b540fea50 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:09:23.731 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:23.726+0000 7f1b5bdb9700 1 -- 172.21.15.84:0/2573386140 >> 172.21.15.84:0/2573386140 conn(0x7f1b540f6530 msgr2=0x7f1b540f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:09:23.731 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:23.726+0000 7f1b5bdb9700 1 -- 172.21.15.84:0/2573386140 shutdown_connections 2024-02-20T00:09:23.731 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:23.726+0000 7f1b5bdb9700 1 -- 172.21.15.84:0/2573386140 wait complete. 2024-02-20T00:09:23.731 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:23.726+0000 7f1b5bdb9700 1 Processor -- start 2024-02-20T00:09:23.731 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:23.726+0000 7f1b5bdb9700 1 -- start start 2024-02-20T00:09:23.732 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:23.727+0000 7f1b5bdb9700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1b54070430 0x7f1b5410ab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:09:23.732 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:23.727+0000 7f1b5bdb9700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b5410b020 con 0x7f1b54070430 2024-02-20T00:09:23.732 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:23.727+0000 7f1b59b55700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1b54070430 0x7f1b5410ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:09:23.732 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:23.727+0000 7f1b59b55700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1b54070430 0x7f1b5410ab30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:54666/0 (socket says 172.21.15.84:54666) 2024-02-20T00:09:23.732 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:23.727+0000 7f1b59b55700 1 -- 172.21.15.84:0/4108782514 learned_addr learned my addr 172.21.15.84:0/4108782514 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-02-20T00:09:23.732 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:23.727+0000 7f1b59b55700 1 -- 172.21.15.84:0/4108782514 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b4400b200 con 0x7f1b54070430 2024-02-20T00:09:23.732 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:23.727+0000 7f1b59b55700 1 --2- 172.21.15.84:0/4108782514 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1b54070430 0x7f1b5410ab30 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f1b44006800 tx=0x7f1b44006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:09:23.732 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:23.727+0000 7f1b427fc700 1 -- 172.21.15.84:0/4108782514 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1b4401b570 con 0x7f1b54070430 2024-02-20T00:09:23.732 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:23.727+0000 7f1b427fc700 1 -- 172.21.15.84:0/4108782514 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f1b4401bb60 con 0x7f1b54070430 2024-02-20T00:09:23.732 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:23.727+0000 7f1b427fc700 1 -- 172.21.15.84:0/4108782514 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1b4401a9f0 con 0x7f1b54070430 2024-02-20T00:09:23.732 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:23.727+0000 7f1b5bdb9700 1 -- 172.21.15.84:0/4108782514 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b5410b220 con 0x7f1b54070430 2024-02-20T00:09:23.732 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:23.727+0000 7f1b5bdb9700 1 -- 172.21.15.84:0/4108782514 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b5410b6b0 con 0x7f1b54070430 2024-02-20T00:09:23.732 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:23.728+0000 7f1b427fc700 1 -- 172.21.15.84:0/4108782514 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f1b4400f070 con 0x7f1b54070430 2024-02-20T00:09:23.733 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:23.728+0000 7f1b427fc700 1 --2- 172.21.15.84:0/4108782514 >> [v2:172.21.15.84:6800/1412904880,v1:172.21.15.84:6801/1412904880] conn(0x7f1b3c02fe70 0x7f1b3c032330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:09:23.733 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:23.728+0000 7f1b427fc700 1 -- 172.21.15.84:0/4108782514 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f1b44041440 con 0x7f1b54070430 2024-02-20T00:09:23.733 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:23.728+0000 7f1b59354700 1 --2- 172.21.15.84:0/4108782514 >> [v2:172.21.15.84:6800/1412904880,v1:172.21.15.84:6801/1412904880] conn(0x7f1b3c02fe70 0x7f1b3c032330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:09:23.733 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:23.729+0000 7f1b5bdb9700 1 -- 172.21.15.84:0/4108782514 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b54045ad0 con 0x7f1b54070430 2024-02-20T00:09:23.733 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:23.729+0000 7f1b59354700 1 --2- 172.21.15.84:0/4108782514 >> [v2:172.21.15.84:6800/1412904880,v1:172.21.15.84:6801/1412904880] conn(0x7f1b3c02fe70 0x7f1b3c032330 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f1b500096a0 tx=0x7f1b50006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:09:23.735 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:23.733+0000 7f1b427fc700 1 -- 172.21.15.84:0/4108782514 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1b44016070 con 0x7f1b54070430 2024-02-20T00:09:23.912 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:23.910+0000 7f1b5bdb9700 1 -- 172.21.15.84:0/4108782514 --> [v2:172.21.15.84:6800/1412904880,v1:172.21.15.84:6801/1412904880] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7f1b54104bf0 con 0x7f1b3c02fe70 2024-02-20T00:09:23.917 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:23.915+0000 7f1b427fc700 1 -- 172.21.15.84:0/4108782514 <== mgr.14114 v2:172.21.15.84:6800/1412904880 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f1b54104bf0 con 0x7f1b3c02fe70 2024-02-20T00:09:23.920 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:23.916+0000 7f1b5bdb9700 1 -- 172.21.15.84:0/4108782514 >> [v2:172.21.15.84:6800/1412904880,v1:172.21.15.84:6801/1412904880] conn(0x7f1b3c02fe70 msgr2=0x7f1b3c032330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:09:23.920 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:23.916+0000 7f1b5bdb9700 1 --2- 172.21.15.84:0/4108782514 >> [v2:172.21.15.84:6800/1412904880,v1:172.21.15.84:6801/1412904880] conn(0x7f1b3c02fe70 0x7f1b3c032330 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f1b500096a0 tx=0x7f1b50006b40).stop 2024-02-20T00:09:23.920 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:23.916+0000 7f1b5bdb9700 1 -- 172.21.15.84:0/4108782514 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1b54070430 msgr2=0x7f1b5410ab30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:09:23.920 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:23.916+0000 7f1b5bdb9700 1 --2- 172.21.15.84:0/4108782514 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1b54070430 0x7f1b5410ab30 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f1b44006800 tx=0x7f1b44006ee0).stop 2024-02-20T00:09:23.920 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:23.917+0000 7f1b5bdb9700 1 -- 172.21.15.84:0/4108782514 shutdown_connections 2024-02-20T00:09:23.920 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:23.917+0000 7f1b5bdb9700 1 --2- 172.21.15.84:0/4108782514 >> [v2:172.21.15.84:6800/1412904880,v1:172.21.15.84:6801/1412904880] conn(0x7f1b3c02fe70 0x7f1b3c032330 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:09:23.920 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:23.917+0000 7f1b5bdb9700 1 --2- 172.21.15.84:0/4108782514 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1b54070430 0x7f1b5410ab30 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:09:23.920 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:23.917+0000 7f1b5bdb9700 1 -- 172.21.15.84:0/4108782514 >> 172.21.15.84:0/4108782514 conn(0x7f1b540f6530 msgr2=0x7f1b540f7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:09:23.920 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:23.917+0000 7f1b5bdb9700 1 -- 172.21.15.84:0/4108782514 shutdown_connections 2024-02-20T00:09:23.920 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:23.917+0000 7f1b5bdb9700 1 -- 172.21.15.84:0/4108782514 wait complete. 2024-02-20T00:09:24.571 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi084 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/2a2e6c42-cf84-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp3ot6j0xn:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp4xryone4:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm set-user root 2024-02-20T00:09:25.137 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:25.131+0000 7f65346c7700 1 Processor -- start 2024-02-20T00:09:25.137 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:25.131+0000 7f65346c7700 1 -- start start 2024-02-20T00:09:25.137 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:25.132+0000 7f65346c7700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f652c0703f0 0x7f652c0fea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-20T00:09:25.137 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:25.132+0000 7f65346c7700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f652c0fef80 con 0x7f652c0703f0 2024-02-20T00:09:25.137 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:25.132+0000 7f6532463700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f652c0703f0 0x7f652c0fea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:09:25.137 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:25.132+0000 7f6532463700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f652c0703f0 0x7f652c0fea90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:54676/0 (socket says 172.21.15.84:54676) 2024-02-20T00:09:25.137 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:25.132+0000 7f6532463700 1 -- 172.21.15.84:0/1672968935 learned_addr learned my addr 172.21.15.84:0/1672968935 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-02-20T00:09:25.137 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:25.132+0000 7f6532463700 1 -- 172.21.15.84:0/1672968935 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f652c0ff0c0 con 0x7f652c0703f0 2024-02-20T00:09:25.137 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:25.132+0000 7f6532463700 1 --2- 172.21.15.84:0/1672968935 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f652c0703f0 0x7f652c0fea90 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f651c00b5f0 tx=0x7f651c006ee0).ready entity=mon.0 client_cookie=99d970c43d918fa8 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:09:25.138 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:25.133+0000 7f6531461700 1 -- 172.21.15.84:0/1672968935 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f651c00d600 con 0x7f652c0703f0 2024-02-20T00:09:25.138 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:25.133+0000 7f6531461700 1 -- 172.21.15.84:0/1672968935 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f651c00dbf0 con 0x7f652c0703f0 2024-02-20T00:09:25.138 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:25.133+0000 7f6531461700 1 -- 172.21.15.84:0/1672968935 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f651c011ce0 con 0x7f652c0703f0 2024-02-20T00:09:25.138 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:25.133+0000 7f65346c7700 1 -- 172.21.15.84:0/1672968935 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f652c0703f0 msgr2=0x7f652c0fea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:09:25.138 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:25.133+0000 7f65346c7700 1 --2- 172.21.15.84:0/1672968935 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f652c0703f0 0x7f652c0fea90 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f651c00b5f0 tx=0x7f651c006ee0).stop 2024-02-20T00:09:25.138 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:25.133+0000 7f65346c7700 1 -- 172.21.15.84:0/1672968935 shutdown_connections 2024-02-20T00:09:25.138 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:25.133+0000 7f65346c7700 1 --2- 172.21.15.84:0/1672968935 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f652c0703f0 0x7f652c0fea90 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:09:25.138 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:25.133+0000 7f65346c7700 1 -- 172.21.15.84:0/1672968935 >> 172.21.15.84:0/1672968935 conn(0x7f652c0f6530 msgr2=0x7f652c0f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:09:25.138 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:25.133+0000 7f65346c7700 1 -- 172.21.15.84:0/1672968935 shutdown_connections 2024-02-20T00:09:25.138 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:25.133+0000 7f65346c7700 1 -- 172.21.15.84:0/1672968935 wait complete. 2024-02-20T00:09:25.138 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:25.134+0000 7f65346c7700 1 Processor -- start 2024-02-20T00:09:25.138 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:25.134+0000 7f65346c7700 1 -- start start 2024-02-20T00:09:25.138 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:25.134+0000 7f65346c7700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f652c0703f0 0x7f652c0fc510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:09:25.138 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:25.134+0000 7f65346c7700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f652c0fa990 con 0x7f652c0703f0 2024-02-20T00:09:25.139 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:25.134+0000 7f6532463700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f652c0703f0 0x7f652c0fc510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:09:25.139 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:25.134+0000 7f6532463700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f652c0703f0 0x7f652c0fc510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:54686/0 (socket says 172.21.15.84:54686) 2024-02-20T00:09:25.139 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:25.134+0000 7f6532463700 1 -- 172.21.15.84:0/3000789026 learned_addr learned my addr 172.21.15.84:0/3000789026 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-02-20T00:09:25.140 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:25.135+0000 7f6532463700 1 -- 172.21.15.84:0/3000789026 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f651c00b200 con 0x7f652c0703f0 2024-02-20T00:09:25.140 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:25.135+0000 7f6532463700 1 --2- 172.21.15.84:0/3000789026 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f652c0703f0 0x7f652c0fc510 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f651c0115c0 tx=0x7f651c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:09:25.140 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:25.135+0000 7f651affd700 1 -- 172.21.15.84:0/3000789026 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f651c01b410 con 0x7f652c0703f0 2024-02-20T00:09:25.140 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:25.135+0000 7f651affd700 1 -- 172.21.15.84:0/3000789026 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f651c01ba00 con 0x7f652c0703f0 2024-02-20T00:09:25.140 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:25.135+0000 7f651affd700 1 -- 172.21.15.84:0/3000789026 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f651c01aaa0 con 0x7f652c0703f0 2024-02-20T00:09:25.140 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:25.135+0000 7f65346c7700 1 -- 172.21.15.84:0/3000789026 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f652c0fab30 con 0x7f652c0703f0 2024-02-20T00:09:25.140 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:25.135+0000 7f65346c7700 1 -- 172.21.15.84:0/3000789026 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f652c0fafc0 con 0x7f652c0703f0 2024-02-20T00:09:25.140 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:25.136+0000 7f651affd700 1 -- 172.21.15.84:0/3000789026 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f651c00f070 con 0x7f652c0703f0 2024-02-20T00:09:25.140 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:25.136+0000 7f651affd700 1 --2- 172.21.15.84:0/3000789026 >> [v2:172.21.15.84:6800/1412904880,v1:172.21.15.84:6801/1412904880] conn(0x7f651402fe10 0x7f65140322d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:09:25.140 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:25.136+0000 7f651affd700 1 -- 172.21.15.84:0/3000789026 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f651c041010 con 0x7f652c0703f0 2024-02-20T00:09:25.140 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:25.136+0000 7f65346c7700 1 -- 172.21.15.84:0/3000789026 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f652c045ad0 con 0x7f652c0703f0 2024-02-20T00:09:25.140 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:25.136+0000 7f6531c62700 1 --2- 172.21.15.84:0/3000789026 >> [v2:172.21.15.84:6800/1412904880,v1:172.21.15.84:6801/1412904880] conn(0x7f651402fe10 0x7f65140322d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:09:25.141 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:25.136+0000 7f6531c62700 1 --2- 172.21.15.84:0/3000789026 >> [v2:172.21.15.84:6800/1412904880,v1:172.21.15.84:6801/1412904880] conn(0x7f651402fe10 0x7f65140322d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f65280096a0 tx=0x7f6528006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:09:25.142 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:25.140+0000 7f651affd700 1 -- 172.21.15.84:0/3000789026 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f651c016070 con 0x7f652c0703f0 2024-02-20T00:09:25.292 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:25.289+0000 7f65346c7700 1 -- 172.21.15.84:0/3000789026 --> [v2:172.21.15.84:6800/1412904880,v1:172.21.15.84:6801/1412904880] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7f652c0fb200 con 0x7f651402fe10 2024-02-20T00:09:25.292 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:25.290+0000 7f651affd700 1 -- 172.21.15.84:0/3000789026 <== mgr.14114 v2:172.21.15.84:6800/1412904880 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7f652c0fb200 con 0x7f651402fe10 2024-02-20T00:09:25.292 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: value unchanged 2024-02-20T00:09:25.294 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:25.291+0000 7f65346c7700 1 -- 172.21.15.84:0/3000789026 >> [v2:172.21.15.84:6800/1412904880,v1:172.21.15.84:6801/1412904880] conn(0x7f651402fe10 msgr2=0x7f65140322d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:09:25.294 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:25.291+0000 7f65346c7700 1 --2- 172.21.15.84:0/3000789026 >> [v2:172.21.15.84:6800/1412904880,v1:172.21.15.84:6801/1412904880] conn(0x7f651402fe10 0x7f65140322d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f65280096a0 tx=0x7f6528006b40).stop 2024-02-20T00:09:25.295 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:25.291+0000 7f65346c7700 1 -- 172.21.15.84:0/3000789026 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f652c0703f0 msgr2=0x7f652c0fc510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:09:25.295 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:25.291+0000 7f65346c7700 1 --2- 172.21.15.84:0/3000789026 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f652c0703f0 0x7f652c0fc510 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f651c0115c0 tx=0x7f651c006ee0).stop 2024-02-20T00:09:25.295 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:25.291+0000 7f65346c7700 1 -- 172.21.15.84:0/3000789026 shutdown_connections 2024-02-20T00:09:25.295 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:25.291+0000 7f65346c7700 1 --2- 172.21.15.84:0/3000789026 >> [v2:172.21.15.84:6800/1412904880,v1:172.21.15.84:6801/1412904880] conn(0x7f651402fe10 0x7f65140322d0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:09:25.295 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:25.291+0000 7f65346c7700 1 --2- 172.21.15.84:0/3000789026 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f652c0703f0 0x7f652c0fc510 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:09:25.295 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:25.291+0000 7f65346c7700 1 -- 172.21.15.84:0/3000789026 >> 172.21.15.84:0/3000789026 conn(0x7f652c0f6530 msgr2=0x7f652c0f7f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:09:25.295 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:25.291+0000 7f65346c7700 1 -- 172.21.15.84:0/3000789026 shutdown_connections 2024-02-20T00:09:25.295 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:25.291+0000 7f65346c7700 1 -- 172.21.15.84:0/3000789026 wait complete. 2024-02-20T00:09:25.316 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:24 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:09:23.912478+0000 mgr.smithi084.bmqrnf (mgr.14114) 3 : audit [DBG] from='client.14124 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T00:09:25.317 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:24 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:09:23.916405+0000 mon.smithi084 (mon.0) 50 : audit [INF] from='mgr.14114 172.21.15.84:0/849529258' entity='mgr.smithi084.bmqrnf' 2024-02-20T00:09:25.317 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:24 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:09:23.920209+0000 mon.smithi084 (mon.0) 51 : audit [DBG] from='mgr.14114 172.21.15.84:0/849529258' entity='mgr.smithi084.bmqrnf' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-20T00:09:25.944 INFO:teuthology.orchestra.run.smithi084.stderr:Generating ssh key... 2024-02-20T00:09:25.944 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi084 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/2a2e6c42-cf84-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp3ot6j0xn:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp4xryone4:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm generate-key 2024-02-20T00:09:26.464 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:26.438+0000 7fbb410b6700 1 Processor -- start 2024-02-20T00:09:26.465 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:26.438+0000 7fbb410b6700 1 -- start start 2024-02-20T00:09:26.465 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:26.438+0000 7fbb410b6700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbb3c070390 0x7fbb3c100d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-20T00:09:26.465 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:26.438+0000 7fbb410b6700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb3c0fc720 con 0x7fbb3c070390 2024-02-20T00:09:26.465 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:26.438+0000 7fbb3bfff700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbb3c070390 0x7fbb3c100d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:09:26.465 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:26.438+0000 7fbb3bfff700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbb3c070390 0x7fbb3c100d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:54694/0 (socket says 172.21.15.84:54694) 2024-02-20T00:09:26.465 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:26.438+0000 7fbb3bfff700 1 -- 172.21.15.84:0/3844931778 learned_addr learned my addr 172.21.15.84:0/3844931778 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-02-20T00:09:26.465 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:26.438+0000 7fbb3bfff700 1 -- 172.21.15.84:0/3844931778 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb3c101220 con 0x7fbb3c070390 2024-02-20T00:09:26.465 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:26.439+0000 7fbb3bfff700 1 --2- 172.21.15.84:0/3844931778 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbb3c070390 0x7fbb3c100d30 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7fbb2400b5f0 tx=0x7fbb24006ee0).ready entity=mon.0 client_cookie=baaaef626deb66a3 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:09:26.465 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:26.439+0000 7fbb3affd700 1 -- 172.21.15.84:0/3844931778 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbb2400d600 con 0x7fbb3c070390 2024-02-20T00:09:26.465 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:26.439+0000 7fbb3affd700 1 -- 172.21.15.84:0/3844931778 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fbb2400dbf0 con 0x7fbb3c070390 2024-02-20T00:09:26.465 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:26.439+0000 7fbb3affd700 1 -- 172.21.15.84:0/3844931778 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbb24011cb0 con 0x7fbb3c070390 2024-02-20T00:09:26.466 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:26.439+0000 7fbb410b6700 1 -- 172.21.15.84:0/3844931778 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbb3c070390 msgr2=0x7fbb3c100d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:09:26.466 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:26.439+0000 7fbb410b6700 1 --2- 172.21.15.84:0/3844931778 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbb3c070390 0x7fbb3c100d30 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7fbb2400b5f0 tx=0x7fbb24006ee0).stop 2024-02-20T00:09:26.466 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:26.439+0000 7fbb410b6700 1 -- 172.21.15.84:0/3844931778 shutdown_connections 2024-02-20T00:09:26.466 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:26.439+0000 7fbb410b6700 1 --2- 172.21.15.84:0/3844931778 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbb3c070390 0x7fbb3c100d30 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:09:26.466 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:26.439+0000 7fbb410b6700 1 -- 172.21.15.84:0/3844931778 >> 172.21.15.84:0/3844931778 conn(0x7fbb3c0f6390 msgr2=0x7fbb3c0f87b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:09:26.466 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:26.439+0000 7fbb410b6700 1 -- 172.21.15.84:0/3844931778 shutdown_connections 2024-02-20T00:09:26.466 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:26.439+0000 7fbb410b6700 1 -- 172.21.15.84:0/3844931778 wait complete. 2024-02-20T00:09:26.466 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:26.440+0000 7fbb410b6700 1 Processor -- start 2024-02-20T00:09:26.466 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:26.440+0000 7fbb410b6700 1 -- start start 2024-02-20T00:09:26.466 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:26.440+0000 7fbb410b6700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbb3c070390 0x7fbb3c10cb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:09:26.466 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:26.440+0000 7fbb410b6700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb3c10d060 con 0x7fbb3c070390 2024-02-20T00:09:26.466 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:26.440+0000 7fbb3bfff700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbb3c070390 0x7fbb3c10cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:09:26.466 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:26.440+0000 7fbb3bfff700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbb3c070390 0x7fbb3c10cb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:54706/0 (socket says 172.21.15.84:54706) 2024-02-20T00:09:26.466 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:26.440+0000 7fbb3bfff700 1 -- 172.21.15.84:0/2450669738 learned_addr learned my addr 172.21.15.84:0/2450669738 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-02-20T00:09:26.466 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:26.440+0000 7fbb3bfff700 1 -- 172.21.15.84:0/2450669738 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb2400b200 con 0x7fbb3c070390 2024-02-20T00:09:26.467 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:26.441+0000 7fbb3bfff700 1 --2- 172.21.15.84:0/2450669738 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbb3c070390 0x7fbb3c10cb70 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7fbb24006800 tx=0x7fbb24006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:09:26.467 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:26.441+0000 7fbb38ff9700 1 -- 172.21.15.84:0/2450669738 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbb2401b570 con 0x7fbb3c070390 2024-02-20T00:09:26.467 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:26.441+0000 7fbb38ff9700 1 -- 172.21.15.84:0/2450669738 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fbb2401bb60 con 0x7fbb3c070390 2024-02-20T00:09:26.467 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:26.441+0000 7fbb410b6700 1 -- 172.21.15.84:0/2450669738 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb3c10d260 con 0x7fbb3c070390 2024-02-20T00:09:26.467 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:26.441+0000 7fbb38ff9700 1 -- 172.21.15.84:0/2450669738 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbb2401a9f0 con 0x7fbb3c070390 2024-02-20T00:09:26.467 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:26.441+0000 7fbb410b6700 1 -- 172.21.15.84:0/2450669738 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb3c10d6f0 con 0x7fbb3c070390 2024-02-20T00:09:26.467 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:26.442+0000 7fbb38ff9700 1 -- 172.21.15.84:0/2450669738 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fbb2400f070 con 0x7fbb3c070390 2024-02-20T00:09:26.467 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:26.442+0000 7fbb410b6700 1 -- 172.21.15.84:0/2450669738 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb3c045ad0 con 0x7fbb3c070390 2024-02-20T00:09:26.467 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:26.442+0000 7fbb38ff9700 1 --2- 172.21.15.84:0/2450669738 >> [v2:172.21.15.84:6800/1412904880,v1:172.21.15.84:6801/1412904880] conn(0x7fbb1c02fe70 0x7fbb1c032330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:09:26.467 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:26.442+0000 7fbb38ff9700 1 -- 172.21.15.84:0/2450669738 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7fbb240413b0 con 0x7fbb3c070390 2024-02-20T00:09:26.467 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:26.442+0000 7fbb3b7fe700 1 --2- 172.21.15.84:0/2450669738 >> [v2:172.21.15.84:6800/1412904880,v1:172.21.15.84:6801/1412904880] conn(0x7fbb1c02fe70 0x7fbb1c032330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:09:26.467 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:26.442+0000 7fbb3b7fe700 1 --2- 172.21.15.84:0/2450669738 >> [v2:172.21.15.84:6800/1412904880,v1:172.21.15.84:6801/1412904880] conn(0x7fbb1c02fe70 0x7fbb1c032330 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7fbb2c0096a0 tx=0x7fbb2c006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:09:26.467 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:26.446+0000 7fbb38ff9700 1 -- 172.21.15.84:0/2450669738 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbb24022d70 con 0x7fbb3c070390 2024-02-20T00:09:26.600 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:26.597+0000 7fbb410b6700 1 -- 172.21.15.84:0/2450669738 --> [v2:172.21.15.84:6800/1412904880,v1:172.21.15.84:6801/1412904880] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7fbb3c10e010 con 0x7fbb1c02fe70 2024-02-20T00:09:26.988 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:26.986+0000 7fbb38ff9700 1 -- 172.21.15.84:0/2450669738 <== mgr.14114 v2:172.21.15.84:6800/1412904880 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fbb3c10e010 con 0x7fbb1c02fe70 2024-02-20T00:09:26.990 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:26.987+0000 7fbb410b6700 1 -- 172.21.15.84:0/2450669738 >> [v2:172.21.15.84:6800/1412904880,v1:172.21.15.84:6801/1412904880] conn(0x7fbb1c02fe70 msgr2=0x7fbb1c032330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:09:26.991 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:26.987+0000 7fbb410b6700 1 --2- 172.21.15.84:0/2450669738 >> [v2:172.21.15.84:6800/1412904880,v1:172.21.15.84:6801/1412904880] conn(0x7fbb1c02fe70 0x7fbb1c032330 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7fbb2c0096a0 tx=0x7fbb2c006b40).stop 2024-02-20T00:09:26.991 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:26.987+0000 7fbb410b6700 1 -- 172.21.15.84:0/2450669738 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbb3c070390 msgr2=0x7fbb3c10cb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:09:26.991 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:26.987+0000 7fbb410b6700 1 --2- 172.21.15.84:0/2450669738 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbb3c070390 0x7fbb3c10cb70 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7fbb24006800 tx=0x7fbb24006ee0).stop 2024-02-20T00:09:26.991 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:26.987+0000 7fbb410b6700 1 -- 172.21.15.84:0/2450669738 shutdown_connections 2024-02-20T00:09:26.991 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:26.987+0000 7fbb410b6700 1 --2- 172.21.15.84:0/2450669738 >> [v2:172.21.15.84:6800/1412904880,v1:172.21.15.84:6801/1412904880] conn(0x7fbb1c02fe70 0x7fbb1c032330 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:09:26.991 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:26.987+0000 7fbb410b6700 1 --2- 172.21.15.84:0/2450669738 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbb3c070390 0x7fbb3c10cb70 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:09:26.991 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:26.987+0000 7fbb410b6700 1 -- 172.21.15.84:0/2450669738 >> 172.21.15.84:0/2450669738 conn(0x7fbb3c0f6390 msgr2=0x7fbb3c0f7da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:09:26.991 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:26.988+0000 7fbb410b6700 1 -- 172.21.15.84:0/2450669738 shutdown_connections 2024-02-20T00:09:26.991 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:26.988+0000 7fbb410b6700 1 -- 172.21.15.84:0/2450669738 wait complete. 2024-02-20T00:09:27.107 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:26 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:09:25.291417+0000 mgr.smithi084.bmqrnf (mgr.14114) 4 : audit [DBG] from='client.14126 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T00:09:27.634 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi084 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/2a2e6c42-cf84-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp3ot6j0xn:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp4xryone4:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm get-pub-key 2024-02-20T00:09:27.900 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:27 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cephadm 2024-02-20T00:09:26.599569+0000 mgr.smithi084.bmqrnf (mgr.14114) 6 : cephadm [INF] Generating ssh key... 2024-02-20T00:09:27.900 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:27 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:09:26.985056+0000 mon.smithi084 (mon.0) 52 : audit [INF] from='mgr.14114 172.21.15.84:0/849529258' entity='mgr.smithi084.bmqrnf' 2024-02-20T00:09:27.900 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:27 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:09:26.986705+0000 mon.smithi084 (mon.0) 53 : audit [INF] from='mgr.14114 172.21.15.84:0/849529258' entity='mgr.smithi084.bmqrnf' 2024-02-20T00:09:28.122 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:28.117+0000 7f0a16262700 1 Processor -- start 2024-02-20T00:09:28.122 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:28.117+0000 7f0a16262700 1 -- start start 2024-02-20T00:09:28.123 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:28.117+0000 7f0a16262700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0a10070430 0x7f0a100fea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-20T00:09:28.123 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:28.117+0000 7f0a16262700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a100fef40 con 0x7f0a10070430 2024-02-20T00:09:28.123 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:28.117+0000 7f0a0f7fe700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0a10070430 0x7f0a100fea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:09:28.123 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:28.117+0000 7f0a0f7fe700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0a10070430 0x7f0a100fea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:54714/0 (socket says 172.21.15.84:54714) 2024-02-20T00:09:28.123 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:28.117+0000 7f0a0f7fe700 1 -- 172.21.15.84:0/1851032783 learned_addr learned my addr 172.21.15.84:0/1851032783 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-02-20T00:09:28.123 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:28.117+0000 7f0a0f7fe700 1 -- 172.21.15.84:0/1851032783 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a100ff080 con 0x7f0a10070430 2024-02-20T00:09:28.123 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:28.117+0000 7f0a0f7fe700 1 --2- 172.21.15.84:0/1851032783 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0a10070430 0x7f0a100fea50 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f0a0000b5f0 tx=0x7f0a00006ee0).ready entity=mon.0 client_cookie=68f720ae4d2a3035 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:09:28.123 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:28.118+0000 7f0a0e7fc700 1 -- 172.21.15.84:0/1851032783 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0a0000d600 con 0x7f0a10070430 2024-02-20T00:09:28.123 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:28.118+0000 7f0a0e7fc700 1 -- 172.21.15.84:0/1851032783 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f0a0000dbf0 con 0x7f0a10070430 2024-02-20T00:09:28.123 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:28.118+0000 7f0a0e7fc700 1 -- 172.21.15.84:0/1851032783 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0a00011cb0 con 0x7f0a10070430 2024-02-20T00:09:28.123 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:28.118+0000 7f0a16262700 1 -- 172.21.15.84:0/1851032783 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0a10070430 msgr2=0x7f0a100fea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:09:28.123 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:28.118+0000 7f0a16262700 1 --2- 172.21.15.84:0/1851032783 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0a10070430 0x7f0a100fea50 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f0a0000b5f0 tx=0x7f0a00006ee0).stop 2024-02-20T00:09:28.123 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:28.118+0000 7f0a16262700 1 -- 172.21.15.84:0/1851032783 shutdown_connections 2024-02-20T00:09:28.124 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:28.118+0000 7f0a16262700 1 --2- 172.21.15.84:0/1851032783 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0a10070430 0x7f0a100fea50 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:09:28.124 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:28.118+0000 7f0a16262700 1 -- 172.21.15.84:0/1851032783 >> 172.21.15.84:0/1851032783 conn(0x7f0a100f6530 msgr2=0x7f0a100f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:09:28.124 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:28.118+0000 7f0a16262700 1 -- 172.21.15.84:0/1851032783 shutdown_connections 2024-02-20T00:09:28.124 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:28.119+0000 7f0a16262700 1 -- 172.21.15.84:0/1851032783 wait complete. 2024-02-20T00:09:28.124 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:28.119+0000 7f0a16262700 1 Processor -- start 2024-02-20T00:09:28.124 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:28.119+0000 7f0a16262700 1 -- start start 2024-02-20T00:09:28.124 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:28.119+0000 7f0a16262700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0a10070430 0x7f0a1010ab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:09:28.124 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:28.119+0000 7f0a16262700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a1010b030 con 0x7f0a10070430 2024-02-20T00:09:28.124 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:28.119+0000 7f0a0f7fe700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0a10070430 0x7f0a1010ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:09:28.124 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:28.119+0000 7f0a0f7fe700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0a10070430 0x7f0a1010ab40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:54730/0 (socket says 172.21.15.84:54730) 2024-02-20T00:09:28.124 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:28.120+0000 7f0a0f7fe700 1 -- 172.21.15.84:0/3943275633 learned_addr learned my addr 172.21.15.84:0/3943275633 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-02-20T00:09:28.125 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:28.120+0000 7f0a0f7fe700 1 -- 172.21.15.84:0/3943275633 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a0000b200 con 0x7f0a10070430 2024-02-20T00:09:28.125 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:28.120+0000 7f0a0f7fe700 1 --2- 172.21.15.84:0/3943275633 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0a10070430 0x7f0a1010ab40 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f0a00006800 tx=0x7f0a00006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:09:28.125 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:28.120+0000 7f09f7fff700 1 -- 172.21.15.84:0/3943275633 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0a0001b570 con 0x7f0a10070430 2024-02-20T00:09:28.125 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:28.120+0000 7f09f7fff700 1 -- 172.21.15.84:0/3943275633 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f0a0001bb60 con 0x7f0a10070430 2024-02-20T00:09:28.125 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:28.120+0000 7f09f7fff700 1 -- 172.21.15.84:0/3943275633 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0a0001a9f0 con 0x7f0a10070430 2024-02-20T00:09:28.125 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:28.120+0000 7f0a16262700 1 -- 172.21.15.84:0/3943275633 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a1010b230 con 0x7f0a10070430 2024-02-20T00:09:28.125 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:28.120+0000 7f0a16262700 1 -- 172.21.15.84:0/3943275633 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a1010b6c0 con 0x7f0a10070430 2024-02-20T00:09:28.125 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:28.121+0000 7f09f7fff700 1 -- 172.21.15.84:0/3943275633 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f0a0000f070 con 0x7f0a10070430 2024-02-20T00:09:28.125 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:28.121+0000 7f09f7fff700 1 --2- 172.21.15.84:0/3943275633 >> [v2:172.21.15.84:6800/1412904880,v1:172.21.15.84:6801/1412904880] conn(0x7f09fc02fe10 0x7f09fc0322d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:09:28.126 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:28.121+0000 7f09f7fff700 1 -- 172.21.15.84:0/3943275633 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f0a000414b0 con 0x7f0a10070430 2024-02-20T00:09:28.126 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:28.121+0000 7f0a0effd700 1 --2- 172.21.15.84:0/3943275633 >> [v2:172.21.15.84:6800/1412904880,v1:172.21.15.84:6801/1412904880] conn(0x7f09fc02fe10 0x7f09fc0322d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:09:28.126 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:28.121+0000 7f0a0effd700 1 --2- 172.21.15.84:0/3943275633 >> [v2:172.21.15.84:6800/1412904880,v1:172.21.15.84:6801/1412904880] conn(0x7f09fc02fe10 0x7f09fc0322d0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f0a040096a0 tx=0x7f0a04006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:09:28.126 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:28.121+0000 7f0a16262700 1 -- 172.21.15.84:0/3943275633 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a10045ad0 con 0x7f0a10070430 2024-02-20T00:09:28.128 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:28.126+0000 7f09f7fff700 1 -- 172.21.15.84:0/3943275633 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0a00016070 con 0x7f0a10070430 2024-02-20T00:09:28.278 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:28.275+0000 7f0a16262700 1 -- 172.21.15.84:0/3943275633 --> [v2:172.21.15.84:6800/1412904880,v1:172.21.15.84:6801/1412904880] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f0a10104bf0 con 0x7f09fc02fe10 2024-02-20T00:09:28.278 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:28.276+0000 7f09f7fff700 1 -- 172.21.15.84:0/3943275633 <== mgr.14114 v2:172.21.15.84:6800/1412904880 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7f0a10104bf0 con 0x7f09fc02fe10 2024-02-20T00:09:28.278 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDVfsDtzKDpVOBeHn4d4EDiVlt4RcYaogldSPa5IeJCrgeX2uk1+0iHAMxvA2iaYvP1Ib8GH/a/8rjr3LfwcC8MRIxfZs0Vk/r1K133c0wlaJ7Hb3g9VX8kbNcnhYiJ+flthCyymvuTrjuB2lhvIAVR0mdTmuyCLk/nLzrsCW+E98KxlWI3LMvGaf1pQvyLlNTFcf8cwgE6pMcBaK0GkntheJN/viF6yjGTgKtK2yzr/Bzu6mYT/uciD77xVVIfnMN0HPFS17x/urKM3XRkMrLqyttRivF061sSImTr78B7UmpNCideVF0gf2akgbtBonIGBOfm1Nkici/Jn7/k/3NF3fz2r1QiDg8JYLLgO0yE7InGa8Mu6M5RjW4OsRzLLum+gV+Jh1tu/0pU/P+ogq+IdvfhmEwMcNL4d7ghTb1SNybhSb1jL2ai/MA/VzuaMWQi7k1BCE1nZGF3GCsYSxf++CxZQViR9mgOV3tUeHOMI917bi2fAX5Pyp+0pYbhij8= ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715 2024-02-20T00:09:28.281 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:28.277+0000 7f0a16262700 1 -- 172.21.15.84:0/3943275633 >> [v2:172.21.15.84:6800/1412904880,v1:172.21.15.84:6801/1412904880] conn(0x7f09fc02fe10 msgr2=0x7f09fc0322d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:09:28.281 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:28.277+0000 7f0a16262700 1 --2- 172.21.15.84:0/3943275633 >> [v2:172.21.15.84:6800/1412904880,v1:172.21.15.84:6801/1412904880] conn(0x7f09fc02fe10 0x7f09fc0322d0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f0a040096a0 tx=0x7f0a04006b40).stop 2024-02-20T00:09:28.281 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:28.277+0000 7f0a16262700 1 -- 172.21.15.84:0/3943275633 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0a10070430 msgr2=0x7f0a1010ab40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:09:28.281 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:28.277+0000 7f0a16262700 1 --2- 172.21.15.84:0/3943275633 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0a10070430 0x7f0a1010ab40 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f0a00006800 tx=0x7f0a00006ee0).stop 2024-02-20T00:09:28.281 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:28.277+0000 7f0a16262700 1 -- 172.21.15.84:0/3943275633 shutdown_connections 2024-02-20T00:09:28.281 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:28.277+0000 7f0a16262700 1 --2- 172.21.15.84:0/3943275633 >> [v2:172.21.15.84:6800/1412904880,v1:172.21.15.84:6801/1412904880] conn(0x7f09fc02fe10 0x7f09fc0322d0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:09:28.281 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:28.277+0000 7f0a16262700 1 --2- 172.21.15.84:0/3943275633 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0a10070430 0x7f0a1010ab40 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:09:28.281 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:28.277+0000 7f0a16262700 1 -- 172.21.15.84:0/3943275633 >> 172.21.15.84:0/3943275633 conn(0x7f0a100f6530 msgr2=0x7f0a100f7f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:09:28.282 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:28.278+0000 7f0a16262700 1 -- 172.21.15.84:0/3943275633 shutdown_connections 2024-02-20T00:09:28.282 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:28.278+0000 7f0a16262700 1 -- 172.21.15.84:0/3943275633 wait complete. 2024-02-20T00:09:28.957 INFO:teuthology.orchestra.run.smithi084.stderr:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-02-20T00:09:28.957 INFO:teuthology.orchestra.run.smithi084.stderr:Adding key to root@localhost authorized_keys... 2024-02-20T00:09:28.958 INFO:teuthology.orchestra.run.smithi084.stderr:Adding host smithi084... 2024-02-20T00:09:28.958 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi084 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/2a2e6c42-cf84-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp3ot6j0xn:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp4xryone4:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch host add smithi084 172.21.15.84 2024-02-20T00:09:29.213 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:28 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:09:28.277298+0000 mgr.smithi084.bmqrnf (mgr.14114) 7 : audit [DBG] from='client.14130 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T00:09:29.499 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:29.496+0000 7f8bce325700 1 Processor -- start 2024-02-20T00:09:29.499 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:29.496+0000 7f8bce325700 1 -- start start 2024-02-20T00:09:29.500 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:29.496+0000 7f8bce325700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8bc80fad90 0x7f8bc80fb1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-20T00:09:29.500 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:29.496+0000 7f8bce325700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bc80fb6a0 con 0x7f8bc80fad90 2024-02-20T00:09:29.500 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:29.496+0000 7f8bc7fff700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8bc80fad90 0x7f8bc80fb1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:09:29.500 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:29.496+0000 7f8bc7fff700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8bc80fad90 0x7f8bc80fb1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:53204/0 (socket says 172.21.15.84:53204) 2024-02-20T00:09:29.500 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:29.496+0000 7f8bc7fff700 1 -- 172.21.15.84:0/4252213245 learned_addr learned my addr 172.21.15.84:0/4252213245 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-02-20T00:09:29.500 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:29.496+0000 7f8bc7fff700 1 -- 172.21.15.84:0/4252213245 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bc80fb7e0 con 0x7f8bc80fad90 2024-02-20T00:09:29.500 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:29.496+0000 7f8bc7fff700 1 --2- 172.21.15.84:0/4252213245 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8bc80fad90 0x7f8bc80fb1b0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f8bb800b5f0 tx=0x7f8bb8006ee0).ready entity=mon.0 client_cookie=216b21d57b48adcb server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:09:29.502 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:29.497+0000 7f8bc6ffd700 1 -- 172.21.15.84:0/4252213245 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8bb800d600 con 0x7f8bc80fad90 2024-02-20T00:09:29.502 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:29.497+0000 7f8bc6ffd700 1 -- 172.21.15.84:0/4252213245 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f8bb800dbf0 con 0x7f8bc80fad90 2024-02-20T00:09:29.502 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:29.497+0000 7f8bc6ffd700 1 -- 172.21.15.84:0/4252213245 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8bb8011cb0 con 0x7f8bc80fad90 2024-02-20T00:09:29.502 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:29.497+0000 7f8bce325700 1 -- 172.21.15.84:0/4252213245 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8bc80fad90 msgr2=0x7f8bc80fb1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:09:29.502 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:29.497+0000 7f8bce325700 1 --2- 172.21.15.84:0/4252213245 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8bc80fad90 0x7f8bc80fb1b0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f8bb800b5f0 tx=0x7f8bb8006ee0).stop 2024-02-20T00:09:29.502 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:29.497+0000 7f8bce325700 1 -- 172.21.15.84:0/4252213245 shutdown_connections 2024-02-20T00:09:29.502 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:29.497+0000 7f8bce325700 1 --2- 172.21.15.84:0/4252213245 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8bc80fad90 0x7f8bc80fb1b0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:09:29.502 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:29.497+0000 7f8bce325700 1 -- 172.21.15.84:0/4252213245 >> 172.21.15.84:0/4252213245 conn(0x7f8bc80f6530 msgr2=0x7f8bc80f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:09:29.502 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:29.497+0000 7f8bce325700 1 -- 172.21.15.84:0/4252213245 shutdown_connections 2024-02-20T00:09:29.502 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:29.497+0000 7f8bce325700 1 -- 172.21.15.84:0/4252213245 wait complete. 2024-02-20T00:09:29.502 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:29.498+0000 7f8bce325700 1 Processor -- start 2024-02-20T00:09:29.502 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:29.498+0000 7f8bce325700 1 -- start start 2024-02-20T00:09:29.503 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:29.498+0000 7f8bce325700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8bc80fad90 0x7f8bc810ab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:09:29.503 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:29.498+0000 7f8bce325700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bc810b000 con 0x7f8bc80fad90 2024-02-20T00:09:29.503 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:29.498+0000 7f8bc7fff700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8bc80fad90 0x7f8bc810ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:09:29.503 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:29.498+0000 7f8bc7fff700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8bc80fad90 0x7f8bc810ab10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:53208/0 (socket says 172.21.15.84:53208) 2024-02-20T00:09:29.503 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:29.498+0000 7f8bc7fff700 1 -- 172.21.15.84:0/3642417738 learned_addr learned my addr 172.21.15.84:0/3642417738 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-02-20T00:09:29.503 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:29.498+0000 7f8bc7fff700 1 -- 172.21.15.84:0/3642417738 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bb800b200 con 0x7f8bc80fad90 2024-02-20T00:09:29.503 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:29.499+0000 7f8bc7fff700 1 --2- 172.21.15.84:0/3642417738 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8bc80fad90 0x7f8bc810ab10 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f8bb8011ea0 tx=0x7f8bb8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:09:29.503 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:29.499+0000 7f8bc4ff9700 1 -- 172.21.15.84:0/3642417738 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8bb801b570 con 0x7f8bc80fad90 2024-02-20T00:09:29.503 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:29.499+0000 7f8bc4ff9700 1 -- 172.21.15.84:0/3642417738 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f8bb801bb60 con 0x7f8bc80fad90 2024-02-20T00:09:29.503 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:29.499+0000 7f8bce325700 1 -- 172.21.15.84:0/3642417738 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8bc810b200 con 0x7f8bc80fad90 2024-02-20T00:09:29.503 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:29.499+0000 7f8bc4ff9700 1 -- 172.21.15.84:0/3642417738 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8bb801a8d0 con 0x7f8bc80fad90 2024-02-20T00:09:29.504 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:29.499+0000 7f8bce325700 1 -- 172.21.15.84:0/3642417738 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8bc810b690 con 0x7f8bc80fad90 2024-02-20T00:09:29.504 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:29.499+0000 7f8bc4ff9700 1 -- 172.21.15.84:0/3642417738 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f8bb800f070 con 0x7f8bc80fad90 2024-02-20T00:09:29.504 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:29.500+0000 7f8bc4ff9700 1 --2- 172.21.15.84:0/3642417738 >> [v2:172.21.15.84:6800/1412904880,v1:172.21.15.84:6801/1412904880] conn(0x7f8bb402fe10 0x7f8bb40322d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:09:29.504 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:29.500+0000 7f8bc4ff9700 1 -- 172.21.15.84:0/3642417738 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f8bb8042320 con 0x7f8bc80fad90 2024-02-20T00:09:29.504 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:29.500+0000 7f8bc77fe700 1 --2- 172.21.15.84:0/3642417738 >> [v2:172.21.15.84:6800/1412904880,v1:172.21.15.84:6801/1412904880] conn(0x7f8bb402fe10 0x7f8bb40322d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:09:29.504 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:29.500+0000 7f8bce325700 1 -- 172.21.15.84:0/3642417738 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8bc8045ad0 con 0x7f8bc80fad90 2024-02-20T00:09:29.504 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:29.500+0000 7f8bc77fe700 1 --2- 172.21.15.84:0/3642417738 >> [v2:172.21.15.84:6800/1412904880,v1:172.21.15.84:6801/1412904880] conn(0x7f8bb402fe10 0x7f8bb40322d0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f8bbc0096a0 tx=0x7f8bbc006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:09:29.506 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:29.504+0000 7f8bc4ff9700 1 -- 172.21.15.84:0/3642417738 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8bb8016070 con 0x7f8bc80fad90 2024-02-20T00:09:29.674 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:29.672+0000 7f8bce325700 1 -- 172.21.15.84:0/3642417738 --> [v2:172.21.15.84:6800/1412904880,v1:172.21.15.84:6801/1412904880] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi084", "addr": "172.21.15.84", "target": ["mon-mgr", ""]}) v1 -- 0x7f8bc8104d00 con 0x7f8bb402fe10 2024-02-20T00:09:30.107 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:29 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:09:29.674373+0000 mgr.smithi084.bmqrnf (mgr.14114) 8 : audit [DBG] from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi084", "addr": "172.21.15.84", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T00:09:30.272 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: Added host 'smithi084' 2024-02-20T00:09:30.272 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:30.268+0000 7f8bc4ff9700 1 -- 172.21.15.84:0/3642417738 <== mgr.14114 v2:172.21.15.84:6800/1412904880 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+23 (secure 0 0 0) 0x7f8bc8104d00 con 0x7f8bb402fe10 2024-02-20T00:09:30.274 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:30.271+0000 7f8bce325700 1 -- 172.21.15.84:0/3642417738 >> [v2:172.21.15.84:6800/1412904880,v1:172.21.15.84:6801/1412904880] conn(0x7f8bb402fe10 msgr2=0x7f8bb40322d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:09:30.274 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:30.271+0000 7f8bce325700 1 --2- 172.21.15.84:0/3642417738 >> [v2:172.21.15.84:6800/1412904880,v1:172.21.15.84:6801/1412904880] conn(0x7f8bb402fe10 0x7f8bb40322d0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f8bbc0096a0 tx=0x7f8bbc006b40).stop 2024-02-20T00:09:30.274 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:30.271+0000 7f8bce325700 1 -- 172.21.15.84:0/3642417738 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8bc80fad90 msgr2=0x7f8bc810ab10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:09:30.274 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:30.271+0000 7f8bce325700 1 --2- 172.21.15.84:0/3642417738 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8bc80fad90 0x7f8bc810ab10 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f8bb8011ea0 tx=0x7f8bb8006ee0).stop 2024-02-20T00:09:30.274 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:30.271+0000 7f8bce325700 1 -- 172.21.15.84:0/3642417738 shutdown_connections 2024-02-20T00:09:30.275 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:30.271+0000 7f8bce325700 1 --2- 172.21.15.84:0/3642417738 >> [v2:172.21.15.84:6800/1412904880,v1:172.21.15.84:6801/1412904880] conn(0x7f8bb402fe10 0x7f8bb40322d0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:09:30.275 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:30.271+0000 7f8bce325700 1 --2- 172.21.15.84:0/3642417738 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8bc80fad90 0x7f8bc810ab10 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:09:30.275 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:30.271+0000 7f8bce325700 1 -- 172.21.15.84:0/3642417738 >> 172.21.15.84:0/3642417738 conn(0x7f8bc80f6530 msgr2=0x7f8bc80f7f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:09:30.275 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:30.271+0000 7f8bce325700 1 -- 172.21.15.84:0/3642417738 shutdown_connections 2024-02-20T00:09:30.275 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:30.271+0000 7f8bce325700 1 -- 172.21.15.84:0/3642417738 wait complete. 2024-02-20T00:09:30.905 INFO:teuthology.orchestra.run.smithi084.stderr:Deploying mon service with default placement... 2024-02-20T00:09:30.906 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi084 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/2a2e6c42-cf84-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp3ot6j0xn:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp4xryone4:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply mon 2024-02-20T00:09:31.451 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:31.404+0000 7fa5d2165700 1 Processor -- start 2024-02-20T00:09:31.451 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:31.404+0000 7fa5d2165700 1 -- start start 2024-02-20T00:09:31.451 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:31.404+0000 7fa5d2165700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa5cc0703f0 0x7fa5cc0fea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-20T00:09:31.452 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:31.404+0000 7fa5d2165700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5cc0fef80 con 0x7fa5cc0703f0 2024-02-20T00:09:31.452 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:31.405+0000 7fa5cb7fe700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa5cc0703f0 0x7fa5cc0fea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:09:31.452 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:31.405+0000 7fa5cb7fe700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa5cc0703f0 0x7fa5cc0fea90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:53214/0 (socket says 172.21.15.84:53214) 2024-02-20T00:09:31.452 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:31.405+0000 7fa5cb7fe700 1 -- 172.21.15.84:0/1323155064 learned_addr learned my addr 172.21.15.84:0/1323155064 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-02-20T00:09:31.452 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:31.405+0000 7fa5cb7fe700 1 -- 172.21.15.84:0/1323155064 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5cc0ff0c0 con 0x7fa5cc0703f0 2024-02-20T00:09:31.452 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:31.405+0000 7fa5cb7fe700 1 --2- 172.21.15.84:0/1323155064 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa5cc0703f0 0x7fa5cc0fea90 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7fa5c000b5f0 tx=0x7fa5c0006ee0).ready entity=mon.0 client_cookie=d2e3655552997406 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:09:31.452 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:31.405+0000 7fa5ca7fc700 1 -- 172.21.15.84:0/1323155064 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa5c000d600 con 0x7fa5cc0703f0 2024-02-20T00:09:31.452 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:31.405+0000 7fa5ca7fc700 1 -- 172.21.15.84:0/1323155064 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fa5c000dbf0 con 0x7fa5cc0703f0 2024-02-20T00:09:31.452 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:31.405+0000 7fa5ca7fc700 1 -- 172.21.15.84:0/1323155064 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa5c0011ce0 con 0x7fa5cc0703f0 2024-02-20T00:09:31.452 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:31.406+0000 7fa5d2165700 1 -- 172.21.15.84:0/1323155064 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa5cc0703f0 msgr2=0x7fa5cc0fea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:09:31.452 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:31.406+0000 7fa5d2165700 1 --2- 172.21.15.84:0/1323155064 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa5cc0703f0 0x7fa5cc0fea90 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7fa5c000b5f0 tx=0x7fa5c0006ee0).stop 2024-02-20T00:09:31.452 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:31.406+0000 7fa5d2165700 1 -- 172.21.15.84:0/1323155064 shutdown_connections 2024-02-20T00:09:31.452 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:31.406+0000 7fa5d2165700 1 --2- 172.21.15.84:0/1323155064 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa5cc0703f0 0x7fa5cc0fea90 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:09:31.453 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:31.406+0000 7fa5d2165700 1 -- 172.21.15.84:0/1323155064 >> 172.21.15.84:0/1323155064 conn(0x7fa5cc0f6530 msgr2=0x7fa5cc0f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:09:31.453 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:31.406+0000 7fa5d2165700 1 -- 172.21.15.84:0/1323155064 shutdown_connections 2024-02-20T00:09:31.453 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:31.406+0000 7fa5d2165700 1 -- 172.21.15.84:0/1323155064 wait complete. 2024-02-20T00:09:31.453 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:31.406+0000 7fa5d2165700 1 Processor -- start 2024-02-20T00:09:31.453 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:31.407+0000 7fa5d2165700 1 -- start start 2024-02-20T00:09:31.453 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:31.407+0000 7fa5d2165700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa5cc0703f0 0x7fa5cc10ab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:09:31.453 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:31.407+0000 7fa5d2165700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5cc10b080 con 0x7fa5cc0703f0 2024-02-20T00:09:31.453 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:31.407+0000 7fa5cb7fe700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa5cc0703f0 0x7fa5cc10ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:09:31.453 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:31.407+0000 7fa5cb7fe700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa5cc0703f0 0x7fa5cc10ab90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:53222/0 (socket says 172.21.15.84:53222) 2024-02-20T00:09:31.453 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:31.407+0000 7fa5cb7fe700 1 -- 172.21.15.84:0/1962967299 learned_addr learned my addr 172.21.15.84:0/1962967299 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-02-20T00:09:31.453 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:31.407+0000 7fa5cb7fe700 1 -- 172.21.15.84:0/1962967299 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5c000b200 con 0x7fa5cc0703f0 2024-02-20T00:09:31.453 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:31.407+0000 7fa5cb7fe700 1 --2- 172.21.15.84:0/1962967299 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa5cc0703f0 0x7fa5cc10ab90 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7fa5c00115c0 tx=0x7fa5c0006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:09:31.453 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:31.407+0000 7fa5b3fff700 1 -- 172.21.15.84:0/1962967299 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa5c001b410 con 0x7fa5cc0703f0 2024-02-20T00:09:31.453 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:31.407+0000 7fa5b3fff700 1 -- 172.21.15.84:0/1962967299 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fa5c001ba00 con 0x7fa5cc0703f0 2024-02-20T00:09:31.453 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:31.407+0000 7fa5d2165700 1 -- 172.21.15.84:0/1962967299 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5cc10b280 con 0x7fa5cc0703f0 2024-02-20T00:09:31.454 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:31.407+0000 7fa5b3fff700 1 -- 172.21.15.84:0/1962967299 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa5c001aaa0 con 0x7fa5cc0703f0 2024-02-20T00:09:31.454 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:31.407+0000 7fa5d2165700 1 -- 172.21.15.84:0/1962967299 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5cc10b710 con 0x7fa5cc0703f0 2024-02-20T00:09:31.454 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:31.408+0000 7fa5b3fff700 1 -- 172.21.15.84:0/1962967299 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fa5c000f070 con 0x7fa5cc0703f0 2024-02-20T00:09:31.454 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:31.408+0000 7fa5b3fff700 1 --2- 172.21.15.84:0/1962967299 >> [v2:172.21.15.84:6800/1412904880,v1:172.21.15.84:6801/1412904880] conn(0x7fa5b802fe70 0x7fa5b8032330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:09:31.454 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:31.408+0000 7fa5b3fff700 1 -- 172.21.15.84:0/1962967299 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7fa5c0040f90 con 0x7fa5cc0703f0 2024-02-20T00:09:31.454 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:31.408+0000 7fa5d2165700 1 -- 172.21.15.84:0/1962967299 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5cc045ad0 con 0x7fa5cc0703f0 2024-02-20T00:09:31.454 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:31.408+0000 7fa5caffd700 1 --2- 172.21.15.84:0/1962967299 >> [v2:172.21.15.84:6800/1412904880,v1:172.21.15.84:6801/1412904880] conn(0x7fa5b802fe70 0x7fa5b8032330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:09:31.454 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:31.409+0000 7fa5caffd700 1 --2- 172.21.15.84:0/1962967299 >> [v2:172.21.15.84:6800/1412904880,v1:172.21.15.84:6801/1412904880] conn(0x7fa5b802fe70 0x7fa5b8032330 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7fa5bc0096a0 tx=0x7fa5bc006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:09:31.454 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:31.412+0000 7fa5b3fff700 1 -- 172.21.15.84:0/1962967299 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa5c0016070 con 0x7fa5cc0703f0 2024-02-20T00:09:31.589 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:31.587+0000 7fa5d2165700 1 -- 172.21.15.84:0/1962967299 --> [v2:172.21.15.84:6800/1412904880,v1:172.21.15.84:6801/1412904880] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}) v1 -- 0x7fa5cc10bf70 con 0x7fa5b802fe70 2024-02-20T00:09:31.593 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:31.591+0000 7fa5b3fff700 1 -- 172.21.15.84:0/1962967299 <== mgr.14114 v2:172.21.15.84:6800/1412904880 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fa5cc10bf70 con 0x7fa5b802fe70 2024-02-20T00:09:31.593 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: Scheduled mon update... 2024-02-20T00:09:31.595 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:31.592+0000 7fa5d2165700 1 -- 172.21.15.84:0/1962967299 >> [v2:172.21.15.84:6800/1412904880,v1:172.21.15.84:6801/1412904880] conn(0x7fa5b802fe70 msgr2=0x7fa5b8032330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:09:31.596 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:31.592+0000 7fa5d2165700 1 --2- 172.21.15.84:0/1962967299 >> [v2:172.21.15.84:6800/1412904880,v1:172.21.15.84:6801/1412904880] conn(0x7fa5b802fe70 0x7fa5b8032330 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7fa5bc0096a0 tx=0x7fa5bc006b40).stop 2024-02-20T00:09:31.596 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:31.592+0000 7fa5d2165700 1 -- 172.21.15.84:0/1962967299 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa5cc0703f0 msgr2=0x7fa5cc10ab90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:09:31.596 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:31.592+0000 7fa5d2165700 1 --2- 172.21.15.84:0/1962967299 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa5cc0703f0 0x7fa5cc10ab90 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7fa5c00115c0 tx=0x7fa5c0006ee0).stop 2024-02-20T00:09:31.596 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:31.592+0000 7fa5d2165700 1 -- 172.21.15.84:0/1962967299 shutdown_connections 2024-02-20T00:09:31.596 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:31.592+0000 7fa5d2165700 1 --2- 172.21.15.84:0/1962967299 >> [v2:172.21.15.84:6800/1412904880,v1:172.21.15.84:6801/1412904880] conn(0x7fa5b802fe70 0x7fa5b8032330 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:09:31.596 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:31.592+0000 7fa5d2165700 1 --2- 172.21.15.84:0/1962967299 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa5cc0703f0 0x7fa5cc10ab90 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:09:31.596 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:31.592+0000 7fa5d2165700 1 -- 172.21.15.84:0/1962967299 >> 172.21.15.84:0/1962967299 conn(0x7fa5cc0f6530 msgr2=0x7fa5cc0f7f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:09:31.596 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:31.592+0000 7fa5d2165700 1 -- 172.21.15.84:0/1962967299 shutdown_connections 2024-02-20T00:09:31.596 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:31.593+0000 7fa5d2165700 1 -- 172.21.15.84:0/1962967299 wait complete. 2024-02-20T00:09:31.607 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:31 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:09:30.269987+0000 mon.smithi084 (mon.0) 54 : audit [INF] from='mgr.14114 172.21.15.84:0/849529258' entity='mgr.smithi084.bmqrnf' 2024-02-20T00:09:31.607 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:31 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cephadm 2024-02-20T00:09:30.270267+0000 mgr.smithi084.bmqrnf (mgr.14114) 9 : cephadm [INF] Added host smithi084 2024-02-20T00:09:31.607 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:31 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:09:30.270521+0000 mon.smithi084 (mon.0) 55 : audit [DBG] from='mgr.14114 172.21.15.84:0/849529258' entity='mgr.smithi084.bmqrnf' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-20T00:09:32.270 INFO:teuthology.orchestra.run.smithi084.stderr:Deploying mgr service with default placement... 2024-02-20T00:09:32.271 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi084 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/2a2e6c42-cf84-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp3ot6j0xn:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp4xryone4:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply mgr 2024-02-20T00:09:32.791 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:32.784+0000 7f909115c700 1 Processor -- start 2024-02-20T00:09:32.791 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:32.784+0000 7f909115c700 1 -- start start 2024-02-20T00:09:32.791 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:32.784+0000 7f909115c700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f908c0fad90 0x7f908c0fb1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-20T00:09:32.791 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:32.784+0000 7f909115c700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f908c0fb6a0 con 0x7f908c0fad90 2024-02-20T00:09:32.791 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:32.784+0000 7f908ad9d700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f908c0fad90 0x7f908c0fb1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:09:32.791 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:32.785+0000 7f908ad9d700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f908c0fad90 0x7f908c0fb1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:53236/0 (socket says 172.21.15.84:53236) 2024-02-20T00:09:32.791 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:32.785+0000 7f908ad9d700 1 -- 172.21.15.84:0/13461503 learned_addr learned my addr 172.21.15.84:0/13461503 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-02-20T00:09:32.791 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:32.785+0000 7f908ad9d700 1 -- 172.21.15.84:0/13461503 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f908c0fb7e0 con 0x7f908c0fad90 2024-02-20T00:09:32.791 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:32.785+0000 7f908ad9d700 1 --2- 172.21.15.84:0/13461503 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f908c0fad90 0x7f908c0fb1b0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f908000b5f0 tx=0x7f9080006ee0).ready entity=mon.0 client_cookie=a3ea9a0ceea7a270 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:09:32.791 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:32.785+0000 7f9089d9b700 1 -- 172.21.15.84:0/13461503 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f908000d600 con 0x7f908c0fad90 2024-02-20T00:09:32.792 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:32.785+0000 7f9089d9b700 1 -- 172.21.15.84:0/13461503 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f908000dbf0 con 0x7f908c0fad90 2024-02-20T00:09:32.792 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:32.785+0000 7f9089d9b700 1 -- 172.21.15.84:0/13461503 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9080011cb0 con 0x7f908c0fad90 2024-02-20T00:09:32.792 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:32.786+0000 7f909115c700 1 -- 172.21.15.84:0/13461503 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f908c0fad90 msgr2=0x7f908c0fb1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:09:32.792 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:32.786+0000 7f909115c700 1 --2- 172.21.15.84:0/13461503 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f908c0fad90 0x7f908c0fb1b0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f908000b5f0 tx=0x7f9080006ee0).stop 2024-02-20T00:09:32.792 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:32.786+0000 7f909115c700 1 -- 172.21.15.84:0/13461503 shutdown_connections 2024-02-20T00:09:32.792 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:32.786+0000 7f909115c700 1 --2- 172.21.15.84:0/13461503 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f908c0fad90 0x7f908c0fb1b0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:09:32.792 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:32.786+0000 7f909115c700 1 -- 172.21.15.84:0/13461503 >> 172.21.15.84:0/13461503 conn(0x7f908c0f6530 msgr2=0x7f908c0f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:09:32.792 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:32.786+0000 7f909115c700 1 -- 172.21.15.84:0/13461503 shutdown_connections 2024-02-20T00:09:32.792 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:32.786+0000 7f909115c700 1 -- 172.21.15.84:0/13461503 wait complete. 2024-02-20T00:09:32.792 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:32.786+0000 7f909115c700 1 Processor -- start 2024-02-20T00:09:32.792 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:32.786+0000 7f909115c700 1 -- start start 2024-02-20T00:09:32.792 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:32.787+0000 7f909115c700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f908c0fad90 0x7f908c108970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:09:32.792 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:32.787+0000 7f909115c700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f908c108e60 con 0x7f908c0fad90 2024-02-20T00:09:32.792 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:32.787+0000 7f908ad9d700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f908c0fad90 0x7f908c108970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:09:32.793 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:32.787+0000 7f908ad9d700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f908c0fad90 0x7f908c108970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:53244/0 (socket says 172.21.15.84:53244) 2024-02-20T00:09:32.793 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:32.787+0000 7f908ad9d700 1 -- 172.21.15.84:0/1622404966 learned_addr learned my addr 172.21.15.84:0/1622404966 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-02-20T00:09:32.793 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:32.787+0000 7f908ad9d700 1 -- 172.21.15.84:0/1622404966 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f908000b200 con 0x7f908c0fad90 2024-02-20T00:09:32.793 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:32.787+0000 7f908ad9d700 1 --2- 172.21.15.84:0/1622404966 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f908c0fad90 0x7f908c108970 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f9080002f40 tx=0x7f9080006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:09:32.793 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:32.787+0000 7f90737fe700 1 -- 172.21.15.84:0/1622404966 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f908001b410 con 0x7f908c0fad90 2024-02-20T00:09:32.793 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:32.787+0000 7f90737fe700 1 -- 172.21.15.84:0/1622404966 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f908001ba00 con 0x7f908c0fad90 2024-02-20T00:09:32.793 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:32.787+0000 7f909115c700 1 -- 172.21.15.84:0/1622404966 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f908c109060 con 0x7f908c0fad90 2024-02-20T00:09:32.793 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:32.787+0000 7f90737fe700 1 -- 172.21.15.84:0/1622404966 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f908001a900 con 0x7f908c0fad90 2024-02-20T00:09:32.793 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:32.787+0000 7f909115c700 1 -- 172.21.15.84:0/1622404966 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f908c1094f0 con 0x7f908c0fad90 2024-02-20T00:09:32.793 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:32.788+0000 7f90737fe700 1 -- 172.21.15.84:0/1622404966 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f908000f070 con 0x7f908c0fad90 2024-02-20T00:09:32.793 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:32.788+0000 7f90737fe700 1 --2- 172.21.15.84:0/1622404966 >> [v2:172.21.15.84:6800/1412904880,v1:172.21.15.84:6801/1412904880] conn(0x7f90780320b0 0x7f9078034570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:09:32.793 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:32.788+0000 7f90737fe700 1 -- 172.21.15.84:0/1622404966 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f90800412e0 con 0x7f908c0fad90 2024-02-20T00:09:32.793 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:32.788+0000 7f908a59c700 1 --2- 172.21.15.84:0/1622404966 >> [v2:172.21.15.84:6800/1412904880,v1:172.21.15.84:6801/1412904880] conn(0x7f90780320b0 0x7f9078034570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:09:32.793 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:32.789+0000 7f908a59c700 1 --2- 172.21.15.84:0/1622404966 >> [v2:172.21.15.84:6800/1412904880,v1:172.21.15.84:6801/1412904880] conn(0x7f90780320b0 0x7f9078034570 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f907c0096a0 tx=0x7f907c006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:09:32.794 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:32.789+0000 7f909115c700 1 -- 172.21.15.84:0/1622404966 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f908c045ad0 con 0x7f908c0fad90 2024-02-20T00:09:32.795 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:32.793+0000 7f90737fe700 1 -- 172.21.15.84:0/1622404966 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9080016070 con 0x7f908c0fad90 2024-02-20T00:09:32.859 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:32 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:09:31.589341+0000 mgr.smithi084.bmqrnf (mgr.14114) 10 : audit [DBG] from='client.14134 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T00:09:32.859 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:32 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cephadm 2024-02-20T00:09:31.590045+0000 mgr.smithi084.bmqrnf (mgr.14114) 11 : cephadm [INF] Saving service mon spec with placement count:5 2024-02-20T00:09:32.859 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:32 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:09:31.592231+0000 mon.smithi084 (mon.0) 56 : audit [INF] from='mgr.14114 172.21.15.84:0/849529258' entity='mgr.smithi084.bmqrnf' 2024-02-20T00:09:32.971 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:32.969+0000 7f909115c700 1 -- 172.21.15.84:0/1622404966 --> [v2:172.21.15.84:6800/1412904880,v1:172.21.15.84:6801/1412904880] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7f908c102840 con 0x7f90780320b0 2024-02-20T00:09:32.975 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:32.972+0000 7f90737fe700 1 -- 172.21.15.84:0/1622404966 <== mgr.14114 v2:172.21.15.84:6800/1412904880 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f908c102840 con 0x7f90780320b0 2024-02-20T00:09:32.975 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: Scheduled mgr update... 2024-02-20T00:09:32.977 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:32.974+0000 7f909115c700 1 -- 172.21.15.84:0/1622404966 >> [v2:172.21.15.84:6800/1412904880,v1:172.21.15.84:6801/1412904880] conn(0x7f90780320b0 msgr2=0x7f9078034570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:09:32.977 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:32.974+0000 7f909115c700 1 --2- 172.21.15.84:0/1622404966 >> [v2:172.21.15.84:6800/1412904880,v1:172.21.15.84:6801/1412904880] conn(0x7f90780320b0 0x7f9078034570 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f907c0096a0 tx=0x7f907c006b40).stop 2024-02-20T00:09:32.977 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:32.974+0000 7f909115c700 1 -- 172.21.15.84:0/1622404966 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f908c0fad90 msgr2=0x7f908c108970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:09:32.978 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:32.974+0000 7f909115c700 1 --2- 172.21.15.84:0/1622404966 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f908c0fad90 0x7f908c108970 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f9080002f40 tx=0x7f9080006ee0).stop 2024-02-20T00:09:32.978 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:32.974+0000 7f909115c700 1 -- 172.21.15.84:0/1622404966 shutdown_connections 2024-02-20T00:09:32.978 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:32.974+0000 7f909115c700 1 --2- 172.21.15.84:0/1622404966 >> [v2:172.21.15.84:6800/1412904880,v1:172.21.15.84:6801/1412904880] conn(0x7f90780320b0 0x7f9078034570 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:09:32.978 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:32.974+0000 7f909115c700 1 --2- 172.21.15.84:0/1622404966 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f908c0fad90 0x7f908c108970 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:09:32.978 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:32.974+0000 7f909115c700 1 -- 172.21.15.84:0/1622404966 >> 172.21.15.84:0/1622404966 conn(0x7f908c0f6530 msgr2=0x7f908c0ff9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:09:32.978 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:32.975+0000 7f909115c700 1 -- 172.21.15.84:0/1622404966 shutdown_connections 2024-02-20T00:09:32.978 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:32.975+0000 7f909115c700 1 -- 172.21.15.84:0/1622404966 wait complete. 2024-02-20T00:09:33.666 INFO:teuthology.orchestra.run.smithi084.stderr:Deploying crash service with default placement... 2024-02-20T00:09:33.666 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi084 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/2a2e6c42-cf84-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp3ot6j0xn:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp4xryone4:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply crash 2024-02-20T00:09:34.281 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:34.271+0000 7f2a99347700 1 Processor -- start 2024-02-20T00:09:34.282 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:34.271+0000 7f2a99347700 1 -- start start 2024-02-20T00:09:34.282 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:34.272+0000 7f2a99347700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2a940fbe30 0x7f2a940fc250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-20T00:09:34.282 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:34.272+0000 7f2a99347700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a940fc740 con 0x7f2a940fbe30 2024-02-20T00:09:34.282 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:34.272+0000 7f2a92ffd700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2a940fbe30 0x7f2a940fc250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:09:34.282 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:34.272+0000 7f2a92ffd700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2a940fbe30 0x7f2a940fc250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:53254/0 (socket says 172.21.15.84:53254) 2024-02-20T00:09:34.282 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:34.272+0000 7f2a92ffd700 1 -- 172.21.15.84:0/184369528 learned_addr learned my addr 172.21.15.84:0/184369528 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-02-20T00:09:34.282 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:34.272+0000 7f2a92ffd700 1 -- 172.21.15.84:0/184369528 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a940fc880 con 0x7f2a940fbe30 2024-02-20T00:09:34.282 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:34.272+0000 7f2a92ffd700 1 --2- 172.21.15.84:0/184369528 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2a940fbe30 0x7f2a940fc250 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f2a8800b5f0 tx=0x7f2a88006ee0).ready entity=mon.0 client_cookie=462af1cec227dfc3 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:09:34.282 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:34.272+0000 7f2a91ffb700 1 -- 172.21.15.84:0/184369528 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2a8800d600 con 0x7f2a940fbe30 2024-02-20T00:09:34.282 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:34.272+0000 7f2a91ffb700 1 -- 172.21.15.84:0/184369528 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f2a8800dbf0 con 0x7f2a940fbe30 2024-02-20T00:09:34.282 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:34.272+0000 7f2a91ffb700 1 -- 172.21.15.84:0/184369528 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2a88011cb0 con 0x7f2a940fbe30 2024-02-20T00:09:34.282 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:34.273+0000 7f2a99347700 1 -- 172.21.15.84:0/184369528 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2a940fbe30 msgr2=0x7f2a940fc250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:09:34.282 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:34.273+0000 7f2a99347700 1 --2- 172.21.15.84:0/184369528 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2a940fbe30 0x7f2a940fc250 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f2a8800b5f0 tx=0x7f2a88006ee0).stop 2024-02-20T00:09:34.282 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:34.273+0000 7f2a99347700 1 -- 172.21.15.84:0/184369528 shutdown_connections 2024-02-20T00:09:34.282 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:34.273+0000 7f2a99347700 1 --2- 172.21.15.84:0/184369528 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2a940fbe30 0x7f2a940fc250 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:09:34.283 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:34.273+0000 7f2a99347700 1 -- 172.21.15.84:0/184369528 >> 172.21.15.84:0/184369528 conn(0x7f2a940f7670 msgr2=0x7f2a940f9a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:09:34.283 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:34.273+0000 7f2a99347700 1 -- 172.21.15.84:0/184369528 shutdown_connections 2024-02-20T00:09:34.283 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:34.273+0000 7f2a99347700 1 -- 172.21.15.84:0/184369528 wait complete. 2024-02-20T00:09:34.283 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:34.273+0000 7f2a99347700 1 Processor -- start 2024-02-20T00:09:34.283 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:34.273+0000 7f2a99347700 1 -- start start 2024-02-20T00:09:34.283 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:34.274+0000 7f2a99347700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2a940fbe30 0x7f2a940725e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:09:34.283 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:34.274+0000 7f2a99347700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a94070a60 con 0x7f2a940fbe30 2024-02-20T00:09:34.283 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:34.274+0000 7f2a92ffd700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2a940fbe30 0x7f2a940725e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:09:34.283 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:34.274+0000 7f2a92ffd700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2a940fbe30 0x7f2a940725e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:53270/0 (socket says 172.21.15.84:53270) 2024-02-20T00:09:34.283 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:34.274+0000 7f2a92ffd700 1 -- 172.21.15.84:0/2354019086 learned_addr learned my addr 172.21.15.84:0/2354019086 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-02-20T00:09:34.283 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:34.274+0000 7f2a92ffd700 1 -- 172.21.15.84:0/2354019086 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a8800b200 con 0x7f2a940fbe30 2024-02-20T00:09:34.283 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:34.274+0000 7f2a92ffd700 1 --2- 172.21.15.84:0/2354019086 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2a940fbe30 0x7f2a940725e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f2a88011ea0 tx=0x7f2a88006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:09:34.283 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:34.274+0000 7f2a7b7fe700 1 -- 172.21.15.84:0/2354019086 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2a8801b570 con 0x7f2a940fbe30 2024-02-20T00:09:34.283 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:34.274+0000 7f2a7b7fe700 1 -- 172.21.15.84:0/2354019086 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f2a8801bb60 con 0x7f2a940fbe30 2024-02-20T00:09:34.284 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:34.274+0000 7f2a99347700 1 -- 172.21.15.84:0/2354019086 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a94070c00 con 0x7f2a940fbe30 2024-02-20T00:09:34.284 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:34.274+0000 7f2a7b7fe700 1 -- 172.21.15.84:0/2354019086 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2a8801a8d0 con 0x7f2a940fbe30 2024-02-20T00:09:34.284 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:34.274+0000 7f2a99347700 1 -- 172.21.15.84:0/2354019086 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a94071090 con 0x7f2a940fbe30 2024-02-20T00:09:34.284 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:34.275+0000 7f2a7b7fe700 1 -- 172.21.15.84:0/2354019086 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f2a8800f070 con 0x7f2a940fbe30 2024-02-20T00:09:34.284 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:34.275+0000 7f2a7b7fe700 1 --2- 172.21.15.84:0/2354019086 >> [v2:172.21.15.84:6800/1412904880,v1:172.21.15.84:6801/1412904880] conn(0x7f2a8002fe10 0x7f2a800322d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:09:34.284 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:34.275+0000 7f2a7b7fe700 1 -- 172.21.15.84:0/2354019086 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f2a88041470 con 0x7f2a940fbe30 2024-02-20T00:09:34.284 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:34.275+0000 7f2a927fc700 1 --2- 172.21.15.84:0/2354019086 >> [v2:172.21.15.84:6800/1412904880,v1:172.21.15.84:6801/1412904880] conn(0x7f2a8002fe10 0x7f2a800322d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:09:34.284 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:34.275+0000 7f2a99347700 1 -- 172.21.15.84:0/2354019086 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a94059100 con 0x7f2a940fbe30 2024-02-20T00:09:34.284 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:34.276+0000 7f2a927fc700 1 --2- 172.21.15.84:0/2354019086 >> [v2:172.21.15.84:6800/1412904880,v1:172.21.15.84:6801/1412904880] conn(0x7f2a8002fe10 0x7f2a800322d0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f2a840096a0 tx=0x7f2a84006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:09:34.284 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:34.281+0000 7f2a7b7fe700 1 -- 172.21.15.84:0/2354019086 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2a88016070 con 0x7f2a940fbe30 2024-02-20T00:09:34.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:33 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:09:32.971211+0000 mgr.smithi084.bmqrnf (mgr.14114) 12 : audit [DBG] from='client.14136 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T00:09:34.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:33 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cephadm 2024-02-20T00:09:32.971715+0000 mgr.smithi084.bmqrnf (mgr.14114) 13 : cephadm [INF] Saving service mgr spec with placement count:2 2024-02-20T00:09:34.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:33 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:09:32.973939+0000 mon.smithi084 (mon.0) 57 : audit [INF] from='mgr.14114 172.21.15.84:0/849529258' entity='mgr.smithi084.bmqrnf' 2024-02-20T00:09:34.458 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:34.455+0000 7f2a99347700 1 -- 172.21.15.84:0/2354019086 --> [v2:172.21.15.84:6800/1412904880,v1:172.21.15.84:6801/1412904880] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}) v1 -- 0x7f2a94104b50 con 0x7f2a8002fe10 2024-02-20T00:09:34.462 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:34.459+0000 7f2a7b7fe700 1 -- 172.21.15.84:0/2354019086 <== mgr.14114 v2:172.21.15.84:6800/1412904880 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+26 (secure 0 0 0) 0x7f2a94104b50 con 0x7f2a8002fe10 2024-02-20T00:09:34.462 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: Scheduled crash update... 2024-02-20T00:09:34.464 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:34.461+0000 7f2a99347700 1 -- 172.21.15.84:0/2354019086 >> [v2:172.21.15.84:6800/1412904880,v1:172.21.15.84:6801/1412904880] conn(0x7f2a8002fe10 msgr2=0x7f2a800322d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:09:34.464 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:34.461+0000 7f2a99347700 1 --2- 172.21.15.84:0/2354019086 >> [v2:172.21.15.84:6800/1412904880,v1:172.21.15.84:6801/1412904880] conn(0x7f2a8002fe10 0x7f2a800322d0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f2a840096a0 tx=0x7f2a84006b40).stop 2024-02-20T00:09:34.465 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:34.461+0000 7f2a99347700 1 -- 172.21.15.84:0/2354019086 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2a940fbe30 msgr2=0x7f2a940725e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:09:34.465 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:34.461+0000 7f2a99347700 1 --2- 172.21.15.84:0/2354019086 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2a940fbe30 0x7f2a940725e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f2a88011ea0 tx=0x7f2a88006ee0).stop 2024-02-20T00:09:34.465 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:34.461+0000 7f2a99347700 1 -- 172.21.15.84:0/2354019086 shutdown_connections 2024-02-20T00:09:34.465 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:34.461+0000 7f2a99347700 1 --2- 172.21.15.84:0/2354019086 >> [v2:172.21.15.84:6800/1412904880,v1:172.21.15.84:6801/1412904880] conn(0x7f2a8002fe10 0x7f2a800322d0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:09:34.465 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:34.461+0000 7f2a99347700 1 --2- 172.21.15.84:0/2354019086 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2a940fbe30 0x7f2a940725e0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:09:34.465 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:34.461+0000 7f2a99347700 1 -- 172.21.15.84:0/2354019086 >> 172.21.15.84:0/2354019086 conn(0x7f2a940f7670 msgr2=0x7f2a940f8f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:09:34.465 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:34.461+0000 7f2a99347700 1 -- 172.21.15.84:0/2354019086 shutdown_connections 2024-02-20T00:09:34.465 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:34.461+0000 7f2a99347700 1 -- 172.21.15.84:0/2354019086 wait complete. 2024-02-20T00:09:35.041 INFO:teuthology.orchestra.run.smithi084.stderr:Enabling mgr prometheus module... 2024-02-20T00:09:35.042 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi084 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/2a2e6c42-cf84-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp3ot6j0xn:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp4xryone4:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable prometheus 2024-02-20T00:09:35.562 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:35.544+0000 7f23b73cf700 1 Processor -- start 2024-02-20T00:09:35.562 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:35.544+0000 7f23b73cf700 1 -- start start 2024-02-20T00:09:35.563 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:35.544+0000 7f23b73cf700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f23b00fad90 0x7f23b00fb1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-20T00:09:35.563 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:35.544+0000 7f23b73cf700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23b00fb6a0 con 0x7f23b00fad90 2024-02-20T00:09:35.563 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:35.544+0000 7f23b516b700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f23b00fad90 0x7f23b00fb1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:09:35.563 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:35.544+0000 7f23b516b700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f23b00fad90 0x7f23b00fb1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:53276/0 (socket says 172.21.15.84:53276) 2024-02-20T00:09:35.563 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:35.544+0000 7f23b516b700 1 -- 172.21.15.84:0/3388898530 learned_addr learned my addr 172.21.15.84:0/3388898530 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-02-20T00:09:35.563 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:35.545+0000 7f23b516b700 1 -- 172.21.15.84:0/3388898530 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23b00fb7e0 con 0x7f23b00fad90 2024-02-20T00:09:35.563 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:35.545+0000 7f23b516b700 1 --2- 172.21.15.84:0/3388898530 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f23b00fad90 0x7f23b00fb1b0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f23a000b5f0 tx=0x7f23a0006ee0).ready entity=mon.0 client_cookie=bfeab6b19558b781 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:09:35.563 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:35.545+0000 7f239ffff700 1 -- 172.21.15.84:0/3388898530 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f23a000d600 con 0x7f23b00fad90 2024-02-20T00:09:35.563 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:35.545+0000 7f239ffff700 1 -- 172.21.15.84:0/3388898530 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f23a000dbf0 con 0x7f23b00fad90 2024-02-20T00:09:35.563 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:35.545+0000 7f239ffff700 1 -- 172.21.15.84:0/3388898530 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f23a0011cb0 con 0x7f23b00fad90 2024-02-20T00:09:35.563 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:35.545+0000 7f23b73cf700 1 -- 172.21.15.84:0/3388898530 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f23b00fad90 msgr2=0x7f23b00fb1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:09:35.563 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:35.545+0000 7f23b73cf700 1 --2- 172.21.15.84:0/3388898530 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f23b00fad90 0x7f23b00fb1b0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f23a000b5f0 tx=0x7f23a0006ee0).stop 2024-02-20T00:09:35.563 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:35.545+0000 7f23b73cf700 1 -- 172.21.15.84:0/3388898530 shutdown_connections 2024-02-20T00:09:35.563 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:35.545+0000 7f23b73cf700 1 --2- 172.21.15.84:0/3388898530 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f23b00fad90 0x7f23b00fb1b0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:09:35.564 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:35.545+0000 7f23b73cf700 1 -- 172.21.15.84:0/3388898530 >> 172.21.15.84:0/3388898530 conn(0x7f23b00f6530 msgr2=0x7f23b00f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:09:35.564 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:35.546+0000 7f23b73cf700 1 -- 172.21.15.84:0/3388898530 shutdown_connections 2024-02-20T00:09:35.564 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:35.546+0000 7f23b73cf700 1 -- 172.21.15.84:0/3388898530 wait complete. 2024-02-20T00:09:35.564 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:35.546+0000 7f23b73cf700 1 Processor -- start 2024-02-20T00:09:35.564 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:35.546+0000 7f23b73cf700 1 -- start start 2024-02-20T00:09:35.564 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:35.546+0000 7f23b73cf700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f23b00fad90 0x7f23b0109a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:09:35.564 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:35.546+0000 7f23b73cf700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23b0109ef0 con 0x7f23b00fad90 2024-02-20T00:09:35.564 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:35.546+0000 7f23b516b700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f23b00fad90 0x7f23b0109a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:09:35.564 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:35.546+0000 7f23b516b700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f23b00fad90 0x7f23b0109a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:53282/0 (socket says 172.21.15.84:53282) 2024-02-20T00:09:35.564 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:35.546+0000 7f23b516b700 1 -- 172.21.15.84:0/214661533 learned_addr learned my addr 172.21.15.84:0/214661533 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-02-20T00:09:35.564 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:35.546+0000 7f23b516b700 1 -- 172.21.15.84:0/214661533 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23a000b200 con 0x7f23b00fad90 2024-02-20T00:09:35.564 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:35.547+0000 7f23b516b700 1 --2- 172.21.15.84:0/214661533 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f23b00fad90 0x7f23b0109a00 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f23a0011ea0 tx=0x7f23a0006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:09:35.564 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:35.547+0000 7f239dffb700 1 -- 172.21.15.84:0/214661533 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f23a001b570 con 0x7f23b00fad90 2024-02-20T00:09:35.564 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:35.547+0000 7f239dffb700 1 -- 172.21.15.84:0/214661533 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f23a001bb60 con 0x7f23b00fad90 2024-02-20T00:09:35.565 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:35.547+0000 7f239dffb700 1 -- 172.21.15.84:0/214661533 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f23a001a8d0 con 0x7f23b00fad90 2024-02-20T00:09:35.565 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:35.547+0000 7f23b73cf700 1 -- 172.21.15.84:0/214661533 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23b010a0f0 con 0x7f23b00fad90 2024-02-20T00:09:35.565 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:35.547+0000 7f23b73cf700 1 -- 172.21.15.84:0/214661533 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23b010a580 con 0x7f23b00fad90 2024-02-20T00:09:35.565 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:35.548+0000 7f239dffb700 1 -- 172.21.15.84:0/214661533 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f23a000f070 con 0x7f23b00fad90 2024-02-20T00:09:35.565 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:35.548+0000 7f239dffb700 1 --2- 172.21.15.84:0/214661533 >> [v2:172.21.15.84:6800/1412904880,v1:172.21.15.84:6801/1412904880] conn(0x7f239802fe70 0x7f2398032330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:09:35.565 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:35.548+0000 7f239dffb700 1 -- 172.21.15.84:0/214661533 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f23a0041420 con 0x7f23b00fad90 2024-02-20T00:09:35.565 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:35.548+0000 7f23b73cf700 1 -- 172.21.15.84:0/214661533 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23b0045ad0 con 0x7f23b00fad90 2024-02-20T00:09:35.565 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:35.548+0000 7f23b496a700 1 --2- 172.21.15.84:0/214661533 >> [v2:172.21.15.84:6800/1412904880,v1:172.21.15.84:6801/1412904880] conn(0x7f239802fe70 0x7f2398032330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:09:35.565 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:35.548+0000 7f23b496a700 1 --2- 172.21.15.84:0/214661533 >> [v2:172.21.15.84:6800/1412904880,v1:172.21.15.84:6801/1412904880] conn(0x7f239802fe70 0x7f2398032330 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f23ac0096a0 tx=0x7f23ac006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:09:35.565 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:35.552+0000 7f239dffb700 1 -- 172.21.15.84:0/214661533 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f23a0016070 con 0x7f23b00fad90 2024-02-20T00:09:35.742 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:35.740+0000 7f23b73cf700 1 -- 172.21.15.84:0/214661533 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "prometheus"} v 0) v1 -- 0x7f23b00fcae0 con 0x7f23b00fad90 2024-02-20T00:09:35.858 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:35 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:09:34.457833+0000 mgr.smithi084.bmqrnf (mgr.14114) 14 : audit [DBG] from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T00:09:35.858 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:35 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cephadm 2024-02-20T00:09:34.458476+0000 mgr.smithi084.bmqrnf (mgr.14114) 15 : cephadm [INF] Saving service crash spec with placement * 2024-02-20T00:09:35.858 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:35 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:09:34.460883+0000 mon.smithi084 (mon.0) 58 : audit [INF] from='mgr.14114 172.21.15.84:0/849529258' entity='mgr.smithi084.bmqrnf' 2024-02-20T00:09:36.466 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:36.463+0000 7f239dffb700 1 -- 172.21.15.84:0/214661533 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mgrmap(e 9) v1 ==== 36134+0+0 (secure 0 0 0) 0x7f23a0022d80 con 0x7f23b00fad90 2024-02-20T00:09:36.467 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:36.464+0000 7f239dffb700 1 -- 172.21.15.84:0/214661533 <== mon.0 v2:172.21.15.84:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "prometheus"}]=0 v9) v1 ==== 89+0+0 (secure 0 0 0) 0x7f23a002f8a0 con 0x7f23b00fad90 2024-02-20T00:09:36.469 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:36.467+0000 7f23b73cf700 1 -- 172.21.15.84:0/214661533 >> [v2:172.21.15.84:6800/1412904880,v1:172.21.15.84:6801/1412904880] conn(0x7f239802fe70 msgr2=0x7f2398032330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:09:36.469 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:36.467+0000 7f23b73cf700 1 --2- 172.21.15.84:0/214661533 >> [v2:172.21.15.84:6800/1412904880,v1:172.21.15.84:6801/1412904880] conn(0x7f239802fe70 0x7f2398032330 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f23ac0096a0 tx=0x7f23ac006b40).stop 2024-02-20T00:09:36.470 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:36.467+0000 7f23b73cf700 1 -- 172.21.15.84:0/214661533 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f23b00fad90 msgr2=0x7f23b0109a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:09:36.470 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:36.467+0000 7f23b73cf700 1 --2- 172.21.15.84:0/214661533 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f23b00fad90 0x7f23b0109a00 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f23a0011ea0 tx=0x7f23a0006ee0).stop 2024-02-20T00:09:36.471 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:36.467+0000 7f23b73cf700 1 -- 172.21.15.84:0/214661533 shutdown_connections 2024-02-20T00:09:36.471 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:36.467+0000 7f23b73cf700 1 --2- 172.21.15.84:0/214661533 >> [v2:172.21.15.84:6800/1412904880,v1:172.21.15.84:6801/1412904880] conn(0x7f239802fe70 0x7f2398032330 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:09:36.471 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:36.467+0000 7f23b73cf700 1 --2- 172.21.15.84:0/214661533 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f23b00fad90 0x7f23b0109a00 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:09:36.471 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:36.467+0000 7f23b73cf700 1 -- 172.21.15.84:0/214661533 >> 172.21.15.84:0/214661533 conn(0x7f23b00f6530 msgr2=0x7f23b00f7f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:09:36.471 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:36.467+0000 7f23b73cf700 1 -- 172.21.15.84:0/214661533 shutdown_connections 2024-02-20T00:09:36.471 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:36.467+0000 7f23b73cf700 1 -- 172.21.15.84:0/214661533 wait complete. 2024-02-20T00:09:36.837 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:36 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:09:35.741893+0000 mon.smithi084 (mon.0) 59 : audit [INF] from='client.? 172.21.15.84:0/214661533' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-02-20T00:09:37.139 INFO:teuthology.orchestra.run.smithi084.stderr:Deploying prometheus service with default placement... 2024-02-20T00:09:37.139 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi084 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/2a2e6c42-cf84-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp3ot6j0xn:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp4xryone4:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply prometheus 2024-02-20T00:09:37.787 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:37.711+0000 7fb1e3fbd700 1 Processor -- start 2024-02-20T00:09:37.787 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:37.711+0000 7fb1e3fbd700 1 -- start start 2024-02-20T00:09:37.787 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:37.711+0000 7fb1e3fbd700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb1dc0703f0 0x7fb1dc0fead0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-20T00:09:37.787 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:37.711+0000 7fb1e3fbd700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1dc0fefc0 con 0x7fb1dc0703f0 2024-02-20T00:09:37.787 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:37.711+0000 7fb1e1d59700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb1dc0703f0 0x7fb1dc0fead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:09:37.787 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:37.711+0000 7fb1e1d59700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb1dc0703f0 0x7fb1dc0fead0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:53314/0 (socket says 172.21.15.84:53314) 2024-02-20T00:09:37.787 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:37.711+0000 7fb1e1d59700 1 -- 172.21.15.84:0/2264387091 learned_addr learned my addr 172.21.15.84:0/2264387091 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-02-20T00:09:37.787 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:37.712+0000 7fb1e1d59700 1 -- 172.21.15.84:0/2264387091 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1dc0ff100 con 0x7fb1dc0703f0 2024-02-20T00:09:37.787 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:37.712+0000 7fb1e1d59700 1 --2- 172.21.15.84:0/2264387091 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb1dc0703f0 0x7fb1dc0fead0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7fb1d800b5f0 tx=0x7fb1d8006ee0).ready entity=mon.0 client_cookie=4c52a1fe58c963f4 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:09:37.787 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:37.712+0000 7fb1e0d57700 1 -- 172.21.15.84:0/2264387091 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb1d800d600 con 0x7fb1dc0703f0 2024-02-20T00:09:37.788 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:37.712+0000 7fb1e0d57700 1 -- 172.21.15.84:0/2264387091 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fb1d800dbf0 con 0x7fb1dc0703f0 2024-02-20T00:09:37.788 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:37.712+0000 7fb1e0d57700 1 -- 172.21.15.84:0/2264387091 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb1d8011ce0 con 0x7fb1dc0703f0 2024-02-20T00:09:37.788 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:37.713+0000 7fb1e3fbd700 1 -- 172.21.15.84:0/2264387091 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb1dc0703f0 msgr2=0x7fb1dc0fead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:09:37.788 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:37.713+0000 7fb1e3fbd700 1 --2- 172.21.15.84:0/2264387091 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb1dc0703f0 0x7fb1dc0fead0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7fb1d800b5f0 tx=0x7fb1d8006ee0).stop 2024-02-20T00:09:37.788 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:37.713+0000 7fb1e3fbd700 1 -- 172.21.15.84:0/2264387091 shutdown_connections 2024-02-20T00:09:37.788 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:37.713+0000 7fb1e3fbd700 1 --2- 172.21.15.84:0/2264387091 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb1dc0703f0 0x7fb1dc0fead0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:09:37.788 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:37.713+0000 7fb1e3fbd700 1 -- 172.21.15.84:0/2264387091 >> 172.21.15.84:0/2264387091 conn(0x7fb1dc0f6550 msgr2=0x7fb1dc0f8970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:09:37.788 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:37.713+0000 7fb1e3fbd700 1 -- 172.21.15.84:0/2264387091 shutdown_connections 2024-02-20T00:09:37.788 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:37.713+0000 7fb1e3fbd700 1 -- 172.21.15.84:0/2264387091 wait complete. 2024-02-20T00:09:37.788 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:37.714+0000 7fb1e3fbd700 1 Processor -- start 2024-02-20T00:09:37.788 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:37.714+0000 7fb1e3fbd700 1 -- start start 2024-02-20T00:09:37.788 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:37.714+0000 7fb1e3fbd700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb1dc0703f0 0x7fb1dc119990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:09:37.788 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:37.714+0000 7fb1e3fbd700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1dc119e80 con 0x7fb1dc0703f0 2024-02-20T00:09:37.789 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:37.714+0000 7fb1e1d59700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb1dc0703f0 0x7fb1dc119990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:09:37.789 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:37.715+0000 7fb1e1d59700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb1dc0703f0 0x7fb1dc119990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:53328/0 (socket says 172.21.15.84:53328) 2024-02-20T00:09:37.789 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:37.715+0000 7fb1e1d59700 1 -- 172.21.15.84:0/386744130 learned_addr learned my addr 172.21.15.84:0/386744130 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-02-20T00:09:37.789 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:37.716+0000 7fb1e1d59700 1 -- 172.21.15.84:0/386744130 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1d800b200 con 0x7fb1dc0703f0 2024-02-20T00:09:37.789 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:37.716+0000 7fb1e1d59700 1 --2- 172.21.15.84:0/386744130 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb1dc0703f0 0x7fb1dc119990 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7fb1d800df90 tx=0x7fb1d8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:09:37.789 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:37.717+0000 7fb1ca7fc700 1 -- 172.21.15.84:0/386744130 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb1d802c580 con 0x7fb1dc0703f0 2024-02-20T00:09:37.789 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:37.717+0000 7fb1ca7fc700 1 -- 172.21.15.84:0/386744130 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fb1d802cb70 con 0x7fb1dc0703f0 2024-02-20T00:09:37.789 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:37.717+0000 7fb1e3fbd700 1 -- 172.21.15.84:0/386744130 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1dc11a080 con 0x7fb1dc0703f0 2024-02-20T00:09:37.789 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:37.717+0000 7fb1e3fbd700 1 -- 172.21.15.84:0/386744130 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1dc11a510 con 0x7fb1dc0703f0 2024-02-20T00:09:37.789 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:37.717+0000 7fb1ca7fc700 1 -- 172.21.15.84:0/386744130 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb1d802bb20 con 0x7fb1dc0703f0 2024-02-20T00:09:37.789 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:37.718+0000 7fb1e3fbd700 1 -- 172.21.15.84:0/386744130 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1bc004fa0 con 0x7fb1dc0703f0 2024-02-20T00:09:37.789 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:37.718+0000 7fb1ca7fc700 1 -- 172.21.15.84:0/386744130 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 9) v1 ==== 36134+0+0 (secure 0 0 0) 0x7fb1d800f070 con 0x7fb1dc0703f0 2024-02-20T00:09:37.789 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:37.719+0000 7fb1ca7fc700 1 --2- 172.21.15.84:0/386744130 >> [v2:172.21.15.84:6800/1412904880,v1:172.21.15.84:6801/1412904880] conn(0x7fb1c402ff50 0x7fb1c4032410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:09:37.789 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:37.719+0000 7fb1ca7fc700 1 -- 172.21.15.84:0/386744130 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7fb1d80521d0 con 0x7fb1dc0703f0 2024-02-20T00:09:37.789 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:37.719+0000 7fb1e1558700 1 -- 172.21.15.84:0/386744130 >> [v2:172.21.15.84:6800/1412904880,v1:172.21.15.84:6801/1412904880] conn(0x7fb1c402ff50 msgr2=0x7fb1c4032410 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.84:6800/1412904880 2024-02-20T00:09:37.790 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:37.719+0000 7fb1e1558700 1 --2- 172.21.15.84:0/386744130 >> [v2:172.21.15.84:6800/1412904880,v1:172.21.15.84:6801/1412904880] conn(0x7fb1c402ff50 0x7fb1c4032410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-02-20T00:09:37.790 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:37.726+0000 7fb1ca7fc700 1 -- 172.21.15.84:0/386744130 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb1d802ab00 con 0x7fb1dc0703f0 2024-02-20T00:09:37.857 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:37 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:09:36.466220+0000 mon.smithi084 (mon.0) 60 : audit [INF] from='client.? 172.21.15.84:0/214661533' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-02-20T00:09:37.857 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:37 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:09:36.466292+0000 mon.smithi084 (mon.0) 61 : cluster [DBG] mgrmap e9: smithi084.bmqrnf(active, since 19s) 2024-02-20T00:09:37.921 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:37.920+0000 7fb1e1558700 1 -- 172.21.15.84:0/386744130 >> [v2:172.21.15.84:6800/1412904880,v1:172.21.15.84:6801/1412904880] conn(0x7fb1c402ff50 msgr2=0x7fb1c4032410 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.84:6800/1412904880 2024-02-20T00:09:37.921 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:37.920+0000 7fb1e1558700 1 --2- 172.21.15.84:0/386744130 >> [v2:172.21.15.84:6800/1412904880,v1:172.21.15.84:6801/1412904880] conn(0x7fb1c402ff50 0x7fb1c4032410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2024-02-20T00:09:37.938 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:37.937+0000 7fb1e3fbd700 1 -- 172.21.15.84:0/386744130 --> [v2:172.21.15.84:6800/1412904880,v1:172.21.15.84:6801/1412904880] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7fb1bc000bc0 con 0x7fb1c402ff50 2024-02-20T00:09:38.323 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:38.321+0000 7fb1e1558700 1 -- 172.21.15.84:0/386744130 >> [v2:172.21.15.84:6800/1412904880,v1:172.21.15.84:6801/1412904880] conn(0x7fb1c402ff50 msgr2=0x7fb1c4032410 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.84:6800/1412904880 2024-02-20T00:09:38.323 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:38.321+0000 7fb1e1558700 1 --2- 172.21.15.84:0/386744130 >> [v2:172.21.15.84:6800/1412904880,v1:172.21.15.84:6801/1412904880] conn(0x7fb1c402ff50 0x7fb1c4032410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2024-02-20T00:09:39.123 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:39.122+0000 7fb1e1558700 1 -- 172.21.15.84:0/386744130 >> [v2:172.21.15.84:6800/1412904880,v1:172.21.15.84:6801/1412904880] conn(0x7fb1c402ff50 msgr2=0x7fb1c4032410 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.84:6800/1412904880 2024-02-20T00:09:39.124 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:39.122+0000 7fb1e1558700 1 --2- 172.21.15.84:0/386744130 >> [v2:172.21.15.84:6800/1412904880,v1:172.21.15.84:6801/1412904880] conn(0x7fb1c402ff50 0x7fb1c4032410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2024-02-20T00:09:40.724 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:40.723+0000 7fb1e1558700 1 -- 172.21.15.84:0/386744130 >> [v2:172.21.15.84:6800/1412904880,v1:172.21.15.84:6801/1412904880] conn(0x7fb1c402ff50 msgr2=0x7fb1c4032410 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.84:6800/1412904880 2024-02-20T00:09:40.724 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:40.723+0000 7fb1e1558700 1 --2- 172.21.15.84:0/386744130 >> [v2:172.21.15.84:6800/1412904880,v1:172.21.15.84:6801/1412904880] conn(0x7fb1c402ff50 0x7fb1c4032410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 3.200000 2024-02-20T00:09:41.832 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:41.830+0000 7fb1ca7fc700 1 -- 172.21.15.84:0/386744130 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mgrmap(e 10) v1 ==== 35984+0+0 (secure 0 0 0) 0x7fb1d802b380 con 0x7fb1dc0703f0 2024-02-20T00:09:41.833 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:41.831+0000 7fb1ca7fc700 1 -- 172.21.15.84:0/386744130 >> [v2:172.21.15.84:6800/1412904880,v1:172.21.15.84:6801/1412904880] conn(0x7fb1c402ff50 msgr2=0x7fb1c4032410 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-20T00:09:41.833 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:41.831+0000 7fb1ca7fc700 1 --2- 172.21.15.84:0/386744130 >> [v2:172.21.15.84:6800/1412904880,v1:172.21.15.84:6801/1412904880] conn(0x7fb1c402ff50 0x7fb1c4032410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-20T00:09:42.838 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:42.834+0000 7fb1ca7fc700 1 -- 172.21.15.84:0/386744130 <== mon.0 v2:172.21.15.84:3300/0 8 ==== mgrmap(e 11) v1 ==== 36054+0+0 (secure 0 0 0) 0x7fb1d802c8b0 con 0x7fb1dc0703f0 2024-02-20T00:09:42.838 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:42.835+0000 7fb1ca7fc700 1 --2- 172.21.15.84:0/386744130 >> [v2:172.21.15.84:6800/1962643952,v1:172.21.15.84:6801/1962643952] conn(0x7fb1c4033d70 0x7fb1c4036160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:09:42.838 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:42.835+0000 7fb1ca7fc700 1 -- 172.21.15.84:0/386744130 --> [v2:172.21.15.84:6800/1962643952,v1:172.21.15.84:6801/1962643952] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7fb1bc000bc0 con 0x7fb1c4033d70 2024-02-20T00:09:42.838 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:42.835+0000 7fb1e1558700 1 --2- 172.21.15.84:0/386744130 >> [v2:172.21.15.84:6800/1962643952,v1:172.21.15.84:6801/1962643952] conn(0x7fb1c4033d70 0x7fb1c4036160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:09:42.838 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:42.836+0000 7fb1e1558700 1 --2- 172.21.15.84:0/386744130 >> [v2:172.21.15.84:6800/1962643952,v1:172.21.15.84:6801/1962643952] conn(0x7fb1c4033d70 0x7fb1c4036160 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7fb1cc0036d0 tx=0x7fb1cc006d10).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:09:42.841 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:42.840+0000 7fb1ca7fc700 1 -- 172.21.15.84:0/386744130 <== mgr.14142 v2:172.21.15.84:6800/1962643952 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+31 (secure 0 0 0) 0x7fb1bc000bc0 con 0x7fb1c4033d70 2024-02-20T00:09:42.841 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: Scheduled prometheus update... 2024-02-20T00:09:42.845 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:42.842+0000 7fb1e3fbd700 1 -- 172.21.15.84:0/386744130 >> [v2:172.21.15.84:6800/1962643952,v1:172.21.15.84:6801/1962643952] conn(0x7fb1c4033d70 msgr2=0x7fb1c4036160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:09:42.845 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:42.842+0000 7fb1e3fbd700 1 --2- 172.21.15.84:0/386744130 >> [v2:172.21.15.84:6800/1962643952,v1:172.21.15.84:6801/1962643952] conn(0x7fb1c4033d70 0x7fb1c4036160 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7fb1cc0036d0 tx=0x7fb1cc006d10).stop 2024-02-20T00:09:42.845 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:42.842+0000 7fb1e3fbd700 1 -- 172.21.15.84:0/386744130 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb1dc0703f0 msgr2=0x7fb1dc119990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:09:42.845 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:42.842+0000 7fb1e3fbd700 1 --2- 172.21.15.84:0/386744130 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb1dc0703f0 0x7fb1dc119990 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7fb1d800df90 tx=0x7fb1d8006ee0).stop 2024-02-20T00:09:42.845 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:42.842+0000 7fb1e3fbd700 1 -- 172.21.15.84:0/386744130 shutdown_connections 2024-02-20T00:09:42.845 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:42.842+0000 7fb1e3fbd700 1 --2- 172.21.15.84:0/386744130 >> [v2:172.21.15.84:6800/1962643952,v1:172.21.15.84:6801/1962643952] conn(0x7fb1c4033d70 0x7fb1c4036160 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:09:42.845 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:42.842+0000 7fb1e3fbd700 1 --2- 172.21.15.84:0/386744130 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb1dc0703f0 0x7fb1dc119990 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:09:42.845 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:42.842+0000 7fb1e3fbd700 1 --2- 172.21.15.84:0/386744130 >> [v2:172.21.15.84:6800/1412904880,v1:172.21.15.84:6801/1412904880] conn(0x7fb1c402ff50 0x7fb1c4032410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-20T00:09:42.845 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:42.842+0000 7fb1e3fbd700 1 -- 172.21.15.84:0/386744130 >> 172.21.15.84:0/386744130 conn(0x7fb1dc0f6550 msgr2=0x7fb1dc0fce80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:09:42.845 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:42.842+0000 7fb1e3fbd700 1 -- 172.21.15.84:0/386744130 shutdown_connections 2024-02-20T00:09:42.845 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:42.842+0000 7fb1e3fbd700 1 -- 172.21.15.84:0/386744130 wait complete. 2024-02-20T00:09:42.857 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:42 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:09:41.464246+0000 mon.smithi084 (mon.0) 62 : cluster [INF] Active manager daemon smithi084.bmqrnf restarted 2024-02-20T00:09:42.858 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:42 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:09:41.464754+0000 mon.smithi084 (mon.0) 63 : cluster [INF] Activating manager daemon smithi084.bmqrnf 2024-02-20T00:09:42.858 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:42 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:09:41.466401+0000 mon.smithi084 (mon.0) 64 : cluster [DBG] osdmap e3: 0 total, 0 up, 0 in 2024-02-20T00:09:42.858 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:42 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:09:41.832529+0000 mon.smithi084 (mon.0) 65 : cluster [DBG] mgrmap e10: smithi084.bmqrnf(active, starting, since 0.3679s) 2024-02-20T00:09:42.858 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:42 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:09:41.833464+0000 mon.smithi084 (mon.0) 66 : audit [DBG] from='mgr.14142 172.21.15.84:0/3829884648' entity='mgr.smithi084.bmqrnf' cmd=[{"prefix": "mon metadata", "id": "smithi084"}]: dispatch 2024-02-20T00:09:42.858 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:42 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:09:41.834058+0000 mon.smithi084 (mon.0) 67 : audit [DBG] from='mgr.14142 172.21.15.84:0/3829884648' entity='mgr.smithi084.bmqrnf' cmd=[{"prefix": "mgr metadata", "who": "smithi084.bmqrnf", "id": "smithi084.bmqrnf"}]: dispatch 2024-02-20T00:09:42.858 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:42 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:09:41.834938+0000 mon.smithi084 (mon.0) 68 : audit [DBG] from='mgr.14142 172.21.15.84:0/3829884648' entity='mgr.smithi084.bmqrnf' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-20T00:09:42.858 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:42 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:09:41.835081+0000 mon.smithi084 (mon.0) 69 : audit [DBG] from='mgr.14142 172.21.15.84:0/3829884648' entity='mgr.smithi084.bmqrnf' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-20T00:09:42.858 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:42 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:09:41.835189+0000 mon.smithi084 (mon.0) 70 : audit [DBG] from='mgr.14142 172.21.15.84:0/3829884648' entity='mgr.smithi084.bmqrnf' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-20T00:09:42.858 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:42 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:09:41.836993+0000 mon.smithi084 (mon.0) 71 : cluster [INF] Manager daemon smithi084.bmqrnf is now available 2024-02-20T00:09:42.858 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:42 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:09:41.934814+0000 mon.smithi084 (mon.0) 72 : audit [DBG] from='mgr.14142 172.21.15.84:0/3829884648' entity='mgr.smithi084.bmqrnf' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-20T00:09:42.858 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:42 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:09:41.936005+0000 mon.smithi084 (mon.0) 73 : audit [DBG] from='mgr.14142 172.21.15.84:0/3829884648' entity='mgr.smithi084.bmqrnf' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-20T00:09:42.858 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:42 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:09:41.951854+0000 mon.smithi084 (mon.0) 74 : audit [INF] from='mgr.14142 172.21.15.84:0/3829884648' entity='mgr.smithi084.bmqrnf' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi084.bmqrnf/mirror_snapshot_schedule"}]: dispatch 2024-02-20T00:09:42.858 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:42 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:09:41.959707+0000 mon.smithi084 (mon.0) 75 : audit [INF] from='mgr.14142 172.21.15.84:0/3829884648' entity='mgr.smithi084.bmqrnf' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi084.bmqrnf/trash_purge_schedule"}]: dispatch 2024-02-20T00:09:43.416 INFO:teuthology.orchestra.run.smithi084.stderr:Deploying grafana service with default placement... 2024-02-20T00:09:43.416 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi084 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/2a2e6c42-cf84-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp3ot6j0xn:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp4xryone4:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply grafana 2024-02-20T00:09:43.956 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:43.951+0000 7f60e37d8700 1 Processor -- start 2024-02-20T00:09:43.956 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:43.951+0000 7f60e37d8700 1 -- start start 2024-02-20T00:09:43.956 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:43.952+0000 7f60e37d8700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f60dc0704a0 0x7f60dc0feb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-20T00:09:43.956 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:43.952+0000 7f60e37d8700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60dc0feff0 con 0x7f60dc0704a0 2024-02-20T00:09:43.957 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:43.952+0000 7f60e1574700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f60dc0704a0 0x7f60dc0feb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:09:43.957 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:43.952+0000 7f60e1574700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f60dc0704a0 0x7f60dc0feb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:53436/0 (socket says 172.21.15.84:53436) 2024-02-20T00:09:43.957 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:43.952+0000 7f60e1574700 1 -- 172.21.15.84:0/3421882926 learned_addr learned my addr 172.21.15.84:0/3421882926 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-02-20T00:09:43.957 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:43.952+0000 7f60e1574700 1 -- 172.21.15.84:0/3421882926 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60dc0ff130 con 0x7f60dc0704a0 2024-02-20T00:09:43.957 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:43.952+0000 7f60e1574700 1 --2- 172.21.15.84:0/3421882926 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f60dc0704a0 0x7f60dc0feb00 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f60cc00b5f0 tx=0x7f60cc006ee0).ready entity=mon.0 client_cookie=15c328cea84bcf2a server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:09:43.957 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:43.952+0000 7f60cbfff700 1 -- 172.21.15.84:0/3421882926 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f60cc00d600 con 0x7f60dc0704a0 2024-02-20T00:09:43.957 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:43.952+0000 7f60cbfff700 1 -- 172.21.15.84:0/3421882926 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f60cc00dbf0 con 0x7f60dc0704a0 2024-02-20T00:09:43.957 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:43.952+0000 7f60cbfff700 1 -- 172.21.15.84:0/3421882926 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f60cc011cb0 con 0x7f60dc0704a0 2024-02-20T00:09:43.957 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:43.953+0000 7f60e37d8700 1 -- 172.21.15.84:0/3421882926 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f60dc0704a0 msgr2=0x7f60dc0feb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:09:43.957 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:43.953+0000 7f60e37d8700 1 --2- 172.21.15.84:0/3421882926 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f60dc0704a0 0x7f60dc0feb00 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f60cc00b5f0 tx=0x7f60cc006ee0).stop 2024-02-20T00:09:43.957 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:43.953+0000 7f60e37d8700 1 -- 172.21.15.84:0/3421882926 shutdown_connections 2024-02-20T00:09:43.957 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:43.953+0000 7f60e37d8700 1 --2- 172.21.15.84:0/3421882926 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f60dc0704a0 0x7f60dc0feb00 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:09:43.957 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:43.953+0000 7f60e37d8700 1 -- 172.21.15.84:0/3421882926 >> 172.21.15.84:0/3421882926 conn(0x7f60dc0f65c0 msgr2=0x7f60dc0f89e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:09:43.957 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:43.953+0000 7f60e37d8700 1 -- 172.21.15.84:0/3421882926 shutdown_connections 2024-02-20T00:09:43.957 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:43.953+0000 7f60e37d8700 1 -- 172.21.15.84:0/3421882926 wait complete. 2024-02-20T00:09:43.958 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:43.953+0000 7f60e37d8700 1 Processor -- start 2024-02-20T00:09:43.958 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:43.954+0000 7f60e37d8700 1 -- start start 2024-02-20T00:09:43.958 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:43.954+0000 7f60e37d8700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f60dc0704a0 0x7f60dc10ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:09:43.958 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:43.954+0000 7f60e37d8700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60dc10b060 con 0x7f60dc0704a0 2024-02-20T00:09:43.958 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:43.954+0000 7f60e1574700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f60dc0704a0 0x7f60dc10ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:09:43.958 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:43.954+0000 7f60e1574700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f60dc0704a0 0x7f60dc10ab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:53440/0 (socket says 172.21.15.84:53440) 2024-02-20T00:09:43.958 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:43.954+0000 7f60e1574700 1 -- 172.21.15.84:0/553749749 learned_addr learned my addr 172.21.15.84:0/553749749 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-02-20T00:09:43.958 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:43.954+0000 7f60e1574700 1 -- 172.21.15.84:0/553749749 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60cc00b200 con 0x7f60dc0704a0 2024-02-20T00:09:43.958 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:43.954+0000 7f60e1574700 1 --2- 172.21.15.84:0/553749749 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f60dc0704a0 0x7f60dc10ab70 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f60cc006800 tx=0x7f60cc006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:09:43.958 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:43.954+0000 7f60c9ffb700 1 -- 172.21.15.84:0/553749749 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f60cc01b570 con 0x7f60dc0704a0 2024-02-20T00:09:43.958 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:43.954+0000 7f60c9ffb700 1 -- 172.21.15.84:0/553749749 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f60cc01bb60 con 0x7f60dc0704a0 2024-02-20T00:09:43.958 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:43.954+0000 7f60e37d8700 1 -- 172.21.15.84:0/553749749 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60dc10b260 con 0x7f60dc0704a0 2024-02-20T00:09:43.958 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:43.954+0000 7f60c9ffb700 1 -- 172.21.15.84:0/553749749 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f60cc01a9f0 con 0x7f60dc0704a0 2024-02-20T00:09:43.959 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:43.955+0000 7f60e37d8700 1 -- 172.21.15.84:0/553749749 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60dc10b6f0 con 0x7f60dc0704a0 2024-02-20T00:09:43.959 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:43.955+0000 7f60c9ffb700 1 -- 172.21.15.84:0/553749749 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f60cc00f070 con 0x7f60dc0704a0 2024-02-20T00:09:43.959 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:43.955+0000 7f60e37d8700 1 -- 172.21.15.84:0/553749749 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60dc045ad0 con 0x7f60dc0704a0 2024-02-20T00:09:43.959 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:43.956+0000 7f60c9ffb700 1 --2- 172.21.15.84:0/553749749 >> [v2:172.21.15.84:6800/1962643952,v1:172.21.15.84:6801/1962643952] conn(0x7f60c402ff70 0x7f60c4032430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:09:43.959 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:43.956+0000 7f60c9ffb700 1 -- 172.21.15.84:0/553749749 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f60cc041570 con 0x7f60dc0704a0 2024-02-20T00:09:43.959 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:43.956+0000 7f60e0d73700 1 --2- 172.21.15.84:0/553749749 >> [v2:172.21.15.84:6800/1962643952,v1:172.21.15.84:6801/1962643952] conn(0x7f60c402ff70 0x7f60c4032430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:09:43.959 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:43.956+0000 7f60e0d73700 1 --2- 172.21.15.84:0/553749749 >> [v2:172.21.15.84:6800/1962643952,v1:172.21.15.84:6801/1962643952] conn(0x7f60c402ff70 0x7f60c4032430 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f60d80096a0 tx=0x7f60d8006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:09:43.961 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:43.960+0000 7f60c9ffb700 1 -- 172.21.15.84:0/553749749 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f60cc022310 con 0x7f60dc0704a0 2024-02-20T00:09:44.107 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:43 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:09:42.836865+0000 mon.smithi084 (mon.0) 76 : cluster [DBG] mgrmap e11: smithi084.bmqrnf(active, since 1.37223s) 2024-02-20T00:09:44.107 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:43 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cephadm 2024-02-20T00:09:42.838336+0000 mgr.smithi084.bmqrnf (mgr.14142) 2 : cephadm [INF] Saving service prometheus spec with placement count:1 2024-02-20T00:09:44.107 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:43 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:09:42.840489+0000 mon.smithi084 (mon.0) 77 : audit [INF] from='mgr.14142 172.21.15.84:0/3829884648' entity='mgr.smithi084.bmqrnf' 2024-02-20T00:09:44.140 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:44.138+0000 7f60e37d8700 1 -- 172.21.15.84:0/553749749 --> [v2:172.21.15.84:6800/1962643952,v1:172.21.15.84:6801/1962643952] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}) v1 -- 0x7f60dc10c010 con 0x7f60c402ff70 2024-02-20T00:09:44.146 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:44.145+0000 7f60c9ffb700 1 -- 172.21.15.84:0/553749749 <== mgr.14142 v2:172.21.15.84:6800/1962643952 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7f60dc10c010 con 0x7f60c402ff70 2024-02-20T00:09:44.146 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: Scheduled grafana update... 2024-02-20T00:09:44.149 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:44.147+0000 7f60e37d8700 1 -- 172.21.15.84:0/553749749 >> [v2:172.21.15.84:6800/1962643952,v1:172.21.15.84:6801/1962643952] conn(0x7f60c402ff70 msgr2=0x7f60c4032430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:09:44.149 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:44.147+0000 7f60e37d8700 1 --2- 172.21.15.84:0/553749749 >> [v2:172.21.15.84:6800/1962643952,v1:172.21.15.84:6801/1962643952] conn(0x7f60c402ff70 0x7f60c4032430 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f60d80096a0 tx=0x7f60d8006b40).stop 2024-02-20T00:09:44.149 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:44.147+0000 7f60e37d8700 1 -- 172.21.15.84:0/553749749 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f60dc0704a0 msgr2=0x7f60dc10ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:09:44.149 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:44.147+0000 7f60e37d8700 1 --2- 172.21.15.84:0/553749749 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f60dc0704a0 0x7f60dc10ab70 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f60cc006800 tx=0x7f60cc006ee0).stop 2024-02-20T00:09:44.149 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:44.147+0000 7f60e37d8700 1 -- 172.21.15.84:0/553749749 shutdown_connections 2024-02-20T00:09:44.149 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:44.147+0000 7f60e37d8700 1 --2- 172.21.15.84:0/553749749 >> [v2:172.21.15.84:6800/1962643952,v1:172.21.15.84:6801/1962643952] conn(0x7f60c402ff70 0x7f60c4032430 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:09:44.149 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:44.147+0000 7f60e37d8700 1 --2- 172.21.15.84:0/553749749 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f60dc0704a0 0x7f60dc10ab70 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:09:44.149 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:44.147+0000 7f60e37d8700 1 -- 172.21.15.84:0/553749749 >> 172.21.15.84:0/553749749 conn(0x7f60dc0f65c0 msgr2=0x7f60dc0f7f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:09:44.149 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:44.147+0000 7f60e37d8700 1 -- 172.21.15.84:0/553749749 shutdown_connections 2024-02-20T00:09:44.150 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:44.147+0000 7f60e37d8700 1 -- 172.21.15.84:0/553749749 wait complete. 2024-02-20T00:09:44.697 INFO:teuthology.orchestra.run.smithi084.stderr:Deploying node-exporter service with default placement... 2024-02-20T00:09:44.698 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi084 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/2a2e6c42-cf84-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp3ot6j0xn:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp4xryone4:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply node-exporter 2024-02-20T00:09:44.951 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:44 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:09:43.842170+0000 mon.smithi084 (mon.0) 78 : cluster [DBG] mgrmap e12: smithi084.bmqrnf(active, since 2s) 2024-02-20T00:09:44.951 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:44 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:09:44.139862+0000 mgr.smithi084.bmqrnf (mgr.14142) 3 : audit [DBG] from='client.14150 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T00:09:44.952 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:44 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cephadm 2024-02-20T00:09:44.140358+0000 mgr.smithi084.bmqrnf (mgr.14142) 4 : cephadm [INF] Saving service grafana spec with placement count:1 2024-02-20T00:09:44.952 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:44 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:09:44.145431+0000 mon.smithi084 (mon.0) 79 : audit [INF] from='mgr.14142 172.21.15.84:0/3829884648' entity='mgr.smithi084.bmqrnf' 2024-02-20T00:09:45.216 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:45.211+0000 7f6a0759e700 1 Processor -- start 2024-02-20T00:09:45.216 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:45.212+0000 7f6a0759e700 1 -- start start 2024-02-20T00:09:45.216 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:45.212+0000 7f6a0759e700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6a08070430 0x7f6a080fe890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-20T00:09:45.216 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:45.212+0000 7f6a0759e700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a080fed80 con 0x7f6a08070430 2024-02-20T00:09:45.216 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:45.212+0000 7f6a0659c700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6a08070430 0x7f6a080fe890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:09:45.216 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:45.212+0000 7f6a0659c700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6a08070430 0x7f6a080fe890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:53456/0 (socket says 172.21.15.84:53456) 2024-02-20T00:09:45.216 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:45.212+0000 7f6a0659c700 1 -- 172.21.15.84:0/3404791491 learned_addr learned my addr 172.21.15.84:0/3404791491 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-02-20T00:09:45.216 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:45.212+0000 7f6a0659c700 1 -- 172.21.15.84:0/3404791491 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a080feec0 con 0x7f6a08070430 2024-02-20T00:09:45.217 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:45.212+0000 7f6a0659c700 1 --2- 172.21.15.84:0/3404791491 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6a08070430 0x7f6a080fe890 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f69fc00b5f0 tx=0x7f69fc006ee0).ready entity=mon.0 client_cookie=d569750c276464dd server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:09:45.217 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:45.213+0000 7f6a0559a700 1 -- 172.21.15.84:0/3404791491 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f69fc00d600 con 0x7f6a08070430 2024-02-20T00:09:45.217 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:45.213+0000 7f6a0559a700 1 -- 172.21.15.84:0/3404791491 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f69fc00dbf0 con 0x7f6a08070430 2024-02-20T00:09:45.217 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:45.213+0000 7f6a0559a700 1 -- 172.21.15.84:0/3404791491 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f69fc011cb0 con 0x7f6a08070430 2024-02-20T00:09:45.217 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:45.213+0000 7f6a0759e700 1 -- 172.21.15.84:0/3404791491 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6a08070430 msgr2=0x7f6a080fe890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:09:45.217 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:45.213+0000 7f6a0759e700 1 --2- 172.21.15.84:0/3404791491 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6a08070430 0x7f6a080fe890 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f69fc00b5f0 tx=0x7f69fc006ee0).stop 2024-02-20T00:09:45.217 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:45.213+0000 7f6a0759e700 1 -- 172.21.15.84:0/3404791491 shutdown_connections 2024-02-20T00:09:45.217 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:45.213+0000 7f6a0759e700 1 --2- 172.21.15.84:0/3404791491 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6a08070430 0x7f6a080fe890 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:09:45.217 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:45.213+0000 7f6a0759e700 1 -- 172.21.15.84:0/3404791491 >> 172.21.15.84:0/3404791491 conn(0x7f6a080f6390 msgr2=0x7f6a080f87b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:09:45.217 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:45.213+0000 7f6a0759e700 1 -- 172.21.15.84:0/3404791491 shutdown_connections 2024-02-20T00:09:45.217 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:45.213+0000 7f6a0759e700 1 -- 172.21.15.84:0/3404791491 wait complete. 2024-02-20T00:09:45.217 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:45.214+0000 7f6a0759e700 1 Processor -- start 2024-02-20T00:09:45.217 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:45.214+0000 7f6a0759e700 1 -- start start 2024-02-20T00:09:45.217 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:45.214+0000 7f6a0759e700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6a08070430 0x7f6a080fc310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:09:45.217 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:45.214+0000 7f6a0759e700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a080fa790 con 0x7f6a08070430 2024-02-20T00:09:45.218 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:45.214+0000 7f6a0659c700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6a08070430 0x7f6a080fc310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:09:45.218 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:45.214+0000 7f6a0659c700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6a08070430 0x7f6a080fc310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:53468/0 (socket says 172.21.15.84:53468) 2024-02-20T00:09:45.218 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:45.214+0000 7f6a0659c700 1 -- 172.21.15.84:0/3165986653 learned_addr learned my addr 172.21.15.84:0/3165986653 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-02-20T00:09:45.218 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:45.214+0000 7f6a0659c700 1 -- 172.21.15.84:0/3165986653 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69fc00b200 con 0x7f6a08070430 2024-02-20T00:09:45.219 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:45.215+0000 7f6a0659c700 1 --2- 172.21.15.84:0/3165986653 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6a08070430 0x7f6a080fc310 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f69fc006800 tx=0x7f69fc006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:09:45.219 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:45.215+0000 7f69f6ffd700 1 -- 172.21.15.84:0/3165986653 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f69fc01b570 con 0x7f6a08070430 2024-02-20T00:09:45.219 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:45.215+0000 7f69f6ffd700 1 -- 172.21.15.84:0/3165986653 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f69fc01bb60 con 0x7f6a08070430 2024-02-20T00:09:45.219 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:45.215+0000 7f6a0759e700 1 -- 172.21.15.84:0/3165986653 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a080fa930 con 0x7f6a08070430 2024-02-20T00:09:45.219 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:45.215+0000 7f69f6ffd700 1 -- 172.21.15.84:0/3165986653 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f69fc01a9f0 con 0x7f6a08070430 2024-02-20T00:09:45.219 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:45.215+0000 7f6a0759e700 1 -- 172.21.15.84:0/3165986653 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a080fadc0 con 0x7f6a08070430 2024-02-20T00:09:45.219 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:45.216+0000 7f69f6ffd700 1 -- 172.21.15.84:0/3165986653 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f69fc00f070 con 0x7f6a08070430 2024-02-20T00:09:45.219 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:45.216+0000 7f6a0759e700 1 -- 172.21.15.84:0/3165986653 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a08045ad0 con 0x7f6a08070430 2024-02-20T00:09:45.219 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:45.216+0000 7f69f6ffd700 1 --2- 172.21.15.84:0/3165986653 >> [v2:172.21.15.84:6800/1962643952,v1:172.21.15.84:6801/1962643952] conn(0x7f69ec02ff70 0x7f69ec032430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:09:45.219 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:45.216+0000 7f69f6ffd700 1 -- 172.21.15.84:0/3165986653 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f69fc0414d0 con 0x7f6a08070430 2024-02-20T00:09:45.220 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:45.216+0000 7f6a05d9b700 1 --2- 172.21.15.84:0/3165986653 >> [v2:172.21.15.84:6800/1962643952,v1:172.21.15.84:6801/1962643952] conn(0x7f69ec02ff70 0x7f69ec032430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:09:45.220 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:45.216+0000 7f6a05d9b700 1 --2- 172.21.15.84:0/3165986653 >> [v2:172.21.15.84:6800/1962643952,v1:172.21.15.84:6801/1962643952] conn(0x7f69ec02ff70 0x7f69ec032430 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f69f80096a0 tx=0x7f69f8006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:09:45.221 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:45.220+0000 7f69f6ffd700 1 -- 172.21.15.84:0/3165986653 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f69fc019c90 con 0x7f6a08070430 2024-02-20T00:09:45.396 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:45.395+0000 7f6a0759e700 1 -- 172.21.15.84:0/3165986653 --> [v2:172.21.15.84:6800/1962643952,v1:172.21.15.84:6801/1962643952] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}) v1 -- 0x7f6a080fb000 con 0x7f69ec02ff70 2024-02-20T00:09:45.400 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:45.399+0000 7f69f6ffd700 1 -- 172.21.15.84:0/3165986653 <== mgr.14142 v2:172.21.15.84:6800/1962643952 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7f6a080fb000 con 0x7f69ec02ff70 2024-02-20T00:09:45.400 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: Scheduled node-exporter update... 2024-02-20T00:09:45.402 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:45.400+0000 7f6a0759e700 1 -- 172.21.15.84:0/3165986653 >> [v2:172.21.15.84:6800/1962643952,v1:172.21.15.84:6801/1962643952] conn(0x7f69ec02ff70 msgr2=0x7f69ec032430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:09:45.403 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:45.400+0000 7f6a0759e700 1 --2- 172.21.15.84:0/3165986653 >> [v2:172.21.15.84:6800/1962643952,v1:172.21.15.84:6801/1962643952] conn(0x7f69ec02ff70 0x7f69ec032430 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f69f80096a0 tx=0x7f69f8006b40).stop 2024-02-20T00:09:45.403 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:45.400+0000 7f6a0759e700 1 -- 172.21.15.84:0/3165986653 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6a08070430 msgr2=0x7f6a080fc310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:09:45.403 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:45.400+0000 7f6a0759e700 1 --2- 172.21.15.84:0/3165986653 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6a08070430 0x7f6a080fc310 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f69fc006800 tx=0x7f69fc006ee0).stop 2024-02-20T00:09:45.403 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:45.400+0000 7f6a0759e700 1 -- 172.21.15.84:0/3165986653 shutdown_connections 2024-02-20T00:09:45.403 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:45.400+0000 7f6a0759e700 1 --2- 172.21.15.84:0/3165986653 >> [v2:172.21.15.84:6800/1962643952,v1:172.21.15.84:6801/1962643952] conn(0x7f69ec02ff70 0x7f69ec032430 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:09:45.403 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:45.400+0000 7f6a0759e700 1 --2- 172.21.15.84:0/3165986653 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6a08070430 0x7f6a080fc310 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:09:45.403 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:45.400+0000 7f6a0759e700 1 -- 172.21.15.84:0/3165986653 >> 172.21.15.84:0/3165986653 conn(0x7f6a080f6390 msgr2=0x7f6a080f7d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:09:45.403 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:45.401+0000 7f6a0759e700 1 -- 172.21.15.84:0/3165986653 shutdown_connections 2024-02-20T00:09:45.403 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:45.401+0000 7f6a0759e700 1 -- 172.21.15.84:0/3165986653 wait complete. 2024-02-20T00:09:45.970 INFO:teuthology.orchestra.run.smithi084.stderr:Deploying alertmanager service with default placement... 2024-02-20T00:09:45.970 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi084 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/2a2e6c42-cf84-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp3ot6j0xn:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp4xryone4:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply alertmanager 2024-02-20T00:09:46.450 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:46.448+0000 7f29c9f96700 1 Processor -- start 2024-02-20T00:09:46.451 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:46.448+0000 7f29c9f96700 1 -- start start 2024-02-20T00:09:46.451 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:46.448+0000 7f29c9f96700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f29c40fad90 0x7f29c40fb1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-20T00:09:46.451 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:46.448+0000 7f29c9f96700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29c40fb6a0 con 0x7f29c40fad90 2024-02-20T00:09:46.451 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:46.448+0000 7f29c37fe700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f29c40fad90 0x7f29c40fb1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:09:46.451 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:46.448+0000 7f29c37fe700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f29c40fad90 0x7f29c40fb1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:53476/0 (socket says 172.21.15.84:53476) 2024-02-20T00:09:46.451 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:46.448+0000 7f29c37fe700 1 -- 172.21.15.84:0/600375901 learned_addr learned my addr 172.21.15.84:0/600375901 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-02-20T00:09:46.451 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:46.448+0000 7f29c37fe700 1 -- 172.21.15.84:0/600375901 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29c40fb7e0 con 0x7f29c40fad90 2024-02-20T00:09:46.451 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:46.448+0000 7f29c37fe700 1 --2- 172.21.15.84:0/600375901 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f29c40fad90 0x7f29c40fb1b0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f29b400b5f0 tx=0x7f29b4006ee0).ready entity=mon.0 client_cookie=7bf56ff8ed40905f server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:09:46.451 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:46.449+0000 7f29c27fc700 1 -- 172.21.15.84:0/600375901 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f29b400d600 con 0x7f29c40fad90 2024-02-20T00:09:46.453 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:46.449+0000 7f29c27fc700 1 -- 172.21.15.84:0/600375901 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f29b400dbf0 con 0x7f29c40fad90 2024-02-20T00:09:46.453 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:46.449+0000 7f29c27fc700 1 -- 172.21.15.84:0/600375901 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f29b4011cb0 con 0x7f29c40fad90 2024-02-20T00:09:46.453 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:46.449+0000 7f29c9f96700 1 -- 172.21.15.84:0/600375901 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f29c40fad90 msgr2=0x7f29c40fb1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:09:46.453 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:46.449+0000 7f29c9f96700 1 --2- 172.21.15.84:0/600375901 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f29c40fad90 0x7f29c40fb1b0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f29b400b5f0 tx=0x7f29b4006ee0).stop 2024-02-20T00:09:46.453 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:46.449+0000 7f29c9f96700 1 -- 172.21.15.84:0/600375901 shutdown_connections 2024-02-20T00:09:46.453 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:46.449+0000 7f29c9f96700 1 --2- 172.21.15.84:0/600375901 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f29c40fad90 0x7f29c40fb1b0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:09:46.453 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:46.449+0000 7f29c9f96700 1 -- 172.21.15.84:0/600375901 >> 172.21.15.84:0/600375901 conn(0x7f29c40f6530 msgr2=0x7f29c40f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:09:46.453 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:46.450+0000 7f29c9f96700 1 -- 172.21.15.84:0/600375901 shutdown_connections 2024-02-20T00:09:46.453 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:46.450+0000 7f29c9f96700 1 -- 172.21.15.84:0/600375901 wait complete. 2024-02-20T00:09:46.453 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:46.450+0000 7f29c9f96700 1 Processor -- start 2024-02-20T00:09:46.454 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:46.450+0000 7f29c9f96700 1 -- start start 2024-02-20T00:09:46.454 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:46.451+0000 7f29c9f96700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f29c40fad90 0x7f29c410ce10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:09:46.454 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:46.451+0000 7f29c9f96700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29c410d300 con 0x7f29c40fad90 2024-02-20T00:09:46.454 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:46.451+0000 7f29c37fe700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f29c40fad90 0x7f29c410ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:09:46.454 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:46.451+0000 7f29c37fe700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f29c40fad90 0x7f29c410ce10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:53480/0 (socket says 172.21.15.84:53480) 2024-02-20T00:09:46.454 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:46.451+0000 7f29c37fe700 1 -- 172.21.15.84:0/2984372774 learned_addr learned my addr 172.21.15.84:0/2984372774 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-02-20T00:09:46.454 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:46.451+0000 7f29c37fe700 1 -- 172.21.15.84:0/2984372774 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29b400b200 con 0x7f29c40fad90 2024-02-20T00:09:46.454 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:46.451+0000 7f29c37fe700 1 --2- 172.21.15.84:0/2984372774 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f29c40fad90 0x7f29c410ce10 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f29b400bf40 tx=0x7f29b4006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:09:46.454 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:46.451+0000 7f29abfff700 1 -- 172.21.15.84:0/2984372774 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f29b401b570 con 0x7f29c40fad90 2024-02-20T00:09:46.454 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:46.451+0000 7f29abfff700 1 -- 172.21.15.84:0/2984372774 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f29b401bb60 con 0x7f29c40fad90 2024-02-20T00:09:46.454 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:46.451+0000 7f29c9f96700 1 -- 172.21.15.84:0/2984372774 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29c410d500 con 0x7f29c40fad90 2024-02-20T00:09:46.454 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:46.451+0000 7f29abfff700 1 -- 172.21.15.84:0/2984372774 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f29b401aa20 con 0x7f29c40fad90 2024-02-20T00:09:46.455 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:46.451+0000 7f29c9f96700 1 -- 172.21.15.84:0/2984372774 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29c410e110 con 0x7f29c40fad90 2024-02-20T00:09:46.455 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:46.452+0000 7f29abfff700 1 -- 172.21.15.84:0/2984372774 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f29b400f070 con 0x7f29c40fad90 2024-02-20T00:09:46.455 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:46.452+0000 7f29c9f96700 1 -- 172.21.15.84:0/2984372774 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29c4059100 con 0x7f29c40fad90 2024-02-20T00:09:46.455 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:46.452+0000 7f29abfff700 1 --2- 172.21.15.84:0/2984372774 >> [v2:172.21.15.84:6800/1962643952,v1:172.21.15.84:6801/1962643952] conn(0x7f29b002ff70 0x7f29b0032430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:09:46.456 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:46.452+0000 7f29abfff700 1 -- 172.21.15.84:0/2984372774 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f29b40415b0 con 0x7f29c40fad90 2024-02-20T00:09:46.456 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:46.453+0000 7f29c2ffd700 1 --2- 172.21.15.84:0/2984372774 >> [v2:172.21.15.84:6800/1962643952,v1:172.21.15.84:6801/1962643952] conn(0x7f29b002ff70 0x7f29b0032430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:09:46.456 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:46.453+0000 7f29c2ffd700 1 --2- 172.21.15.84:0/2984372774 >> [v2:172.21.15.84:6800/1962643952,v1:172.21.15.84:6801/1962643952] conn(0x7f29b002ff70 0x7f29b0032430 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f29b80096a0 tx=0x7f29b8006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:09:46.458 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:46.457+0000 7f29abfff700 1 -- 172.21.15.84:0/2984372774 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f29b4019af0 con 0x7f29c40fad90 2024-02-20T00:09:46.607 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:46 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:09:45.396559+0000 mgr.smithi084.bmqrnf (mgr.14142) 5 : audit [DBG] from='client.14152 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T00:09:46.607 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:46 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cephadm 2024-02-20T00:09:45.397176+0000 mgr.smithi084.bmqrnf (mgr.14142) 6 : cephadm [INF] Saving service node-exporter spec with placement * 2024-02-20T00:09:46.607 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:46 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:09:45.399258+0000 mon.smithi084 (mon.0) 80 : audit [INF] from='mgr.14142 172.21.15.84:0/3829884648' entity='mgr.smithi084.bmqrnf' 2024-02-20T00:09:46.634 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:46.633+0000 7f29c9f96700 1 -- 172.21.15.84:0/2984372774 --> [v2:172.21.15.84:6800/1962643952,v1:172.21.15.84:6801/1962643952] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}) v1 -- 0x7f29c410e350 con 0x7f29b002ff70 2024-02-20T00:09:46.638 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:46.636+0000 7f29abfff700 1 -- 172.21.15.84:0/2984372774 <== mgr.14142 v2:172.21.15.84:6800/1962643952 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+33 (secure 0 0 0) 0x7f29c410e350 con 0x7f29b002ff70 2024-02-20T00:09:46.638 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: Scheduled alertmanager update... 2024-02-20T00:09:46.640 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:46.638+0000 7f29c9f96700 1 -- 172.21.15.84:0/2984372774 >> [v2:172.21.15.84:6800/1962643952,v1:172.21.15.84:6801/1962643952] conn(0x7f29b002ff70 msgr2=0x7f29b0032430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:09:46.640 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:46.638+0000 7f29c9f96700 1 --2- 172.21.15.84:0/2984372774 >> [v2:172.21.15.84:6800/1962643952,v1:172.21.15.84:6801/1962643952] conn(0x7f29b002ff70 0x7f29b0032430 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f29b80096a0 tx=0x7f29b8006b40).stop 2024-02-20T00:09:46.640 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:46.638+0000 7f29c9f96700 1 -- 172.21.15.84:0/2984372774 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f29c40fad90 msgr2=0x7f29c410ce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:09:46.640 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:46.638+0000 7f29c9f96700 1 --2- 172.21.15.84:0/2984372774 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f29c40fad90 0x7f29c410ce10 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f29b400bf40 tx=0x7f29b4006ee0).stop 2024-02-20T00:09:46.641 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:46.638+0000 7f29c9f96700 1 -- 172.21.15.84:0/2984372774 shutdown_connections 2024-02-20T00:09:46.641 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:46.638+0000 7f29c9f96700 1 --2- 172.21.15.84:0/2984372774 >> [v2:172.21.15.84:6800/1962643952,v1:172.21.15.84:6801/1962643952] conn(0x7f29b002ff70 0x7f29b0032430 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:09:46.641 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:46.638+0000 7f29c9f96700 1 --2- 172.21.15.84:0/2984372774 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f29c40fad90 0x7f29c410ce10 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:09:46.641 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:46.638+0000 7f29c9f96700 1 -- 172.21.15.84:0/2984372774 >> 172.21.15.84:0/2984372774 conn(0x7f29c40f6530 msgr2=0x7f29c4070070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:09:46.641 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:46.638+0000 7f29c9f96700 1 -- 172.21.15.84:0/2984372774 shutdown_connections 2024-02-20T00:09:46.641 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:46.638+0000 7f29c9f96700 1 -- 172.21.15.84:0/2984372774 wait complete. 2024-02-20T00:09:47.251 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi084 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/2a2e6c42-cf84-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp3ot6j0xn:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp4xryone4:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mgr mgr/cephadm/container_init True --force 2024-02-20T00:09:47.767 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:47.761+0000 7f85a6739700 1 Processor -- start 2024-02-20T00:09:47.767 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:47.761+0000 7f85a6739700 1 -- start start 2024-02-20T00:09:47.767 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:47.761+0000 7f85a6739700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f85a00703b0 0x7f85a0100ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-20T00:09:47.767 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:47.761+0000 7f85a6739700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85a00fc8c0 con 0x7f85a00703b0 2024-02-20T00:09:47.767 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:47.761+0000 7f859ffff700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f85a00703b0 0x7f85a0100ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:09:47.767 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:47.761+0000 7f859ffff700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f85a00703b0 0x7f85a0100ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:53490/0 (socket says 172.21.15.84:53490) 2024-02-20T00:09:47.767 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:47.761+0000 7f859ffff700 1 -- 172.21.15.84:0/1655885471 learned_addr learned my addr 172.21.15.84:0/1655885471 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-02-20T00:09:47.767 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:47.762+0000 7f859ffff700 1 -- 172.21.15.84:0/1655885471 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85a01013c0 con 0x7f85a00703b0 2024-02-20T00:09:47.767 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:47.762+0000 7f859ffff700 1 --2- 172.21.15.84:0/1655885471 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f85a00703b0 0x7f85a0100ed0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f859400b5f0 tx=0x7f8594006ee0).ready entity=mon.0 client_cookie=cdd5ba765e35ed61 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:09:47.768 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:47.762+0000 7f859effd700 1 -- 172.21.15.84:0/1655885471 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f859400d600 con 0x7f85a00703b0 2024-02-20T00:09:47.768 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:47.762+0000 7f859effd700 1 -- 172.21.15.84:0/1655885471 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f859400dbf0 con 0x7f85a00703b0 2024-02-20T00:09:47.768 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:47.762+0000 7f859effd700 1 -- 172.21.15.84:0/1655885471 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8594011cb0 con 0x7f85a00703b0 2024-02-20T00:09:47.768 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:47.762+0000 7f85a6739700 1 -- 172.21.15.84:0/1655885471 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f85a00703b0 msgr2=0x7f85a0100ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:09:47.768 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:47.762+0000 7f85a6739700 1 --2- 172.21.15.84:0/1655885471 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f85a00703b0 0x7f85a0100ed0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f859400b5f0 tx=0x7f8594006ee0).stop 2024-02-20T00:09:47.768 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:47.762+0000 7f85a6739700 1 -- 172.21.15.84:0/1655885471 shutdown_connections 2024-02-20T00:09:47.768 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:47.762+0000 7f85a6739700 1 --2- 172.21.15.84:0/1655885471 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f85a00703b0 0x7f85a0100ed0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:09:47.768 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:47.762+0000 7f85a6739700 1 -- 172.21.15.84:0/1655885471 >> 172.21.15.84:0/1655885471 conn(0x7f85a00f6530 msgr2=0x7f85a00f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:09:47.768 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:47.762+0000 7f85a6739700 1 -- 172.21.15.84:0/1655885471 shutdown_connections 2024-02-20T00:09:47.768 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:47.763+0000 7f85a6739700 1 -- 172.21.15.84:0/1655885471 wait complete. 2024-02-20T00:09:47.768 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:47.763+0000 7f85a6739700 1 Processor -- start 2024-02-20T00:09:47.768 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:47.763+0000 7f85a6739700 1 -- start start 2024-02-20T00:09:47.768 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:47.763+0000 7f85a6739700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f85a00703b0 0x7f85a010cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:09:47.769 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:47.763+0000 7f85a6739700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85a010d290 con 0x7f85a00703b0 2024-02-20T00:09:47.769 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:47.763+0000 7f859ffff700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f85a00703b0 0x7f85a010cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:09:47.769 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:47.763+0000 7f859ffff700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f85a00703b0 0x7f85a010cda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:53504/0 (socket says 172.21.15.84:53504) 2024-02-20T00:09:47.769 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:47.764+0000 7f859ffff700 1 -- 172.21.15.84:0/691100407 learned_addr learned my addr 172.21.15.84:0/691100407 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-02-20T00:09:47.769 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:47.764+0000 7f859ffff700 1 -- 172.21.15.84:0/691100407 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f859400b200 con 0x7f85a00703b0 2024-02-20T00:09:47.769 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:47.764+0000 7f859ffff700 1 --2- 172.21.15.84:0/691100407 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f85a00703b0 0x7f85a010cda0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f8594006800 tx=0x7f8594006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:09:47.769 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:47.764+0000 7f859cff9700 1 -- 172.21.15.84:0/691100407 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f859401b570 con 0x7f85a00703b0 2024-02-20T00:09:47.769 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:47.764+0000 7f859cff9700 1 -- 172.21.15.84:0/691100407 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f859401bb60 con 0x7f85a00703b0 2024-02-20T00:09:47.769 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:47.764+0000 7f85a6739700 1 -- 172.21.15.84:0/691100407 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85a010d490 con 0x7f85a00703b0 2024-02-20T00:09:47.769 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:47.764+0000 7f859cff9700 1 -- 172.21.15.84:0/691100407 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f859401a9f0 con 0x7f85a00703b0 2024-02-20T00:09:47.769 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:47.764+0000 7f85a6739700 1 -- 172.21.15.84:0/691100407 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85a010e0d0 con 0x7f85a00703b0 2024-02-20T00:09:47.769 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:47.765+0000 7f859cff9700 1 -- 172.21.15.84:0/691100407 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f859400f070 con 0x7f85a00703b0 2024-02-20T00:09:47.769 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:47.765+0000 7f859cff9700 1 --2- 172.21.15.84:0/691100407 >> [v2:172.21.15.84:6800/1962643952,v1:172.21.15.84:6801/1962643952] conn(0x7f858c02ff70 0x7f858c032430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:09:47.770 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:47.765+0000 7f859cff9700 1 -- 172.21.15.84:0/691100407 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f85940416a0 con 0x7f85a00703b0 2024-02-20T00:09:47.770 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:47.765+0000 7f85a6739700 1 -- 172.21.15.84:0/691100407 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85a010d920 con 0x7f85a00703b0 2024-02-20T00:09:47.770 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:47.765+0000 7f859f7fe700 1 --2- 172.21.15.84:0/691100407 >> [v2:172.21.15.84:6800/1962643952,v1:172.21.15.84:6801/1962643952] conn(0x7f858c02ff70 0x7f858c032430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:09:47.770 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:47.765+0000 7f859f7fe700 1 --2- 172.21.15.84:0/691100407 >> [v2:172.21.15.84:6800/1962643952,v1:172.21.15.84:6801/1962643952] conn(0x7f858c02ff70 0x7f858c032430 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f85900096a0 tx=0x7f8590006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:09:47.771 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:47.769+0000 7f859cff9700 1 -- 172.21.15.84:0/691100407 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8594016070 con 0x7f85a00703b0 2024-02-20T00:09:47.919 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:47.917+0000 7f85a6739700 1 -- 172.21.15.84:0/691100407 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7f85a010ef00 con 0x7f85a00703b0 2024-02-20T00:09:47.921 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:47.920+0000 7f859cff9700 1 -- 172.21.15.84:0/691100407 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v8)=0 v8) v1 ==== 142+0+0 (secure 0 0 0) 0x7f8594025460 con 0x7f85a00703b0 2024-02-20T00:09:47.925 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:47.922+0000 7f85a6739700 1 -- 172.21.15.84:0/691100407 >> [v2:172.21.15.84:6800/1962643952,v1:172.21.15.84:6801/1962643952] conn(0x7f858c02ff70 msgr2=0x7f858c032430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:09:47.925 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:47.922+0000 7f85a6739700 1 --2- 172.21.15.84:0/691100407 >> [v2:172.21.15.84:6800/1962643952,v1:172.21.15.84:6801/1962643952] conn(0x7f858c02ff70 0x7f858c032430 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f85900096a0 tx=0x7f8590006b40).stop 2024-02-20T00:09:47.925 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:47.922+0000 7f85a6739700 1 -- 172.21.15.84:0/691100407 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f85a00703b0 msgr2=0x7f85a010cda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:09:47.925 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:47.922+0000 7f85a6739700 1 --2- 172.21.15.84:0/691100407 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f85a00703b0 0x7f85a010cda0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f8594006800 tx=0x7f8594006ee0).stop 2024-02-20T00:09:47.925 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:47.922+0000 7f85a6739700 1 -- 172.21.15.84:0/691100407 shutdown_connections 2024-02-20T00:09:47.925 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:47.922+0000 7f85a6739700 1 --2- 172.21.15.84:0/691100407 >> [v2:172.21.15.84:6800/1962643952,v1:172.21.15.84:6801/1962643952] conn(0x7f858c02ff70 0x7f858c032430 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:09:47.925 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:47.922+0000 7f85a6739700 1 --2- 172.21.15.84:0/691100407 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f85a00703b0 0x7f85a010cda0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:09:47.925 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:47.922+0000 7f85a6739700 1 -- 172.21.15.84:0/691100407 >> 172.21.15.84:0/691100407 conn(0x7f85a00f6530 msgr2=0x7f85a00f7f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:09:47.925 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:47.923+0000 7f85a6739700 1 -- 172.21.15.84:0/691100407 shutdown_connections 2024-02-20T00:09:47.925 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:47.923+0000 7f85a6739700 1 -- 172.21.15.84:0/691100407 wait complete. 2024-02-20T00:09:47.946 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:47 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:09:46.634199+0000 mgr.smithi084.bmqrnf (mgr.14142) 7 : audit [DBG] from='client.14154 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T00:09:47.946 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:47 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cephadm 2024-02-20T00:09:46.634759+0000 mgr.smithi084.bmqrnf (mgr.14142) 8 : cephadm [INF] Saving service alertmanager spec with placement count:1 2024-02-20T00:09:47.946 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:47 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:09:46.637002+0000 mon.smithi084 (mon.0) 81 : audit [INF] from='mgr.14142 172.21.15.84:0/3829884648' entity='mgr.smithi084.bmqrnf' 2024-02-20T00:09:48.492 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi084 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/2a2e6c42-cf84-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp3ot6j0xn:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp4xryone4:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mgr mgr/dashboard/ssl_server_port 8443 2024-02-20T00:09:49.052 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:49.015+0000 7ff0ac088700 1 Processor -- start 2024-02-20T00:09:49.052 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:49.015+0000 7ff0ac088700 1 -- start start 2024-02-20T00:09:49.052 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:49.015+0000 7ff0ac088700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff0a4072520 0x7ff0a4070ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-20T00:09:49.052 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:49.015+0000 7ff0ac088700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0a4072940 con 0x7ff0a4072520 2024-02-20T00:09:49.052 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:49.016+0000 7ff0a9e24700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff0a4072520 0x7ff0a4070ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:09:49.052 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:49.016+0000 7ff0a9e24700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff0a4072520 0x7ff0a4070ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:59336/0 (socket says 172.21.15.84:59336) 2024-02-20T00:09:49.052 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:49.016+0000 7ff0a9e24700 1 -- 172.21.15.84:0/2022991489 learned_addr learned my addr 172.21.15.84:0/2022991489 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-02-20T00:09:49.052 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:49.016+0000 7ff0a9e24700 1 -- 172.21.15.84:0/2022991489 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0a4070fb0 con 0x7ff0a4072520 2024-02-20T00:09:49.053 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:49.016+0000 7ff0a9e24700 1 --2- 172.21.15.84:0/2022991489 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff0a4072520 0x7ff0a4070ac0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7ff09400b5f0 tx=0x7ff094006ee0).ready entity=mon.0 client_cookie=ef98fb928af877c0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:09:49.053 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:49.016+0000 7ff0a8e22700 1 -- 172.21.15.84:0/2022991489 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff09400d600 con 0x7ff0a4072520 2024-02-20T00:09:49.053 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:49.016+0000 7ff0a8e22700 1 -- 172.21.15.84:0/2022991489 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7ff09400dbf0 con 0x7ff0a4072520 2024-02-20T00:09:49.056 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:49.016+0000 7ff0a8e22700 1 -- 172.21.15.84:0/2022991489 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff094011cb0 con 0x7ff0a4072520 2024-02-20T00:09:49.056 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:49.017+0000 7ff0ac088700 1 -- 172.21.15.84:0/2022991489 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff0a4072520 msgr2=0x7ff0a4070ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:09:49.056 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:49.017+0000 7ff0ac088700 1 --2- 172.21.15.84:0/2022991489 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff0a4072520 0x7ff0a4070ac0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7ff09400b5f0 tx=0x7ff094006ee0).stop 2024-02-20T00:09:49.056 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:49.017+0000 7ff0ac088700 1 -- 172.21.15.84:0/2022991489 shutdown_connections 2024-02-20T00:09:49.056 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:49.017+0000 7ff0ac088700 1 --2- 172.21.15.84:0/2022991489 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff0a4072520 0x7ff0a4070ac0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:09:49.056 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:49.017+0000 7ff0ac088700 1 -- 172.21.15.84:0/2022991489 >> 172.21.15.84:0/2022991489 conn(0x7ff0a40f76a0 msgr2=0x7ff0a40f9ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:09:49.056 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:49.017+0000 7ff0ac088700 1 -- 172.21.15.84:0/2022991489 shutdown_connections 2024-02-20T00:09:49.056 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:49.017+0000 7ff0ac088700 1 -- 172.21.15.84:0/2022991489 wait complete. 2024-02-20T00:09:49.056 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:49.017+0000 7ff0ac088700 1 Processor -- start 2024-02-20T00:09:49.056 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:49.017+0000 7ff0ac088700 1 -- start start 2024-02-20T00:09:49.057 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:49.017+0000 7ff0ac088700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff0a4072520 0x7ff0a410ce40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:09:49.057 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:49.017+0000 7ff0ac088700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0a410d330 con 0x7ff0a4072520 2024-02-20T00:09:49.057 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:49.017+0000 7ff0a9e24700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff0a4072520 0x7ff0a410ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:09:49.057 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:49.017+0000 7ff0a9e24700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff0a4072520 0x7ff0a410ce40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:59350/0 (socket says 172.21.15.84:59350) 2024-02-20T00:09:49.057 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:49.018+0000 7ff0a9e24700 1 -- 172.21.15.84:0/3386236090 learned_addr learned my addr 172.21.15.84:0/3386236090 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-02-20T00:09:49.057 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:49.018+0000 7ff0a9e24700 1 -- 172.21.15.84:0/3386236090 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff09400b200 con 0x7ff0a4072520 2024-02-20T00:09:49.057 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:49.018+0000 7ff0a9e24700 1 --2- 172.21.15.84:0/3386236090 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff0a4072520 0x7ff0a410ce40 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7ff09400dfc0 tx=0x7ff094006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:09:49.057 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:49.018+0000 7ff0927fc700 1 -- 172.21.15.84:0/3386236090 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff09401b410 con 0x7ff0a4072520 2024-02-20T00:09:49.057 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:49.018+0000 7ff0927fc700 1 -- 172.21.15.84:0/3386236090 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7ff09401ba00 con 0x7ff0a4072520 2024-02-20T00:09:49.057 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:49.018+0000 7ff0927fc700 1 -- 172.21.15.84:0/3386236090 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff09401a990 con 0x7ff0a4072520 2024-02-20T00:09:49.057 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:49.018+0000 7ff0ac088700 1 -- 172.21.15.84:0/3386236090 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0a410d530 con 0x7ff0a4072520 2024-02-20T00:09:49.057 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:49.018+0000 7ff0ac088700 1 -- 172.21.15.84:0/3386236090 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0a410e140 con 0x7ff0a4072520 2024-02-20T00:09:49.057 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:49.019+0000 7ff0927fc700 1 -- 172.21.15.84:0/3386236090 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7ff09400f070 con 0x7ff0a4072520 2024-02-20T00:09:49.057 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:49.019+0000 7ff0927fc700 1 --2- 172.21.15.84:0/3386236090 >> [v2:172.21.15.84:6800/1962643952,v1:172.21.15.84:6801/1962643952] conn(0x7ff08c02ff70 0x7ff08c032430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:09:49.058 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:49.019+0000 7ff0927fc700 1 -- 172.21.15.84:0/3386236090 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7ff094041410 con 0x7ff0a4072520 2024-02-20T00:09:49.058 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:49.019+0000 7ff0a9623700 1 --2- 172.21.15.84:0/3386236090 >> [v2:172.21.15.84:6800/1962643952,v1:172.21.15.84:6801/1962643952] conn(0x7ff08c02ff70 0x7ff08c032430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:09:49.058 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:49.019+0000 7ff0a9623700 1 --2- 172.21.15.84:0/3386236090 >> [v2:172.21.15.84:6800/1962643952,v1:172.21.15.84:6801/1962643952] conn(0x7ff08c02ff70 0x7ff08c032430 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7ff0a00096a0 tx=0x7ff0a0006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:09:49.058 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:49.019+0000 7ff0ac088700 1 -- 172.21.15.84:0/3386236090 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0a4059100 con 0x7ff0a4072520 2024-02-20T00:09:49.058 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:49.023+0000 7ff0927fc700 1 -- 172.21.15.84:0/3386236090 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff094016070 con 0x7ff0a4072520 2024-02-20T00:09:49.178 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:49.177+0000 7ff0ac088700 1 -- 172.21.15.84:0/3386236090 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7ff0a410ef70 con 0x7ff0a4072520 2024-02-20T00:09:49.181 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:49.179+0000 7ff0927fc700 1 -- 172.21.15.84:0/3386236090 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v9)=0 v9) v1 ==== 130+0+0 (secure 0 0 0) 0x7ff0940224f0 con 0x7ff0a4072520 2024-02-20T00:09:49.183 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:49.181+0000 7ff0ac088700 1 -- 172.21.15.84:0/3386236090 >> [v2:172.21.15.84:6800/1962643952,v1:172.21.15.84:6801/1962643952] conn(0x7ff08c02ff70 msgr2=0x7ff08c032430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:09:49.183 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:49.181+0000 7ff0ac088700 1 --2- 172.21.15.84:0/3386236090 >> [v2:172.21.15.84:6800/1962643952,v1:172.21.15.84:6801/1962643952] conn(0x7ff08c02ff70 0x7ff08c032430 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7ff0a00096a0 tx=0x7ff0a0006b40).stop 2024-02-20T00:09:49.183 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:49.181+0000 7ff0ac088700 1 -- 172.21.15.84:0/3386236090 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff0a4072520 msgr2=0x7ff0a410ce40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:09:49.184 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:49.181+0000 7ff0ac088700 1 --2- 172.21.15.84:0/3386236090 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff0a4072520 0x7ff0a410ce40 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7ff09400dfc0 tx=0x7ff094006ee0).stop 2024-02-20T00:09:49.184 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:49.181+0000 7ff0ac088700 1 -- 172.21.15.84:0/3386236090 shutdown_connections 2024-02-20T00:09:49.184 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:49.181+0000 7ff0ac088700 1 --2- 172.21.15.84:0/3386236090 >> [v2:172.21.15.84:6800/1962643952,v1:172.21.15.84:6801/1962643952] conn(0x7ff08c02ff70 0x7ff08c032430 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:09:49.184 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:49.181+0000 7ff0ac088700 1 --2- 172.21.15.84:0/3386236090 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff0a4072520 0x7ff0a410ce40 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:09:49.184 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:49.181+0000 7ff0ac088700 1 -- 172.21.15.84:0/3386236090 >> 172.21.15.84:0/3386236090 conn(0x7ff0a40f76a0 msgr2=0x7ff0a40f90c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:09:49.184 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:49.182+0000 7ff0ac088700 1 -- 172.21.15.84:0/3386236090 shutdown_connections 2024-02-20T00:09:49.184 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:49.182+0000 7ff0ac088700 1 -- 172.21.15.84:0/3386236090 wait complete. 2024-02-20T00:09:49.202 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:48 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:09:47.921042+0000 mon.smithi084 (mon.0) 82 : audit [INF] from='client.? 172.21.15.84:0/691100407' entity='client.admin' 2024-02-20T00:09:49.800 INFO:teuthology.orchestra.run.smithi084.stderr:Enabling the dashboard module... 2024-02-20T00:09:49.800 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi084 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/2a2e6c42-cf84-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp3ot6j0xn:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp4xryone4:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable dashboard 2024-02-20T00:09:50.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:50 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:09:49.180287+0000 mon.smithi084 (mon.0) 83 : audit [INF] from='client.? 172.21.15.84:0/3386236090' entity='client.admin' 2024-02-20T00:09:50.563 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:50.556+0000 7f10aea11700 1 Processor -- start 2024-02-20T00:09:50.563 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:50.556+0000 7f10aea11700 1 -- start start 2024-02-20T00:09:50.563 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:50.556+0000 7f10aea11700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f10a8070430 0x7f10a80fea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-20T00:09:50.563 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:50.556+0000 7f10aea11700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10a80fef40 con 0x7f10a8070430 2024-02-20T00:09:50.563 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:50.557+0000 7f10a7fff700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f10a8070430 0x7f10a80fea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:09:50.564 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:50.557+0000 7f10a7fff700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f10a8070430 0x7f10a80fea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:59366/0 (socket says 172.21.15.84:59366) 2024-02-20T00:09:50.564 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:50.557+0000 7f10a7fff700 1 -- 172.21.15.84:0/1620164082 learned_addr learned my addr 172.21.15.84:0/1620164082 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-02-20T00:09:50.564 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:50.557+0000 7f10a7fff700 1 -- 172.21.15.84:0/1620164082 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10a80ff080 con 0x7f10a8070430 2024-02-20T00:09:50.564 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:50.557+0000 7f10a7fff700 1 --2- 172.21.15.84:0/1620164082 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f10a8070430 0x7f10a80fea50 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f109800b5f0 tx=0x7f1098006ee0).ready entity=mon.0 client_cookie=3dc346bc47a5aaa4 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:09:50.564 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:50.557+0000 7f10a6ffd700 1 -- 172.21.15.84:0/1620164082 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f109800d600 con 0x7f10a8070430 2024-02-20T00:09:50.564 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:50.557+0000 7f10a6ffd700 1 -- 172.21.15.84:0/1620164082 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f109800dbf0 con 0x7f10a8070430 2024-02-20T00:09:50.564 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:50.557+0000 7f10a6ffd700 1 -- 172.21.15.84:0/1620164082 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1098011cb0 con 0x7f10a8070430 2024-02-20T00:09:50.564 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:50.558+0000 7f10aea11700 1 -- 172.21.15.84:0/1620164082 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f10a8070430 msgr2=0x7f10a80fea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:09:50.564 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:50.558+0000 7f10aea11700 1 --2- 172.21.15.84:0/1620164082 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f10a8070430 0x7f10a80fea50 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f109800b5f0 tx=0x7f1098006ee0).stop 2024-02-20T00:09:50.564 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:50.558+0000 7f10aea11700 1 -- 172.21.15.84:0/1620164082 shutdown_connections 2024-02-20T00:09:50.564 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:50.558+0000 7f10aea11700 1 --2- 172.21.15.84:0/1620164082 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f10a8070430 0x7f10a80fea50 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:09:50.564 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:50.558+0000 7f10aea11700 1 -- 172.21.15.84:0/1620164082 >> 172.21.15.84:0/1620164082 conn(0x7f10a80f6530 msgr2=0x7f10a80f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:09:50.564 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:50.558+0000 7f10aea11700 1 -- 172.21.15.84:0/1620164082 shutdown_connections 2024-02-20T00:09:50.565 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:50.558+0000 7f10aea11700 1 -- 172.21.15.84:0/1620164082 wait complete. 2024-02-20T00:09:50.565 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:50.558+0000 7f10aea11700 1 Processor -- start 2024-02-20T00:09:50.565 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:50.559+0000 7f10aea11700 1 -- start start 2024-02-20T00:09:50.565 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:50.559+0000 7f10aea11700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f10a8070430 0x7f10a810ab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:09:50.565 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:50.559+0000 7f10aea11700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10a810b020 con 0x7f10a8070430 2024-02-20T00:09:50.565 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:50.559+0000 7f10a7fff700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f10a8070430 0x7f10a810ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:09:50.565 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:50.559+0000 7f10a7fff700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f10a8070430 0x7f10a810ab30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:59378/0 (socket says 172.21.15.84:59378) 2024-02-20T00:09:50.566 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:50.559+0000 7f10a7fff700 1 -- 172.21.15.84:0/1834703525 learned_addr learned my addr 172.21.15.84:0/1834703525 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-02-20T00:09:50.566 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:50.559+0000 7f10a7fff700 1 -- 172.21.15.84:0/1834703525 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f109800b200 con 0x7f10a8070430 2024-02-20T00:09:50.566 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:50.559+0000 7f10a7fff700 1 --2- 172.21.15.84:0/1834703525 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f10a8070430 0x7f10a810ab30 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f1098006800 tx=0x7f1098006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:09:50.566 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:50.560+0000 7f10a4ff9700 1 -- 172.21.15.84:0/1834703525 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f109801b570 con 0x7f10a8070430 2024-02-20T00:09:50.566 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:50.560+0000 7f10a4ff9700 1 -- 172.21.15.84:0/1834703525 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f109801bb60 con 0x7f10a8070430 2024-02-20T00:09:50.566 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:50.560+0000 7f10aea11700 1 -- 172.21.15.84:0/1834703525 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10a810b220 con 0x7f10a8070430 2024-02-20T00:09:50.566 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:50.560+0000 7f10a4ff9700 1 -- 172.21.15.84:0/1834703525 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f109801a9f0 con 0x7f10a8070430 2024-02-20T00:09:50.566 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:50.560+0000 7f10aea11700 1 -- 172.21.15.84:0/1834703525 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10a810b6b0 con 0x7f10a8070430 2024-02-20T00:09:50.566 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:50.561+0000 7f10a4ff9700 1 -- 172.21.15.84:0/1834703525 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f109800f070 con 0x7f10a8070430 2024-02-20T00:09:50.566 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:50.561+0000 7f10aea11700 1 -- 172.21.15.84:0/1834703525 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10a8045ad0 con 0x7f10a8070430 2024-02-20T00:09:50.566 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:50.561+0000 7f10a4ff9700 1 --2- 172.21.15.84:0/1834703525 >> [v2:172.21.15.84:6800/1962643952,v1:172.21.15.84:6801/1962643952] conn(0x7f109402ff20 0x7f10940323e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:09:50.566 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:50.561+0000 7f10a4ff9700 1 -- 172.21.15.84:0/1834703525 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f1098041620 con 0x7f10a8070430 2024-02-20T00:09:50.566 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:50.561+0000 7f10a77fe700 1 --2- 172.21.15.84:0/1834703525 >> [v2:172.21.15.84:6800/1962643952,v1:172.21.15.84:6801/1962643952] conn(0x7f109402ff20 0x7f10940323e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:09:50.567 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:50.561+0000 7f10a77fe700 1 --2- 172.21.15.84:0/1834703525 >> [v2:172.21.15.84:6800/1962643952,v1:172.21.15.84:6801/1962643952] conn(0x7f109402ff20 0x7f10940323e0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f109c0096a0 tx=0x7f109c006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:09:50.567 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:50.565+0000 7f10a4ff9700 1 -- 172.21.15.84:0/1834703525 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1098010360 con 0x7f10a8070430 2024-02-20T00:09:50.754 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:50.753+0000 7f10aea11700 1 -- 172.21.15.84:0/1834703525 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7f10a80ffdc0 con 0x7f10a8070430 2024-02-20T00:09:51.127 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:51.125+0000 7f10a4ff9700 1 -- 172.21.15.84:0/1834703525 <== mon.0 v2:172.21.15.84:3300/0 7 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1098010360 con 0x7f10a8070430 2024-02-20T00:09:51.187 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:51.185+0000 7f10a4ff9700 1 -- 172.21.15.84:0/1834703525 <== mon.0 v2:172.21.15.84:3300/0 8 ==== mgrmap(e 13) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f1098025460 con 0x7f10a8070430 2024-02-20T00:09:51.188 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:51.186+0000 7f10a4ff9700 1 -- 172.21.15.84:0/1834703525 <== mon.0 v2:172.21.15.84:3300/0 9 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v13) v1 ==== 88+0+0 (secure 0 0 0) 0x7f10980236e0 con 0x7f10a8070430 2024-02-20T00:09:51.190 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:51.188+0000 7f10aea11700 1 -- 172.21.15.84:0/1834703525 >> [v2:172.21.15.84:6800/1962643952,v1:172.21.15.84:6801/1962643952] conn(0x7f109402ff20 msgr2=0x7f10940323e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:09:51.190 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:51.188+0000 7f10aea11700 1 --2- 172.21.15.84:0/1834703525 >> [v2:172.21.15.84:6800/1962643952,v1:172.21.15.84:6801/1962643952] conn(0x7f109402ff20 0x7f10940323e0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f109c0096a0 tx=0x7f109c006b40).stop 2024-02-20T00:09:51.191 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:51.188+0000 7f10aea11700 1 -- 172.21.15.84:0/1834703525 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f10a8070430 msgr2=0x7f10a810ab30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:09:51.191 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:51.188+0000 7f10aea11700 1 --2- 172.21.15.84:0/1834703525 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f10a8070430 0x7f10a810ab30 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f1098006800 tx=0x7f1098006ee0).stop 2024-02-20T00:09:51.191 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:51.188+0000 7f10aea11700 1 -- 172.21.15.84:0/1834703525 shutdown_connections 2024-02-20T00:09:51.191 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:51.188+0000 7f10aea11700 1 --2- 172.21.15.84:0/1834703525 >> [v2:172.21.15.84:6800/1962643952,v1:172.21.15.84:6801/1962643952] conn(0x7f109402ff20 0x7f10940323e0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:09:51.191 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:51.188+0000 7f10aea11700 1 --2- 172.21.15.84:0/1834703525 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f10a8070430 0x7f10a810ab30 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:09:51.191 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:51.188+0000 7f10aea11700 1 -- 172.21.15.84:0/1834703525 >> 172.21.15.84:0/1834703525 conn(0x7f10a80f6530 msgr2=0x7f10a80f7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:09:51.191 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:51.188+0000 7f10aea11700 1 -- 172.21.15.84:0/1834703525 shutdown_connections 2024-02-20T00:09:51.191 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:51.188+0000 7f10aea11700 1 -- 172.21.15.84:0/1834703525 wait complete. 2024-02-20T00:09:51.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:51 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:09:50.753952+0000 mon.smithi084 (mon.0) 84 : audit [INF] from='client.? 172.21.15.84:0/1834703525' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-02-20T00:09:51.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:51 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:09:51.126232+0000 mon.smithi084 (mon.0) 85 : audit [INF] from='mgr.14142 172.21.15.84:0/3829884648' entity='mgr.smithi084.bmqrnf' 2024-02-20T00:09:51.889 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi084 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/2a2e6c42-cf84-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp3ot6j0xn:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp4xryone4:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr stat 2024-02-20T00:09:52.525 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:52.522+0000 7f9b5b59e700 1 Processor -- start 2024-02-20T00:09:52.525 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:52.522+0000 7f9b5b59e700 1 -- start start 2024-02-20T00:09:52.525 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:52.522+0000 7f9b5b59e700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9b5c0703d0 0x7f9b5c0fe8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-20T00:09:52.525 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:52.522+0000 7f9b5b59e700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b5c0fede0 con 0x7f9b5c0703d0 2024-02-20T00:09:52.525 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:52.522+0000 7f9b5a59c700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9b5c0703d0 0x7f9b5c0fe8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:09:52.525 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:52.523+0000 7f9b5a59c700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9b5c0703d0 0x7f9b5c0fe8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:59396/0 (socket says 172.21.15.84:59396) 2024-02-20T00:09:52.525 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:52.523+0000 7f9b5a59c700 1 -- 172.21.15.84:0/2669992956 learned_addr learned my addr 172.21.15.84:0/2669992956 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-02-20T00:09:52.525 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:52.523+0000 7f9b5a59c700 1 -- 172.21.15.84:0/2669992956 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b5c0fef20 con 0x7f9b5c0703d0 2024-02-20T00:09:52.525 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:52.523+0000 7f9b5a59c700 1 --2- 172.21.15.84:0/2669992956 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9b5c0703d0 0x7f9b5c0fe8f0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f9b4c00b5f0 tx=0x7f9b4c006ee0).ready entity=mon.0 client_cookie=e5a480db8f814fa4 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:09:52.525 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:52.523+0000 7f9b5959a700 1 -- 172.21.15.84:0/2669992956 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9b4c00d600 con 0x7f9b5c0703d0 2024-02-20T00:09:52.526 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:52.523+0000 7f9b5959a700 1 -- 172.21.15.84:0/2669992956 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9b4c00dc30 con 0x7f9b5c0703d0 2024-02-20T00:09:52.526 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:52.523+0000 7f9b5959a700 1 -- 172.21.15.84:0/2669992956 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9b4c011d30 con 0x7f9b5c0703d0 2024-02-20T00:09:52.528 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:52.524+0000 7f9b5b59e700 1 -- 172.21.15.84:0/2669992956 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9b5c0703d0 msgr2=0x7f9b5c0fe8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:09:52.529 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:52.524+0000 7f9b5b59e700 1 --2- 172.21.15.84:0/2669992956 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9b5c0703d0 0x7f9b5c0fe8f0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f9b4c00b5f0 tx=0x7f9b4c006ee0).stop 2024-02-20T00:09:52.529 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:52.524+0000 7f9b5b59e700 1 -- 172.21.15.84:0/2669992956 shutdown_connections 2024-02-20T00:09:52.529 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:52.524+0000 7f9b5b59e700 1 --2- 172.21.15.84:0/2669992956 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9b5c0703d0 0x7f9b5c0fe8f0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:09:52.529 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:52.524+0000 7f9b5b59e700 1 -- 172.21.15.84:0/2669992956 >> 172.21.15.84:0/2669992956 conn(0x7f9b5c0f6390 msgr2=0x7f9b5c0f87b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:09:52.529 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:52.524+0000 7f9b5b59e700 1 -- 172.21.15.84:0/2669992956 shutdown_connections 2024-02-20T00:09:52.529 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:52.524+0000 7f9b5b59e700 1 -- 172.21.15.84:0/2669992956 wait complete. 2024-02-20T00:09:52.529 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:52.524+0000 7f9b5b59e700 1 Processor -- start 2024-02-20T00:09:52.529 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:52.525+0000 7f9b5b59e700 1 -- start start 2024-02-20T00:09:52.529 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:52.525+0000 7f9b5b59e700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9b5c0703d0 0x7f9b5c0fc370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:09:52.529 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:52.525+0000 7f9b5b59e700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b5c0fa7f0 con 0x7f9b5c0703d0 2024-02-20T00:09:52.529 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:52.525+0000 7f9b5a59c700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9b5c0703d0 0x7f9b5c0fc370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:09:52.529 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:52.525+0000 7f9b5a59c700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9b5c0703d0 0x7f9b5c0fc370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:59408/0 (socket says 172.21.15.84:59408) 2024-02-20T00:09:52.530 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:52.525+0000 7f9b5a59c700 1 -- 172.21.15.84:0/3194622285 learned_addr learned my addr 172.21.15.84:0/3194622285 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-02-20T00:09:52.530 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:52.525+0000 7f9b5a59c700 1 -- 172.21.15.84:0/3194622285 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b4c00b200 con 0x7f9b5c0703d0 2024-02-20T00:09:52.530 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:52.525+0000 7f9b5a59c700 1 --2- 172.21.15.84:0/3194622285 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9b5c0703d0 0x7f9b5c0fc370 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f9b4c006c70 tx=0x7f9b4c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:09:52.530 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:52.526+0000 7f9b4affd700 1 -- 172.21.15.84:0/3194622285 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9b4c00d850 con 0x7f9b5c0703d0 2024-02-20T00:09:52.530 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:52.526+0000 7f9b5b59e700 1 -- 172.21.15.84:0/3194622285 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b5c0fa990 con 0x7f9b5c0703d0 2024-02-20T00:09:52.530 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:52.526+0000 7f9b4affd700 1 -- 172.21.15.84:0/3194622285 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9b4c00d9b0 con 0x7f9b5c0703d0 2024-02-20T00:09:52.530 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:52.526+0000 7f9b4affd700 1 -- 172.21.15.84:0/3194622285 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9b4c01a750 con 0x7f9b5c0703d0 2024-02-20T00:09:52.530 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:52.526+0000 7f9b5b59e700 1 -- 172.21.15.84:0/3194622285 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b5c0fae20 con 0x7f9b5c0703d0 2024-02-20T00:09:52.530 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:52.527+0000 7f9b4affd700 1 -- 172.21.15.84:0/3194622285 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 13) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f9b4c00f070 con 0x7f9b5c0703d0 2024-02-20T00:09:52.530 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:52.527+0000 7f9b4affd700 1 --2- 172.21.15.84:0/3194622285 >> [v2:172.21.15.84:6800/1962643952,v1:172.21.15.84:6801/1962643952] conn(0x7f9b4002ff70 0x7f9b40032430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:09:52.530 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:52.527+0000 7f9b4affd700 1 -- 172.21.15.84:0/3194622285 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f9b4c040f00 con 0x7f9b5c0703d0 2024-02-20T00:09:52.530 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:52.527+0000 7f9b59d9b700 1 -- 172.21.15.84:0/3194622285 >> [v2:172.21.15.84:6800/1962643952,v1:172.21.15.84:6801/1962643952] conn(0x7f9b4002ff70 msgr2=0x7f9b40032430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.84:6800/1962643952 2024-02-20T00:09:52.530 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:52.527+0000 7f9b59d9b700 1 --2- 172.21.15.84:0/3194622285 >> [v2:172.21.15.84:6800/1962643952,v1:172.21.15.84:6801/1962643952] conn(0x7f9b4002ff70 0x7f9b40032430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-02-20T00:09:52.530 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:52.527+0000 7f9b5b59e700 1 -- 172.21.15.84:0/3194622285 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b5c059100 con 0x7f9b5c0703d0 2024-02-20T00:09:52.533 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:52.532+0000 7f9b4affd700 1 -- 172.21.15.84:0/3194622285 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9b4c016070 con 0x7f9b5c0703d0 2024-02-20T00:09:52.607 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:52 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:09:51.187080+0000 mon.smithi084 (mon.0) 86 : audit [INF] from='client.? 172.21.15.84:0/1834703525' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-02-20T00:09:52.608 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:52 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:09:51.187147+0000 mon.smithi084 (mon.0) 87 : cluster [DBG] mgrmap e13: smithi084.bmqrnf(active, since 9s) 2024-02-20T00:09:52.729 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:52.726+0000 7f9b5b59e700 1 -- 172.21.15.84:0/3194622285 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f9b5c0ffcd0 con 0x7f9b5c0703d0 2024-02-20T00:09:52.729 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:52.727+0000 7f9b4affd700 1 -- 172.21.15.84:0/3194622285 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v13) v1 ==== 56+0+104 (secure 0 0 0) 0x7f9b4c01b410 con 0x7f9b5c0703d0 2024-02-20T00:09:52.729 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: { 2024-02-20T00:09:52.729 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "epoch": 13, 2024-02-20T00:09:52.729 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "available": true, 2024-02-20T00:09:52.729 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "active_name": "smithi084.bmqrnf", 2024-02-20T00:09:52.730 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "num_standby": 0 2024-02-20T00:09:52.730 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: } 2024-02-20T00:09:52.730 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:52.728+0000 7f9b59d9b700 1 -- 172.21.15.84:0/3194622285 >> [v2:172.21.15.84:6800/1962643952,v1:172.21.15.84:6801/1962643952] conn(0x7f9b4002ff70 msgr2=0x7f9b40032430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.84:6800/1962643952 2024-02-20T00:09:52.730 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:52.728+0000 7f9b59d9b700 1 --2- 172.21.15.84:0/3194622285 >> [v2:172.21.15.84:6800/1962643952,v1:172.21.15.84:6801/1962643952] conn(0x7f9b4002ff70 0x7f9b40032430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2024-02-20T00:09:52.731 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:52.729+0000 7f9b5b59e700 1 -- 172.21.15.84:0/3194622285 >> [v2:172.21.15.84:6800/1962643952,v1:172.21.15.84:6801/1962643952] conn(0x7f9b4002ff70 msgr2=0x7f9b40032430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-20T00:09:52.731 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:52.729+0000 7f9b5b59e700 1 --2- 172.21.15.84:0/3194622285 >> [v2:172.21.15.84:6800/1962643952,v1:172.21.15.84:6801/1962643952] conn(0x7f9b4002ff70 0x7f9b40032430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-20T00:09:52.731 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:52.729+0000 7f9b5b59e700 1 -- 172.21.15.84:0/3194622285 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9b5c0703d0 msgr2=0x7f9b5c0fc370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:09:52.731 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:52.729+0000 7f9b5b59e700 1 --2- 172.21.15.84:0/3194622285 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9b5c0703d0 0x7f9b5c0fc370 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f9b4c006c70 tx=0x7f9b4c006ee0).stop 2024-02-20T00:09:52.731 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:52.729+0000 7f9b5b59e700 1 -- 172.21.15.84:0/3194622285 shutdown_connections 2024-02-20T00:09:52.731 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:52.729+0000 7f9b5b59e700 1 --2- 172.21.15.84:0/3194622285 >> [v2:172.21.15.84:6800/1962643952,v1:172.21.15.84:6801/1962643952] conn(0x7f9b4002ff70 0x7f9b40032430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-20T00:09:52.731 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:52.729+0000 7f9b5b59e700 1 --2- 172.21.15.84:0/3194622285 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9b5c0703d0 0x7f9b5c0fc370 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:09:52.732 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:52.729+0000 7f9b5b59e700 1 -- 172.21.15.84:0/3194622285 >> 172.21.15.84:0/3194622285 conn(0x7f9b5c0f6390 msgr2=0x7f9b5c0f7e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:09:52.732 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:52.729+0000 7f9b5b59e700 1 -- 172.21.15.84:0/3194622285 shutdown_connections 2024-02-20T00:09:52.732 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:52.729+0000 7f9b5b59e700 1 -- 172.21.15.84:0/3194622285 wait complete. 2024-02-20T00:09:53.337 INFO:teuthology.orchestra.run.smithi084.stderr:Waiting for the mgr to restart... 2024-02-20T00:09:53.337 INFO:teuthology.orchestra.run.smithi084.stderr:Waiting for mgr epoch 13... 2024-02-20T00:09:53.338 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi084 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/2a2e6c42-cf84-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp3ot6j0xn:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp4xryone4:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 tell mgr mgr_status 2024-02-20T00:09:53.339 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:53 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:09:52.727729+0000 mon.smithi084 (mon.0) 88 : audit [DBG] from='client.? 172.21.15.84:0/3194622285' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-02-20T00:09:53.878 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:53.875+0000 7fc53723b700 1 Processor -- start 2024-02-20T00:09:53.879 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:53.876+0000 7fc53723b700 1 -- start start 2024-02-20T00:09:53.879 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:53.876+0000 7fc53723b700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc5300fab30 0x7fc5300faf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-20T00:09:53.879 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:53.876+0000 7fc53723b700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5300fb440 con 0x7fc5300fab30 2024-02-20T00:09:53.879 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:53.876+0000 7fc536239700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc5300fab30 0x7fc5300faf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:09:53.879 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:53.876+0000 7fc536239700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc5300fab30 0x7fc5300faf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:59424/0 (socket says 172.21.15.84:59424) 2024-02-20T00:09:53.879 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:53.876+0000 7fc536239700 1 -- 172.21.15.84:0/4241846964 learned_addr learned my addr 172.21.15.84:0/4241846964 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-02-20T00:09:53.879 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:53.876+0000 7fc536239700 1 -- 172.21.15.84:0/4241846964 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5300fb580 con 0x7fc5300fab30 2024-02-20T00:09:53.879 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:53.876+0000 7fc536239700 1 --2- 172.21.15.84:0/4241846964 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc5300fab30 0x7fc5300faf50 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7fc52c00b870 tx=0x7fc52c006f70).ready entity=mon.0 client_cookie=1209aa7129036bd1 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:09:53.879 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:53.877+0000 7fc535237700 1 -- 172.21.15.84:0/4241846964 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc52c00da20 con 0x7fc5300fab30 2024-02-20T00:09:53.879 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:53.877+0000 7fc535237700 1 -- 172.21.15.84:0/4241846964 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc52c00db80 con 0x7fc5300fab30 2024-02-20T00:09:53.880 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:53.877+0000 7fc535237700 1 -- 172.21.15.84:0/4241846964 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc52c00f5d0 con 0x7fc5300fab30 2024-02-20T00:09:53.881 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:53.877+0000 7fc53723b700 1 -- 172.21.15.84:0/4241846964 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc5300fab30 msgr2=0x7fc5300faf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:09:53.881 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:53.877+0000 7fc53723b700 1 --2- 172.21.15.84:0/4241846964 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc5300fab30 0x7fc5300faf50 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7fc52c00b870 tx=0x7fc52c006f70).stop 2024-02-20T00:09:53.881 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:53.877+0000 7fc53723b700 1 -- 172.21.15.84:0/4241846964 shutdown_connections 2024-02-20T00:09:53.881 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:53.877+0000 7fc53723b700 1 --2- 172.21.15.84:0/4241846964 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc5300fab30 0x7fc5300faf50 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:09:53.881 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:53.877+0000 7fc53723b700 1 -- 172.21.15.84:0/4241846964 >> 172.21.15.84:0/4241846964 conn(0x7fc5300f6390 msgr2=0x7fc5300f87b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:09:53.881 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:53.877+0000 7fc53723b700 1 -- 172.21.15.84:0/4241846964 shutdown_connections 2024-02-20T00:09:53.881 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:53.877+0000 7fc53723b700 1 -- 172.21.15.84:0/4241846964 wait complete. 2024-02-20T00:09:53.881 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:53.878+0000 7fc53723b700 1 Processor -- start 2024-02-20T00:09:53.881 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:53.878+0000 7fc53723b700 1 -- start start 2024-02-20T00:09:53.881 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:53.878+0000 7fc53723b700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc5300fab30 0x7fc530108700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:09:53.881 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:53.878+0000 7fc53723b700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc530108bf0 con 0x7fc5300fab30 2024-02-20T00:09:53.881 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:53.878+0000 7fc536239700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc5300fab30 0x7fc530108700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:09:53.881 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:53.878+0000 7fc536239700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc5300fab30 0x7fc530108700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:59440/0 (socket says 172.21.15.84:59440) 2024-02-20T00:09:53.882 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:53.878+0000 7fc536239700 1 -- 172.21.15.84:0/1863972098 learned_addr learned my addr 172.21.15.84:0/1863972098 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-02-20T00:09:53.882 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:53.878+0000 7fc536239700 1 -- 172.21.15.84:0/1863972098 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc52c00b480 con 0x7fc5300fab30 2024-02-20T00:09:53.882 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:53.879+0000 7fc536239700 1 --2- 172.21.15.84:0/1863972098 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc5300fab30 0x7fc530108700 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7fc52c006a40 tx=0x7fc52c006f70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:09:53.882 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:53.879+0000 7fc51effd700 1 -- 172.21.15.84:0/1863972098 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc52c00f590 con 0x7fc5300fab30 2024-02-20T00:09:53.882 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:53.879+0000 7fc51effd700 1 -- 172.21.15.84:0/1863972098 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc52c00fee0 con 0x7fc5300fab30 2024-02-20T00:09:53.882 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:53.879+0000 7fc53723b700 1 -- 172.21.15.84:0/1863972098 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc530108df0 con 0x7fc5300fab30 2024-02-20T00:09:53.882 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:53.879+0000 7fc53723b700 1 -- 172.21.15.84:0/1863972098 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc530109280 con 0x7fc5300fab30 2024-02-20T00:09:53.882 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:53.879+0000 7fc51effd700 1 -- 172.21.15.84:0/1863972098 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc52c019d30 con 0x7fc5300fab30 2024-02-20T00:09:53.883 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:53.880+0000 7fc51effd700 1 -- 172.21.15.84:0/1863972098 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 13) v1 ==== 36208+0+0 (secure 0 0 0) 0x7fc52c016030 con 0x7fc5300fab30 2024-02-20T00:09:53.883 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:53.880+0000 7fc51effd700 1 --2- 172.21.15.84:0/1863972098 >> [v2:172.21.15.84:6800/1962643952,v1:172.21.15.84:6801/1962643952] conn(0x7fc51802ffc0 0x7fc518032480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:09:53.883 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:53.880+0000 7fc51effd700 1 -- 172.21.15.84:0/1863972098 --> [v2:172.21.15.84:6800/1962643952,v1:172.21.15.84:6801/1962643952] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc518032b40 con 0x7fc51802ffc0 2024-02-20T00:09:53.883 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:53.880+0000 7fc51effd700 1 -- 172.21.15.84:0/1863972098 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7fc52c044490 con 0x7fc5300fab30 2024-02-20T00:09:53.883 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:53.880+0000 7fc535a38700 1 -- 172.21.15.84:0/1863972098 >> [v2:172.21.15.84:6800/1962643952,v1:172.21.15.84:6801/1962643952] conn(0x7fc51802ffc0 msgr2=0x7fc518032480 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.84:6800/1962643952 2024-02-20T00:09:53.883 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:53.880+0000 7fc535a38700 1 --2- 172.21.15.84:0/1863972098 >> [v2:172.21.15.84:6800/1962643952,v1:172.21.15.84:6801/1962643952] conn(0x7fc51802ffc0 0x7fc518032480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-02-20T00:09:54.082 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:54.081+0000 7fc535a38700 1 -- 172.21.15.84:0/1863972098 >> [v2:172.21.15.84:6800/1962643952,v1:172.21.15.84:6801/1962643952] conn(0x7fc51802ffc0 msgr2=0x7fc518032480 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.84:6800/1962643952 2024-02-20T00:09:54.083 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:54.081+0000 7fc535a38700 1 --2- 172.21.15.84:0/1863972098 >> [v2:172.21.15.84:6800/1962643952,v1:172.21.15.84:6801/1962643952] conn(0x7fc51802ffc0 0x7fc518032480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2024-02-20T00:09:54.483 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:54.482+0000 7fc535a38700 1 -- 172.21.15.84:0/1863972098 >> [v2:172.21.15.84:6800/1962643952,v1:172.21.15.84:6801/1962643952] conn(0x7fc51802ffc0 msgr2=0x7fc518032480 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.84:6800/1962643952 2024-02-20T00:09:54.484 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:54.482+0000 7fc535a38700 1 --2- 172.21.15.84:0/1863972098 >> [v2:172.21.15.84:6800/1962643952,v1:172.21.15.84:6801/1962643952] conn(0x7fc51802ffc0 0x7fc518032480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2024-02-20T00:09:55.284 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:55.283+0000 7fc535a38700 1 -- 172.21.15.84:0/1863972098 >> [v2:172.21.15.84:6800/1962643952,v1:172.21.15.84:6801/1962643952] conn(0x7fc51802ffc0 msgr2=0x7fc518032480 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.84:6800/1962643952 2024-02-20T00:09:55.284 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:55.283+0000 7fc535a38700 1 --2- 172.21.15.84:0/1863972098 >> [v2:172.21.15.84:6800/1962643952,v1:172.21.15.84:6801/1962643952] conn(0x7fc51802ffc0 0x7fc518032480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2024-02-20T00:09:56.835 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:56.833+0000 7fc51effd700 1 -- 172.21.15.84:0/1863972098 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mgrmap(e 14) v1 ==== 35997+0+0 (secure 0 0 0) 0x7fc52c019550 con 0x7fc5300fab30 2024-02-20T00:09:56.835 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:56.833+0000 7fc51effd700 1 -- 172.21.15.84:0/1863972098 >> [v2:172.21.15.84:6800/1962643952,v1:172.21.15.84:6801/1962643952] conn(0x7fc51802ffc0 msgr2=0x7fc518032480 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-20T00:09:56.835 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:56.833+0000 7fc51effd700 1 --2- 172.21.15.84:0/1863972098 >> [v2:172.21.15.84:6800/1962643952,v1:172.21.15.84:6801/1962643952] conn(0x7fc51802ffc0 0x7fc518032480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-20T00:09:57.082 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:56 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:09:56.143266+0000 mon.smithi084 (mon.0) 89 : cluster [INF] Active manager daemon smithi084.bmqrnf restarted 2024-02-20T00:09:57.082 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:56 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:09:56.143700+0000 mon.smithi084 (mon.0) 90 : cluster [INF] Activating manager daemon smithi084.bmqrnf 2024-02-20T00:09:57.082 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:56 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:09:56.145315+0000 mon.smithi084 (mon.0) 91 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-02-20T00:09:57.840 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:57.838+0000 7fc51effd700 1 -- 172.21.15.84:0/1863972098 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mgrmap(e 15) v1 ==== 36067+0+0 (secure 0 0 0) 0x7fc52c02e440 con 0x7fc5300fab30 2024-02-20T00:09:57.840 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:57.838+0000 7fc51effd700 1 --2- 172.21.15.84:0/1863972098 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fc518033c50 0x7fc518036040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:09:57.841 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:57.838+0000 7fc51effd700 1 -- 172.21.15.84:0/1863972098 --> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc518032b40 con 0x7fc518033c50 2024-02-20T00:09:57.841 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:57.839+0000 7fc535a38700 1 --2- 172.21.15.84:0/1863972098 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fc518033c50 0x7fc518036040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:09:57.841 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:57.839+0000 7fc535a38700 1 --2- 172.21.15.84:0/1863972098 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fc518033c50 0x7fc518036040 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7fc5200036d0 tx=0x7fc520006d20).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:09:57.841 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:57.840+0000 7fc51effd700 1 -- 172.21.15.84:0/1863972098 <== mgr.14162 v2:172.21.15.84:6800/1303158740 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6438 (secure 0 0 0) 0x7fc518032b40 con 0x7fc518033c50 2024-02-20T00:09:57.848 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:57.845+0000 7fc53723b700 1 -- 172.21.15.84:0/1863972098 --> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7fc510000dc0 con 0x7fc518033c50 2024-02-20T00:09:57.848 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:57.846+0000 7fc51effd700 1 -- 172.21.15.84:0/1863972098 <== mgr.14162 v2:172.21.15.84:6800/1303158740 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7fc510000dc0 con 0x7fc518033c50 2024-02-20T00:09:57.849 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: { 2024-02-20T00:09:57.849 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "mgrmap_epoch": 15, 2024-02-20T00:09:57.849 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "initialized": true 2024-02-20T00:09:57.849 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: } 2024-02-20T00:09:57.849 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:57.846+0000 7fc53723b700 1 -- 172.21.15.84:0/1863972098 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fc518033c50 msgr2=0x7fc518036040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:09:57.849 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:57.846+0000 7fc53723b700 1 --2- 172.21.15.84:0/1863972098 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fc518033c50 0x7fc518036040 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7fc5200036d0 tx=0x7fc520006d20).stop 2024-02-20T00:09:57.849 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:57.846+0000 7fc53723b700 1 -- 172.21.15.84:0/1863972098 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc5300fab30 msgr2=0x7fc530108700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:09:57.849 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:57.846+0000 7fc53723b700 1 --2- 172.21.15.84:0/1863972098 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc5300fab30 0x7fc530108700 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7fc52c006a40 tx=0x7fc52c006f70).stop 2024-02-20T00:09:57.849 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:57.846+0000 7fc53723b700 1 -- 172.21.15.84:0/1863972098 shutdown_connections 2024-02-20T00:09:57.849 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:57.846+0000 7fc53723b700 1 --2- 172.21.15.84:0/1863972098 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fc518033c50 0x7fc518036040 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:09:57.849 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:57.846+0000 7fc53723b700 1 --2- 172.21.15.84:0/1863972098 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc5300fab30 0x7fc530108700 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:09:57.849 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:57.846+0000 7fc53723b700 1 --2- 172.21.15.84:0/1863972098 >> [v2:172.21.15.84:6800/1962643952,v1:172.21.15.84:6801/1962643952] conn(0x7fc51802ffc0 0x7fc518032480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-20T00:09:57.849 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:57.846+0000 7fc53723b700 1 -- 172.21.15.84:0/1863972098 >> 172.21.15.84:0/1863972098 conn(0x7fc5300f6390 msgr2=0x7fc5300ff8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:09:57.849 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:57.846+0000 7fc53723b700 1 -- 172.21.15.84:0/1863972098 shutdown_connections 2024-02-20T00:09:57.850 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:57.846+0000 7fc53723b700 1 -- 172.21.15.84:0/1863972098 wait complete. 2024-02-20T00:09:58.107 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:57 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:09:56.834806+0000 mon.smithi084 (mon.0) 92 : cluster [DBG] mgrmap e14: smithi084.bmqrnf(active, starting, since 0.691189s) 2024-02-20T00:09:58.108 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:57 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:09:56.835691+0000 mon.smithi084 (mon.0) 93 : audit [DBG] from='mgr.14162 172.21.15.84:0/2152933216' entity='mgr.smithi084.bmqrnf' cmd=[{"prefix": "mon metadata", "id": "smithi084"}]: dispatch 2024-02-20T00:09:58.108 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:57 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:09:56.836320+0000 mon.smithi084 (mon.0) 94 : audit [DBG] from='mgr.14162 172.21.15.84:0/2152933216' entity='mgr.smithi084.bmqrnf' cmd=[{"prefix": "mgr metadata", "who": "smithi084.bmqrnf", "id": "smithi084.bmqrnf"}]: dispatch 2024-02-20T00:09:58.108 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:57 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:09:56.837011+0000 mon.smithi084 (mon.0) 95 : audit [DBG] from='mgr.14162 172.21.15.84:0/2152933216' entity='mgr.smithi084.bmqrnf' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-20T00:09:58.108 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:57 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:09:56.837214+0000 mon.smithi084 (mon.0) 96 : audit [DBG] from='mgr.14162 172.21.15.84:0/2152933216' entity='mgr.smithi084.bmqrnf' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-20T00:09:58.108 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:57 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:09:56.837378+0000 mon.smithi084 (mon.0) 97 : audit [DBG] from='mgr.14162 172.21.15.84:0/2152933216' entity='mgr.smithi084.bmqrnf' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-20T00:09:58.108 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:57 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:09:56.851202+0000 mon.smithi084 (mon.0) 98 : cluster [INF] Manager daemon smithi084.bmqrnf is now available 2024-02-20T00:09:58.108 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:57 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:09:56.950093+0000 mon.smithi084 (mon.0) 99 : audit [DBG] from='mgr.14162 172.21.15.84:0/2152933216' entity='mgr.smithi084.bmqrnf' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-20T00:09:58.108 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:57 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:09:56.952419+0000 mon.smithi084 (mon.0) 100 : audit [DBG] from='mgr.14162 172.21.15.84:0/2152933216' entity='mgr.smithi084.bmqrnf' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-20T00:09:58.108 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:57 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:09:56.968041+0000 mon.smithi084 (mon.0) 101 : audit [INF] from='mgr.14162 172.21.15.84:0/2152933216' entity='mgr.smithi084.bmqrnf' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi084.bmqrnf/mirror_snapshot_schedule"}]: dispatch 2024-02-20T00:09:58.108 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:57 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:09:56.970752+0000 mon.smithi084 (mon.0) 102 : audit [INF] from='mgr.14162 172.21.15.84:0/2152933216' entity='mgr.smithi084.bmqrnf' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi084.bmqrnf/trash_purge_schedule"}]: dispatch 2024-02-20T00:09:58.108 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:57 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:09:57.389686+0000 mon.smithi084 (mon.0) 103 : audit [INF] from='mgr.14162 172.21.15.84:0/2152933216' entity='mgr.smithi084.bmqrnf' 2024-02-20T00:09:58.929 INFO:teuthology.orchestra.run.smithi084.stderr:mgr epoch 13 is available 2024-02-20T00:09:58.930 INFO:teuthology.orchestra.run.smithi084.stderr:Generating a dashboard self-signed certificate... 2024-02-20T00:09:58.930 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi084 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/2a2e6c42-cf84-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp3ot6j0xn:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp4xryone4:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 dashboard create-self-signed-cert 2024-02-20T00:09:59.107 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:58 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:09:57.840320+0000 mgr.smithi084.bmqrnf (mgr.14162) 1 : audit [DBG] from='client.14166 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-02-20T00:09:59.107 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:58 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:09:57.840740+0000 mon.smithi084 (mon.0) 104 : cluster [DBG] mgrmap e15: smithi084.bmqrnf(active, since 1.69712s) 2024-02-20T00:09:59.107 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:58 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:09:57.846497+0000 mgr.smithi084.bmqrnf (mgr.14162) 2 : audit [DBG] from='client.14166 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-02-20T00:09:59.108 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:58 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:09:58.646312+0000 mon.smithi084 (mon.0) 105 : audit [INF] from='mgr.14162 172.21.15.84:0/2152933216' entity='mgr.smithi084.bmqrnf' 2024-02-20T00:09:59.728 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:59.638+0000 7f546ffef700 1 Processor -- start 2024-02-20T00:09:59.729 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:59.638+0000 7f546ffef700 1 -- start start 2024-02-20T00:09:59.729 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:59.638+0000 7f546ffef700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f54680703b0 0x7f5468100ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-20T00:09:59.729 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:59.638+0000 7f546ffef700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54680fc8c0 con 0x7f54680703b0 2024-02-20T00:09:59.729 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:59.639+0000 7f546dd8b700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f54680703b0 0x7f5468100ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:09:59.729 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:59.639+0000 7f546dd8b700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f54680703b0 0x7f5468100ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:52520/0 (socket says 172.21.15.84:52520) 2024-02-20T00:09:59.729 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:59.639+0000 7f546dd8b700 1 -- 172.21.15.84:0/1885751941 learned_addr learned my addr 172.21.15.84:0/1885751941 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-02-20T00:09:59.729 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:59.639+0000 7f546dd8b700 1 -- 172.21.15.84:0/1885751941 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54681013c0 con 0x7f54680703b0 2024-02-20T00:09:59.729 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:59.639+0000 7f546dd8b700 1 --2- 172.21.15.84:0/1885751941 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f54680703b0 0x7f5468100ed0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f546400b5f0 tx=0x7f5464006ee0).ready entity=mon.0 client_cookie=c63a42741cdbed5a server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:09:59.729 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:59.639+0000 7f546cd89700 1 -- 172.21.15.84:0/1885751941 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f546400d600 con 0x7f54680703b0 2024-02-20T00:09:59.729 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:59.639+0000 7f546cd89700 1 -- 172.21.15.84:0/1885751941 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f546400dc30 con 0x7f54680703b0 2024-02-20T00:09:59.729 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:59.639+0000 7f546cd89700 1 -- 172.21.15.84:0/1885751941 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5464011d30 con 0x7f54680703b0 2024-02-20T00:09:59.729 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:59.640+0000 7f546ffef700 1 -- 172.21.15.84:0/1885751941 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f54680703b0 msgr2=0x7f5468100ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:09:59.729 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:59.640+0000 7f546ffef700 1 --2- 172.21.15.84:0/1885751941 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f54680703b0 0x7f5468100ed0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f546400b5f0 tx=0x7f5464006ee0).stop 2024-02-20T00:09:59.730 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:59.640+0000 7f546ffef700 1 -- 172.21.15.84:0/1885751941 shutdown_connections 2024-02-20T00:09:59.730 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:59.640+0000 7f546ffef700 1 --2- 172.21.15.84:0/1885751941 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f54680703b0 0x7f5468100ed0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:09:59.730 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:59.640+0000 7f546ffef700 1 -- 172.21.15.84:0/1885751941 >> 172.21.15.84:0/1885751941 conn(0x7f54680f6530 msgr2=0x7f54680f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:09:59.730 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:59.640+0000 7f546ffef700 1 -- 172.21.15.84:0/1885751941 shutdown_connections 2024-02-20T00:09:59.730 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:59.640+0000 7f546ffef700 1 -- 172.21.15.84:0/1885751941 wait complete. 2024-02-20T00:09:59.730 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:59.640+0000 7f546ffef700 1 Processor -- start 2024-02-20T00:09:59.730 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:59.640+0000 7f546ffef700 1 -- start start 2024-02-20T00:09:59.730 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:59.641+0000 7f546ffef700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f54680703b0 0x7f546810cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:09:59.730 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:59.641+0000 7f546ffef700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f546810d290 con 0x7f54680703b0 2024-02-20T00:09:59.730 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:59.641+0000 7f546dd8b700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f54680703b0 0x7f546810cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:09:59.730 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:59.641+0000 7f546dd8b700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f54680703b0 0x7f546810cda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:52534/0 (socket says 172.21.15.84:52534) 2024-02-20T00:09:59.730 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:59.641+0000 7f546dd8b700 1 -- 172.21.15.84:0/962080506 learned_addr learned my addr 172.21.15.84:0/962080506 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-02-20T00:09:59.730 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:59.641+0000 7f546dd8b700 1 -- 172.21.15.84:0/962080506 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f546400b200 con 0x7f54680703b0 2024-02-20T00:09:59.730 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:59.641+0000 7f546dd8b700 1 --2- 172.21.15.84:0/962080506 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f54680703b0 0x7f546810cda0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f546400e040 tx=0x7f5464006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:09:59.730 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:59.641+0000 7f54567fc700 1 -- 172.21.15.84:0/962080506 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f546400da60 con 0x7f54680703b0 2024-02-20T00:09:59.730 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:59.641+0000 7f54567fc700 1 -- 172.21.15.84:0/962080506 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f546401b8e0 con 0x7f54680703b0 2024-02-20T00:09:59.731 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:59.641+0000 7f546ffef700 1 -- 172.21.15.84:0/962080506 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f546810d490 con 0x7f54680703b0 2024-02-20T00:09:59.731 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:59.641+0000 7f54567fc700 1 -- 172.21.15.84:0/962080506 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f546401a900 con 0x7f54680703b0 2024-02-20T00:09:59.731 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:59.641+0000 7f546ffef700 1 -- 172.21.15.84:0/962080506 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f546810e0d0 con 0x7f54680703b0 2024-02-20T00:09:59.731 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:59.642+0000 7f54567fc700 1 -- 172.21.15.84:0/962080506 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 16) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f546400f070 con 0x7f54680703b0 2024-02-20T00:09:59.731 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:59.642+0000 7f54567fc700 1 --2- 172.21.15.84:0/962080506 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f545002ff70 0x7f5450032430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:09:59.731 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:59.642+0000 7f546ffef700 1 -- 172.21.15.84:0/962080506 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f546810d920 con 0x7f54680703b0 2024-02-20T00:09:59.731 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:59.642+0000 7f54567fc700 1 -- 172.21.15.84:0/962080506 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f54640414e0 con 0x7f54680703b0 2024-02-20T00:09:59.731 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:59.643+0000 7f546d58a700 1 --2- 172.21.15.84:0/962080506 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f545002ff70 0x7f5450032430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:09:59.731 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:59.643+0000 7f546d58a700 1 --2- 172.21.15.84:0/962080506 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f545002ff70 0x7f5450032430 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f54580096a0 tx=0x7f5458006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:09:59.731 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:59.647+0000 7f54567fc700 1 -- 172.21.15.84:0/962080506 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5464022960 con 0x7f54680703b0 2024-02-20T00:09:59.816 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:59.815+0000 7f546ffef700 1 -- 172.21.15.84:0/962080506 --> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7f546810e310 con 0x7f545002ff70 2024-02-20T00:09:59.895 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:59.893+0000 7f54567fc700 1 -- 172.21.15.84:0/962080506 <== mgr.14162 v2:172.21.15.84:6800/1303158740 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7f546810e310 con 0x7f545002ff70 2024-02-20T00:09:59.895 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: Self-signed certificate created 2024-02-20T00:09:59.897 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:59.895+0000 7f546ffef700 1 -- 172.21.15.84:0/962080506 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f545002ff70 msgr2=0x7f5450032430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:09:59.897 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:59.895+0000 7f546ffef700 1 --2- 172.21.15.84:0/962080506 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f545002ff70 0x7f5450032430 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f54580096a0 tx=0x7f5458006b40).stop 2024-02-20T00:09:59.897 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:59.895+0000 7f546ffef700 1 -- 172.21.15.84:0/962080506 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f54680703b0 msgr2=0x7f546810cda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:09:59.897 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:59.895+0000 7f546ffef700 1 --2- 172.21.15.84:0/962080506 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f54680703b0 0x7f546810cda0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f546400e040 tx=0x7f5464006ee0).stop 2024-02-20T00:09:59.897 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:59.895+0000 7f546ffef700 1 -- 172.21.15.84:0/962080506 shutdown_connections 2024-02-20T00:09:59.897 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:59.895+0000 7f546ffef700 1 --2- 172.21.15.84:0/962080506 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f545002ff70 0x7f5450032430 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:09:59.897 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:59.895+0000 7f546ffef700 1 --2- 172.21.15.84:0/962080506 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f54680703b0 0x7f546810cda0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:09:59.897 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:59.895+0000 7f546ffef700 1 -- 172.21.15.84:0/962080506 >> 172.21.15.84:0/962080506 conn(0x7f54680f6530 msgr2=0x7f54680f7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:09:59.898 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:59.896+0000 7f546ffef700 1 -- 172.21.15.84:0/962080506 shutdown_connections 2024-02-20T00:09:59.898 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:09:59.896+0000 7f546ffef700 1 -- 172.21.15.84:0/962080506 wait complete. 2024-02-20T00:09:59.915 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:09:59 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:09:58.843407+0000 mon.smithi084 (mon.0) 106 : cluster [DBG] mgrmap e16: smithi084.bmqrnf(active, since 2s) 2024-02-20T00:10:00.565 INFO:teuthology.orchestra.run.smithi084.stderr:Creating initial admin user... 2024-02-20T00:10:00.566 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi084 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/2a2e6c42-cf84-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp3ot6j0xn:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp4xryone4:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp1qwg46_o:/tmp/dashboard.pw:z docker.io/ceph/ceph:v16.2.4 dashboard ac-user-create admin -i /tmp/dashboard.pw administrator --force-password --pwd-update-required 2024-02-20T00:10:01.125 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:10:01.122+0000 7f27eae8a700 1 Processor -- start 2024-02-20T00:10:01.125 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:10:01.122+0000 7f27eae8a700 1 -- start start 2024-02-20T00:10:01.125 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:10:01.123+0000 7f27eae8a700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f27e4070430 0x7f27e40fc840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-20T00:10:01.126 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:10:01.123+0000 7f27eae8a700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27e40fcd30 con 0x7f27e4070430 2024-02-20T00:10:01.126 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:10:01.123+0000 7f27e8c26700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f27e4070430 0x7f27e40fc840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:10:01.126 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:10:01.123+0000 7f27e8c26700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f27e4070430 0x7f27e40fc840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:52548/0 (socket says 172.21.15.84:52548) 2024-02-20T00:10:01.126 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:10:01.123+0000 7f27e8c26700 1 -- 172.21.15.84:0/3422404592 learned_addr learned my addr 172.21.15.84:0/3422404592 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-02-20T00:10:01.128 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:10:01.123+0000 7f27e8c26700 1 -- 172.21.15.84:0/3422404592 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27e40fce70 con 0x7f27e4070430 2024-02-20T00:10:01.129 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:10:01.123+0000 7f27e8c26700 1 --2- 172.21.15.84:0/3422404592 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f27e4070430 0x7f27e40fc840 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f27d800b5f0 tx=0x7f27d8006ee0).ready entity=mon.0 client_cookie=795df085a73ddd79 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:10:01.129 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:10:01.123+0000 7f27e37fe700 1 -- 172.21.15.84:0/3422404592 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f27d800d600 con 0x7f27e4070430 2024-02-20T00:10:01.129 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:10:01.123+0000 7f27e37fe700 1 -- 172.21.15.84:0/3422404592 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f27d800dc30 con 0x7f27e4070430 2024-02-20T00:10:01.129 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:10:01.123+0000 7f27e37fe700 1 -- 172.21.15.84:0/3422404592 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f27d8011d30 con 0x7f27e4070430 2024-02-20T00:10:01.129 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:10:01.124+0000 7f27eae8a700 1 -- 172.21.15.84:0/3422404592 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f27e4070430 msgr2=0x7f27e40fc840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:10:01.129 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:10:01.124+0000 7f27eae8a700 1 --2- 172.21.15.84:0/3422404592 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f27e4070430 0x7f27e40fc840 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f27d800b5f0 tx=0x7f27d8006ee0).stop 2024-02-20T00:10:01.129 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:10:01.124+0000 7f27eae8a700 1 -- 172.21.15.84:0/3422404592 shutdown_connections 2024-02-20T00:10:01.129 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:10:01.124+0000 7f27eae8a700 1 --2- 172.21.15.84:0/3422404592 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f27e4070430 0x7f27e40fc840 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:10:01.129 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:10:01.124+0000 7f27eae8a700 1 -- 172.21.15.84:0/3422404592 >> 172.21.15.84:0/3422404592 conn(0x7f27e40f6530 msgr2=0x7f27e40f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:10:01.129 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:10:01.124+0000 7f27eae8a700 1 -- 172.21.15.84:0/3422404592 shutdown_connections 2024-02-20T00:10:01.129 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:10:01.124+0000 7f27eae8a700 1 -- 172.21.15.84:0/3422404592 wait complete. 2024-02-20T00:10:01.129 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:10:01.125+0000 7f27eae8a700 1 Processor -- start 2024-02-20T00:10:01.129 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:10:01.125+0000 7f27eae8a700 1 -- start start 2024-02-20T00:10:01.130 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:10:01.125+0000 7f27eae8a700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f27e4070430 0x7f27e41111e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:10:01.130 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:10:01.125+0000 7f27eae8a700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27e41116d0 con 0x7f27e4070430 2024-02-20T00:10:01.130 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:10:01.125+0000 7f27e8c26700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f27e4070430 0x7f27e41111e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:10:01.130 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:10:01.125+0000 7f27e8c26700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f27e4070430 0x7f27e41111e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:52550/0 (socket says 172.21.15.84:52550) 2024-02-20T00:10:01.130 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:10:01.125+0000 7f27e8c26700 1 -- 172.21.15.84:0/2952958936 learned_addr learned my addr 172.21.15.84:0/2952958936 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-02-20T00:10:01.130 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:10:01.125+0000 7f27e8c26700 1 -- 172.21.15.84:0/2952958936 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27d800b200 con 0x7f27e4070430 2024-02-20T00:10:01.130 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:10:01.125+0000 7f27e8c26700 1 --2- 172.21.15.84:0/2952958936 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f27e4070430 0x7f27e41111e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f27d800e040 tx=0x7f27d8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:10:01.130 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:10:01.126+0000 7f27e17fa700 1 -- 172.21.15.84:0/2952958936 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f27d800d9e0 con 0x7f27e4070430 2024-02-20T00:10:01.130 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:10:01.126+0000 7f27e17fa700 1 -- 172.21.15.84:0/2952958936 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f27d801b8e0 con 0x7f27e4070430 2024-02-20T00:10:01.130 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:10:01.126+0000 7f27eae8a700 1 -- 172.21.15.84:0/2952958936 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27e41118d0 con 0x7f27e4070430 2024-02-20T00:10:01.130 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:10:01.126+0000 7f27e17fa700 1 -- 172.21.15.84:0/2952958936 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f27d801a820 con 0x7f27e4070430 2024-02-20T00:10:01.130 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:10:01.126+0000 7f27eae8a700 1 -- 172.21.15.84:0/2952958936 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27e4111d60 con 0x7f27e4070430 2024-02-20T00:10:01.130 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:10:01.126+0000 7f27e17fa700 1 -- 172.21.15.84:0/2952958936 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 16) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f27d800f070 con 0x7f27e4070430 2024-02-20T00:10:01.130 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:10:01.127+0000 7f27e17fa700 1 --2- 172.21.15.84:0/2952958936 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f27c802ff70 0x7f27c8032430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:10:01.130 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:10:01.127+0000 7f27e17fa700 1 -- 172.21.15.84:0/2952958936 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f27d80415f0 con 0x7f27e4070430 2024-02-20T00:10:01.131 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:10:01.127+0000 7f27eae8a700 1 -- 172.21.15.84:0/2952958936 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27e4045ad0 con 0x7f27e4070430 2024-02-20T00:10:01.132 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:10:01.127+0000 7f27e3fff700 1 --2- 172.21.15.84:0/2952958936 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f27c802ff70 0x7f27c8032430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:10:01.132 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:10:01.127+0000 7f27e3fff700 1 --2- 172.21.15.84:0/2952958936 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f27c802ff70 0x7f27c8032430 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f27d00096a0 tx=0x7f27d0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:10:01.132 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:10:01.131+0000 7f27e17fa700 1 -- 172.21.15.84:0/2952958936 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f27d8016070 con 0x7f27e4070430 2024-02-20T00:10:01.299 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:10:01.298+0000 7f27eae8a700 1 -- 172.21.15.84:0/2952958936 --> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f27e4058f20 con 0x7f27c802ff70 2024-02-20T00:10:01.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:10:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:09:59.815955+0000 mgr.smithi084.bmqrnf (mgr.14162) 3 : audit [DBG] from='client.14172 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T00:10:01.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:10:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:09:59.892009+0000 mon.smithi084 (mon.0) 107 : audit [INF] from='mgr.14162 172.21.15.84:0/2152933216' entity='mgr.smithi084.bmqrnf' 2024-02-20T00:10:01.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:10:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:09:59.893707+0000 mon.smithi084 (mon.0) 108 : audit [INF] from='mgr.14162 172.21.15.84:0/2152933216' entity='mgr.smithi084.bmqrnf' 2024-02-20T00:10:01.520 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:10:01.518+0000 7f27e17fa700 1 -- 172.21.15.84:0/2952958936 <== mgr.14162 v2:172.21.15.84:6800/1303158740 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7f27e4058f20 con 0x7f27c802ff70 2024-02-20T00:10:01.520 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: {"username": "admin", "password": "$2b$12$2o1hAKhpuHFJgIyH7.Ez8e6HWELOWq2uP//FNodBT5Lm20LSwl4im", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1708387801, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-02-20T00:10:01.522 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:10:01.520+0000 7f27eae8a700 1 -- 172.21.15.84:0/2952958936 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f27c802ff70 msgr2=0x7f27c8032430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:10:01.522 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:10:01.520+0000 7f27eae8a700 1 --2- 172.21.15.84:0/2952958936 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f27c802ff70 0x7f27c8032430 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f27d00096a0 tx=0x7f27d0006b40).stop 2024-02-20T00:10:01.522 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:10:01.520+0000 7f27eae8a700 1 -- 172.21.15.84:0/2952958936 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f27e4070430 msgr2=0x7f27e41111e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:10:01.523 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:10:01.520+0000 7f27eae8a700 1 --2- 172.21.15.84:0/2952958936 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f27e4070430 0x7f27e41111e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f27d800e040 tx=0x7f27d8006ee0).stop 2024-02-20T00:10:01.523 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:10:01.520+0000 7f27eae8a700 1 -- 172.21.15.84:0/2952958936 shutdown_connections 2024-02-20T00:10:01.523 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:10:01.520+0000 7f27eae8a700 1 --2- 172.21.15.84:0/2952958936 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f27c802ff70 0x7f27c8032430 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:10:01.523 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:10:01.520+0000 7f27eae8a700 1 --2- 172.21.15.84:0/2952958936 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f27e4070430 0x7f27e41111e0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:10:01.523 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:10:01.520+0000 7f27eae8a700 1 -- 172.21.15.84:0/2952958936 >> 172.21.15.84:0/2952958936 conn(0x7f27e40f6530 msgr2=0x7f27e40f7f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:10:01.523 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:10:01.520+0000 7f27eae8a700 1 -- 172.21.15.84:0/2952958936 shutdown_connections 2024-02-20T00:10:01.523 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:10:01.520+0000 7f27eae8a700 1 -- 172.21.15.84:0/2952958936 wait complete. 2024-02-20T00:10:02.208 INFO:teuthology.orchestra.run.smithi084.stderr:Fetching dashboard port number... 2024-02-20T00:10:02.209 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi084 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/2a2e6c42-cf84-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp3ot6j0xn:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp4xryone4:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config get mgr mgr/dashboard/ssl_server_port 2024-02-20T00:10:02.830 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:10:02.787+0000 7f714759e700 1 Processor -- start 2024-02-20T00:10:02.830 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:10:02.787+0000 7f714759e700 1 -- start start 2024-02-20T00:10:02.830 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:10:02.787+0000 7f714759e700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7148070370 0x7f71480fe950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-20T00:10:02.830 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:10:02.787+0000 7f714759e700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7148070790 con 0x7f7148070370 2024-02-20T00:10:02.830 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:10:02.788+0000 7f714659c700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7148070370 0x7f71480fe950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:10:02.830 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:10:02.788+0000 7f714659c700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7148070370 0x7f71480fe950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:52566/0 (socket says 172.21.15.84:52566) 2024-02-20T00:10:02.830 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:10:02.788+0000 7f714659c700 1 -- 172.21.15.84:0/273140697 learned_addr learned my addr 172.21.15.84:0/273140697 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-02-20T00:10:02.831 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:10:02.788+0000 7f714659c700 1 -- 172.21.15.84:0/273140697 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71480fee40 con 0x7f7148070370 2024-02-20T00:10:02.831 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:10:02.788+0000 7f714659c700 1 --2- 172.21.15.84:0/273140697 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7148070370 0x7f71480fe950 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f713c00b5f0 tx=0x7f713c006ee0).ready entity=mon.0 client_cookie=973ea062442f2aa8 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:10:02.831 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:10:02.788+0000 7f714559a700 1 -- 172.21.15.84:0/273140697 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f713c00d600 con 0x7f7148070370 2024-02-20T00:10:02.831 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:10:02.788+0000 7f714559a700 1 -- 172.21.15.84:0/273140697 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f713c00dc30 con 0x7f7148070370 2024-02-20T00:10:02.831 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:10:02.788+0000 7f714559a700 1 -- 172.21.15.84:0/273140697 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f713c011d30 con 0x7f7148070370 2024-02-20T00:10:02.831 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:10:02.789+0000 7f714759e700 1 -- 172.21.15.84:0/273140697 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7148070370 msgr2=0x7f71480fe950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:10:02.831 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:10:02.789+0000 7f714759e700 1 --2- 172.21.15.84:0/273140697 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7148070370 0x7f71480fe950 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f713c00b5f0 tx=0x7f713c006ee0).stop 2024-02-20T00:10:02.831 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:10:02.789+0000 7f714759e700 1 -- 172.21.15.84:0/273140697 shutdown_connections 2024-02-20T00:10:02.831 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:10:02.789+0000 7f714759e700 1 --2- 172.21.15.84:0/273140697 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7148070370 0x7f71480fe950 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:10:02.831 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:10:02.789+0000 7f714759e700 1 -- 172.21.15.84:0/273140697 >> 172.21.15.84:0/273140697 conn(0x7f71480f6390 msgr2=0x7f71480f87b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:10:02.831 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:10:02.789+0000 7f714759e700 1 -- 172.21.15.84:0/273140697 shutdown_connections 2024-02-20T00:10:02.831 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:10:02.789+0000 7f714759e700 1 -- 172.21.15.84:0/273140697 wait complete. 2024-02-20T00:10:02.832 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:10:02.789+0000 7f714759e700 1 Processor -- start 2024-02-20T00:10:02.832 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:10:02.789+0000 7f714759e700 1 -- start start 2024-02-20T00:10:02.832 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:10:02.789+0000 7f714759e700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7148070370 0x7f714810a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:10:02.832 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:10:02.789+0000 7f714759e700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f714810aeb0 con 0x7f7148070370 2024-02-20T00:10:02.832 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:10:02.789+0000 7f714659c700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7148070370 0x7f714810a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:10:02.832 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:10:02.789+0000 7f714659c700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7148070370 0x7f714810a9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:52568/0 (socket says 172.21.15.84:52568) 2024-02-20T00:10:02.832 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:10:02.789+0000 7f714659c700 1 -- 172.21.15.84:0/2350302709 learned_addr learned my addr 172.21.15.84:0/2350302709 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-02-20T00:10:02.832 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:10:02.790+0000 7f714659c700 1 -- 172.21.15.84:0/2350302709 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f713c00b200 con 0x7f7148070370 2024-02-20T00:10:02.832 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:10:02.790+0000 7f714659c700 1 --2- 172.21.15.84:0/2350302709 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7148070370 0x7f714810a9c0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f713c00e040 tx=0x7f713c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:10:02.832 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:10:02.790+0000 7f7136ffd700 1 -- 172.21.15.84:0/2350302709 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f713c00da60 con 0x7f7148070370 2024-02-20T00:10:02.832 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:10:02.790+0000 7f7136ffd700 1 -- 172.21.15.84:0/2350302709 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f713c01b8e0 con 0x7f7148070370 2024-02-20T00:10:02.832 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:10:02.790+0000 7f714759e700 1 -- 172.21.15.84:0/2350302709 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f714810b0b0 con 0x7f7148070370 2024-02-20T00:10:02.832 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:10:02.790+0000 7f7136ffd700 1 -- 172.21.15.84:0/2350302709 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f713c01a900 con 0x7f7148070370 2024-02-20T00:10:02.832 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:10:02.790+0000 7f714759e700 1 -- 172.21.15.84:0/2350302709 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f714810b540 con 0x7f7148070370 2024-02-20T00:10:02.833 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:10:02.791+0000 7f7136ffd700 1 -- 172.21.15.84:0/2350302709 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f713c00f070 con 0x7f7148070370 2024-02-20T00:10:02.833 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:10:02.791+0000 7f7136ffd700 1 --2- 172.21.15.84:0/2350302709 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f712c030070 0x7f712c032530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:10:02.833 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:10:02.791+0000 7f7136ffd700 1 -- 172.21.15.84:0/2350302709 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f713c0425e0 con 0x7f7148070370 2024-02-20T00:10:02.833 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:10:02.791+0000 7f7145d9b700 1 --2- 172.21.15.84:0/2350302709 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f712c030070 0x7f712c032530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:10:02.833 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:10:02.791+0000 7f714759e700 1 -- 172.21.15.84:0/2350302709 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7148045ad0 con 0x7f7148070370 2024-02-20T00:10:02.833 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:10:02.791+0000 7f7145d9b700 1 --2- 172.21.15.84:0/2350302709 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f712c030070 0x7f712c032530 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f71380096a0 tx=0x7f7138006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:10:02.833 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:10:02.795+0000 7f7136ffd700 1 -- 172.21.15.84:0/2350302709 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f713c010360 con 0x7f7148070370 2024-02-20T00:10:02.857 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:10:02 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:10:01.299291+0000 mgr.smithi084.bmqrnf (mgr.14162) 4 : audit [DBG] from='client.14174 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T00:10:02.857 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:10:02 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:10:01.519049+0000 mon.smithi084 (mon.0) 109 : audit [INF] from='mgr.14162 172.21.15.84:0/2152933216' entity='mgr.smithi084.bmqrnf' 2024-02-20T00:10:02.951 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:10:02.949+0000 7f714759e700 1 -- 172.21.15.84:0/2350302709 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7f71480ffcb0 con 0x7f7148070370 2024-02-20T00:10:02.951 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:10:02.950+0000 7f7136ffd700 1 -- 172.21.15.84:0/2350302709 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v10) v1 ==== 112+0+5 (secure 0 0 0) 0x7f713c022ac0 con 0x7f7148070370 2024-02-20T00:10:02.951 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 8443 2024-02-20T00:10:02.953 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:10:02.951+0000 7f714759e700 1 -- 172.21.15.84:0/2350302709 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f712c030070 msgr2=0x7f712c032530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:10:02.954 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:10:02.951+0000 7f714759e700 1 --2- 172.21.15.84:0/2350302709 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f712c030070 0x7f712c032530 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f71380096a0 tx=0x7f7138006b40).stop 2024-02-20T00:10:02.954 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:10:02.951+0000 7f714759e700 1 -- 172.21.15.84:0/2350302709 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7148070370 msgr2=0x7f714810a9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:10:02.954 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:10:02.951+0000 7f714759e700 1 --2- 172.21.15.84:0/2350302709 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7148070370 0x7f714810a9c0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f713c00e040 tx=0x7f713c006ee0).stop 2024-02-20T00:10:02.954 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:10:02.951+0000 7f714759e700 1 -- 172.21.15.84:0/2350302709 shutdown_connections 2024-02-20T00:10:02.954 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:10:02.951+0000 7f714759e700 1 --2- 172.21.15.84:0/2350302709 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f712c030070 0x7f712c032530 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:10:02.954 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:10:02.951+0000 7f714759e700 1 --2- 172.21.15.84:0/2350302709 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7148070370 0x7f714810a9c0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:10:02.954 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:10:02.951+0000 7f714759e700 1 -- 172.21.15.84:0/2350302709 >> 172.21.15.84:0/2350302709 conn(0x7f71480f6390 msgr2=0x7f71480f7de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:10:02.954 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:10:02.952+0000 7f714759e700 1 -- 172.21.15.84:0/2350302709 shutdown_connections 2024-02-20T00:10:02.954 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-02-20T00:10:02.952+0000 7f714759e700 1 -- 172.21.15.84:0/2350302709 wait complete. 2024-02-20T00:10:03.715 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: systemctl is-enabled firewalld.service 2024-02-20T00:10:03.718 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:10:03 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:10:02.520162+0000 mon.smithi084 (mon.0) 110 : cluster [DBG] mgrmap e17: smithi084.bmqrnf(active, since 6s) 2024-02-20T00:10:03.718 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:10:03 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:10:02.950586+0000 mon.smithi084 (mon.0) 111 : audit [DBG] from='client.? 172.21.15.84:0/2350302709' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-02-20T00:10:03.724 INFO:teuthology.orchestra.run.smithi084.stderr:systemctl: disabled 2024-02-20T00:10:03.727 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: systemctl is-active firewalld.service 2024-02-20T00:10:03.732 INFO:teuthology.orchestra.run.smithi084.stderr:systemctl: inactive 2024-02-20T00:10:03.732 INFO:teuthology.orchestra.run.smithi084.stderr:firewalld.service is not enabled 2024-02-20T00:10:03.732 INFO:teuthology.orchestra.run.smithi084.stderr:Not possible to open ports <[8443]>. firewalld.service is not available 2024-02-20T00:10:03.735 INFO:teuthology.orchestra.run.smithi084.stderr:Ceph Dashboard is now available at: 2024-02-20T00:10:03.735 INFO:teuthology.orchestra.run.smithi084.stderr: 2024-02-20T00:10:03.735 INFO:teuthology.orchestra.run.smithi084.stderr: URL: https://smithi084.front.sepia.ceph.com:8443/ 2024-02-20T00:10:03.735 INFO:teuthology.orchestra.run.smithi084.stderr: User: admin 2024-02-20T00:10:03.735 INFO:teuthology.orchestra.run.smithi084.stderr: Password: 5dm7y2tmjn 2024-02-20T00:10:03.735 INFO:teuthology.orchestra.run.smithi084.stderr: 2024-02-20T00:10:03.736 INFO:teuthology.orchestra.run.smithi084.stderr:You can access the Ceph CLI with: 2024-02-20T00:10:03.736 INFO:teuthology.orchestra.run.smithi084.stderr: 2024-02-20T00:10:03.736 INFO:teuthology.orchestra.run.smithi084.stderr: sudo /home/ubuntu/cephtest/cephadm shell --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-02-20T00:10:03.736 INFO:teuthology.orchestra.run.smithi084.stderr: 2024-02-20T00:10:03.736 INFO:teuthology.orchestra.run.smithi084.stderr:Please consider enabling telemetry to help improve Ceph: 2024-02-20T00:10:03.737 INFO:teuthology.orchestra.run.smithi084.stderr: 2024-02-20T00:10:03.737 INFO:teuthology.orchestra.run.smithi084.stderr: ceph telemetry on 2024-02-20T00:10:03.737 INFO:teuthology.orchestra.run.smithi084.stderr: 2024-02-20T00:10:03.737 INFO:teuthology.orchestra.run.smithi084.stderr:For more information see: 2024-02-20T00:10:03.737 INFO:teuthology.orchestra.run.smithi084.stderr: 2024-02-20T00:10:03.737 INFO:teuthology.orchestra.run.smithi084.stderr: https://docs.ceph.com/docs/pacific/mgr/telemetry/ 2024-02-20T00:10:03.737 INFO:teuthology.orchestra.run.smithi084.stderr: 2024-02-20T00:10:03.737 INFO:teuthology.orchestra.run.smithi084.stderr:Bootstrap complete. 2024-02-20T00:10:03.737 INFO:teuthology.orchestra.run.smithi084.stderr:Releasing lock 140012450322920 on /run/cephadm/2a2e6c42-cf84-11ee-95bb-87774f69a715.lock 2024-02-20T00:10:03.737 INFO:teuthology.orchestra.run.smithi084.stderr:Lock 140012450322920 released on /run/cephadm/2a2e6c42-cf84-11ee-95bb-87774f69a715.lock 2024-02-20T00:10:03.768 INFO:tasks.cephadm:Fetching config... 2024-02-20T00:10:03.768 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-02-20T00:10:03.768 DEBUG:teuthology.orchestra.run.smithi084:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-02-20T00:10:03.826 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-02-20T00:10:03.826 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-02-20T00:10:03.826 DEBUG:teuthology.orchestra.run.smithi084:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-02-20T00:10:03.884 INFO:tasks.cephadm:Fetching mon keyring... 2024-02-20T00:10:03.884 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-02-20T00:10:03.884 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd if=/var/lib/ceph/2a2e6c42-cf84-11ee-95bb-87774f69a715/mon.smithi084/keyring of=/dev/stdout 2024-02-20T00:10:03.951 INFO:tasks.cephadm:Fetching pub ssh key... 2024-02-20T00:10:03.952 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-02-20T00:10:03.952 DEBUG:teuthology.orchestra.run.smithi084:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-02-20T00:10:04.010 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-02-20T00:10:04.010 DEBUG:teuthology.orchestra.run.smithi084:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDVfsDtzKDpVOBeHn4d4EDiVlt4RcYaogldSPa5IeJCrgeX2uk1+0iHAMxvA2iaYvP1Ib8GH/a/8rjr3LfwcC8MRIxfZs0Vk/r1K133c0wlaJ7Hb3g9VX8kbNcnhYiJ+flthCyymvuTrjuB2lhvIAVR0mdTmuyCLk/nLzrsCW+E98KxlWI3LMvGaf1pQvyLlNTFcf8cwgE6pMcBaK0GkntheJN/viF6yjGTgKtK2yzr/Bzu6mYT/uciD77xVVIfnMN0HPFS17x/urKM3XRkMrLqyttRivF061sSImTr78B7UmpNCideVF0gf2akgbtBonIGBOfm1Nkici/Jn7/k/3NF3fz2r1QiDg8JYLLgO0yE7InGa8Mu6M5RjW4OsRzLLum+gV+Jh1tu/0pU/P+ogq+IdvfhmEwMcNL4d7ghTb1SNybhSb1jL2ai/MA/VzuaMWQi7k1BCE1nZGF3GCsYSxf++CxZQViR9mgOV3tUeHOMI917bi2fAX5Pyp+0pYbhij8= ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-02-20T00:10:04.089 INFO:teuthology.orchestra.run.smithi084.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDVfsDtzKDpVOBeHn4d4EDiVlt4RcYaogldSPa5IeJCrgeX2uk1+0iHAMxvA2iaYvP1Ib8GH/a/8rjr3LfwcC8MRIxfZs0Vk/r1K133c0wlaJ7Hb3g9VX8kbNcnhYiJ+flthCyymvuTrjuB2lhvIAVR0mdTmuyCLk/nLzrsCW+E98KxlWI3LMvGaf1pQvyLlNTFcf8cwgE6pMcBaK0GkntheJN/viF6yjGTgKtK2yzr/Bzu6mYT/uciD77xVVIfnMN0HPFS17x/urKM3XRkMrLqyttRivF061sSImTr78B7UmpNCideVF0gf2akgbtBonIGBOfm1Nkici/Jn7/k/3NF3fz2r1QiDg8JYLLgO0yE7InGa8Mu6M5RjW4OsRzLLum+gV+Jh1tu/0pU/P+ogq+IdvfhmEwMcNL4d7ghTb1SNybhSb1jL2ai/MA/VzuaMWQi7k1BCE1nZGF3GCsYSxf++CxZQViR9mgOV3tUeHOMI917bi2fAX5Pyp+0pYbhij8= ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715 2024-02-20T00:10:04.100 DEBUG:teuthology.orchestra.run.smithi132:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDVfsDtzKDpVOBeHn4d4EDiVlt4RcYaogldSPa5IeJCrgeX2uk1+0iHAMxvA2iaYvP1Ib8GH/a/8rjr3LfwcC8MRIxfZs0Vk/r1K133c0wlaJ7Hb3g9VX8kbNcnhYiJ+flthCyymvuTrjuB2lhvIAVR0mdTmuyCLk/nLzrsCW+E98KxlWI3LMvGaf1pQvyLlNTFcf8cwgE6pMcBaK0GkntheJN/viF6yjGTgKtK2yzr/Bzu6mYT/uciD77xVVIfnMN0HPFS17x/urKM3XRkMrLqyttRivF061sSImTr78B7UmpNCideVF0gf2akgbtBonIGBOfm1Nkici/Jn7/k/3NF3fz2r1QiDg8JYLLgO0yE7InGa8Mu6M5RjW4OsRzLLum+gV+Jh1tu/0pU/P+ogq+IdvfhmEwMcNL4d7ghTb1SNybhSb1jL2ai/MA/VzuaMWQi7k1BCE1nZGF3GCsYSxf++CxZQViR9mgOV3tUeHOMI917bi2fAX5Pyp+0pYbhij8= ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-02-20T00:10:04.147 INFO:teuthology.orchestra.run.smithi132.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDVfsDtzKDpVOBeHn4d4EDiVlt4RcYaogldSPa5IeJCrgeX2uk1+0iHAMxvA2iaYvP1Ib8GH/a/8rjr3LfwcC8MRIxfZs0Vk/r1K133c0wlaJ7Hb3g9VX8kbNcnhYiJ+flthCyymvuTrjuB2lhvIAVR0mdTmuyCLk/nLzrsCW+E98KxlWI3LMvGaf1pQvyLlNTFcf8cwgE6pMcBaK0GkntheJN/viF6yjGTgKtK2yzr/Bzu6mYT/uciD77xVVIfnMN0HPFS17x/urKM3XRkMrLqyttRivF061sSImTr78B7UmpNCideVF0gf2akgbtBonIGBOfm1Nkici/Jn7/k/3NF3fz2r1QiDg8JYLLgO0yE7InGa8Mu6M5RjW4OsRzLLum+gV+Jh1tu/0pU/P+ogq+IdvfhmEwMcNL4d7ghTb1SNybhSb1jL2ai/MA/VzuaMWQi7k1BCE1nZGF3GCsYSxf++CxZQViR9mgOV3tUeHOMI917bi2fAX5Pyp+0pYbhij8= ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715 2024-02-20T00:10:04.159 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-02-20T00:10:06.809 INFO:teuthology.orchestra.run.smithi084.stderr:2024-02-20T00:10:06.804+0000 7f4da94a6700 1 -- 172.21.15.84:0/1156124139 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4da40f6510 msgr2=0x7f4da40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:10:06.809 INFO:teuthology.orchestra.run.smithi084.stderr:2024-02-20T00:10:06.804+0000 7f4da94a6700 1 --2- 172.21.15.84:0/1156124139 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4da40f6510 0x7f4da40f6930 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f4d98009fb0 tx=0x7f4d98009b30).stop 2024-02-20T00:10:06.809 INFO:teuthology.orchestra.run.smithi084.stderr:2024-02-20T00:10:06.804+0000 7f4da94a6700 1 -- 172.21.15.84:0/1156124139 shutdown_connections 2024-02-20T00:10:06.809 INFO:teuthology.orchestra.run.smithi084.stderr:2024-02-20T00:10:06.804+0000 7f4da94a6700 1 --2- 172.21.15.84:0/1156124139 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4da40f6510 0x7f4da40f6930 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:10:06.809 INFO:teuthology.orchestra.run.smithi084.stderr:2024-02-20T00:10:06.804+0000 7f4da94a6700 1 -- 172.21.15.84:0/1156124139 >> 172.21.15.84:0/1156124139 conn(0x7f4da40f1c70 msgr2=0x7f4da40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:10:06.809 INFO:teuthology.orchestra.run.smithi084.stderr:2024-02-20T00:10:06.804+0000 7f4da94a6700 1 -- 172.21.15.84:0/1156124139 shutdown_connections 2024-02-20T00:10:06.809 INFO:teuthology.orchestra.run.smithi084.stderr:2024-02-20T00:10:06.804+0000 7f4da94a6700 1 -- 172.21.15.84:0/1156124139 wait complete. 2024-02-20T00:10:06.809 INFO:teuthology.orchestra.run.smithi084.stderr:2024-02-20T00:10:06.805+0000 7f4da94a6700 1 Processor -- start 2024-02-20T00:10:06.809 INFO:teuthology.orchestra.run.smithi084.stderr:2024-02-20T00:10:06.805+0000 7f4da94a6700 1 -- start start 2024-02-20T00:10:06.809 INFO:teuthology.orchestra.run.smithi084.stderr:2024-02-20T00:10:06.805+0000 7f4da94a6700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4da40f6510 0x7f4da41067b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:10:06.810 INFO:teuthology.orchestra.run.smithi084.stderr:2024-02-20T00:10:06.805+0000 7f4da94a6700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4da4106ca0 con 0x7f4da40f6510 2024-02-20T00:10:06.810 INFO:teuthology.orchestra.run.smithi084.stderr:2024-02-20T00:10:06.805+0000 7f4da2ffd700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4da40f6510 0x7f4da41067b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:10:06.810 INFO:teuthology.orchestra.run.smithi084.stderr:2024-02-20T00:10:06.805+0000 7f4da2ffd700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4da40f6510 0x7f4da41067b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:52584/0 (socket says 172.21.15.84:52584) 2024-02-20T00:10:06.810 INFO:teuthology.orchestra.run.smithi084.stderr:2024-02-20T00:10:06.805+0000 7f4da2ffd700 1 -- 172.21.15.84:0/1602242889 learned_addr learned my addr 172.21.15.84:0/1602242889 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-02-20T00:10:06.810 INFO:teuthology.orchestra.run.smithi084.stderr:2024-02-20T00:10:06.806+0000 7f4da2ffd700 1 -- 172.21.15.84:0/1602242889 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d98005040 con 0x7f4da40f6510 2024-02-20T00:10:06.810 INFO:teuthology.orchestra.run.smithi084.stderr:2024-02-20T00:10:06.806+0000 7f4da2ffd700 1 --2- 172.21.15.84:0/1602242889 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4da40f6510 0x7f4da41067b0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f4d98005010 tx=0x7f4d9800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:10:06.810 INFO:teuthology.orchestra.run.smithi084.stderr:2024-02-20T00:10:06.806+0000 7f4d8b7fe700 1 -- 172.21.15.84:0/1602242889 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4d98009320 con 0x7f4da40f6510 2024-02-20T00:10:06.810 INFO:teuthology.orchestra.run.smithi084.stderr:2024-02-20T00:10:06.806+0000 7f4d8b7fe700 1 -- 172.21.15.84:0/1602242889 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4d98013380 con 0x7f4da40f6510 2024-02-20T00:10:06.810 INFO:teuthology.orchestra.run.smithi084.stderr:2024-02-20T00:10:06.806+0000 7f4da94a6700 1 -- 172.21.15.84:0/1602242889 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4da4106ea0 con 0x7f4da40f6510 2024-02-20T00:10:06.810 INFO:teuthology.orchestra.run.smithi084.stderr:2024-02-20T00:10:06.806+0000 7f4d8b7fe700 1 -- 172.21.15.84:0/1602242889 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4d98014e50 con 0x7f4da40f6510 2024-02-20T00:10:06.810 INFO:teuthology.orchestra.run.smithi084.stderr:2024-02-20T00:10:06.806+0000 7f4da94a6700 1 -- 172.21.15.84:0/1602242889 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4da4107330 con 0x7f4da40f6510 2024-02-20T00:10:06.811 INFO:teuthology.orchestra.run.smithi084.stderr:2024-02-20T00:10:06.807+0000 7f4da94a6700 1 -- 172.21.15.84:0/1602242889 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4da4045ad0 con 0x7f4da40f6510 2024-02-20T00:10:06.811 INFO:teuthology.orchestra.run.smithi084.stderr:2024-02-20T00:10:06.807+0000 7f4d8b7fe700 1 -- 172.21.15.84:0/1602242889 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4d980095f0 con 0x7f4da40f6510 2024-02-20T00:10:06.811 INFO:teuthology.orchestra.run.smithi084.stderr:2024-02-20T00:10:06.807+0000 7f4d8b7fe700 1 --2- 172.21.15.84:0/1602242889 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f4d8c02ff10 0x7f4d8c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:10:06.812 INFO:teuthology.orchestra.run.smithi084.stderr:2024-02-20T00:10:06.807+0000 7f4d8b7fe700 1 -- 172.21.15.84:0/1602242889 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4d98040f60 con 0x7f4da40f6510 2024-02-20T00:10:06.812 INFO:teuthology.orchestra.run.smithi084.stderr:2024-02-20T00:10:06.807+0000 7f4da27fc700 1 --2- 172.21.15.84:0/1602242889 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f4d8c02ff10 0x7f4d8c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:10:06.812 INFO:teuthology.orchestra.run.smithi084.stderr:2024-02-20T00:10:06.808+0000 7f4da27fc700 1 --2- 172.21.15.84:0/1602242889 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f4d8c02ff10 0x7f4d8c0323d0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f4d940096a0 tx=0x7f4d94006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:10:06.813 INFO:teuthology.orchestra.run.smithi084.stderr:2024-02-20T00:10:06.811+0000 7f4d8b7fe700 1 -- 172.21.15.84:0/1602242889 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4d980672b0 con 0x7f4da40f6510 2024-02-20T00:10:07.001 INFO:teuthology.orchestra.run.smithi084.stderr:2024-02-20T00:10:06.998+0000 7f4da94a6700 1 -- 172.21.15.84:0/1602242889 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7f4da40fc0a0 con 0x7f4da40f6510 2024-02-20T00:10:07.004 INFO:teuthology.orchestra.run.smithi084.stderr:2024-02-20T00:10:07.002+0000 7f4d8b7fe700 1 -- 172.21.15.84:0/1602242889 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v11)=0 v11) v1 ==== 125+0+0 (secure 0 0 0) 0x7f4d98014420 con 0x7f4da40f6510 2024-02-20T00:10:07.008 INFO:teuthology.orchestra.run.smithi084.stderr:2024-02-20T00:10:07.003+0000 7f4da94a6700 1 -- 172.21.15.84:0/1602242889 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f4d8c02ff10 msgr2=0x7f4d8c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:10:07.008 INFO:teuthology.orchestra.run.smithi084.stderr:2024-02-20T00:10:07.003+0000 7f4da94a6700 1 --2- 172.21.15.84:0/1602242889 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f4d8c02ff10 0x7f4d8c0323d0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f4d940096a0 tx=0x7f4d94006b40).stop 2024-02-20T00:10:07.008 INFO:teuthology.orchestra.run.smithi084.stderr:2024-02-20T00:10:07.003+0000 7f4da94a6700 1 -- 172.21.15.84:0/1602242889 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4da40f6510 msgr2=0x7f4da41067b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:10:07.008 INFO:teuthology.orchestra.run.smithi084.stderr:2024-02-20T00:10:07.003+0000 7f4da94a6700 1 --2- 172.21.15.84:0/1602242889 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4da40f6510 0x7f4da41067b0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f4d98005010 tx=0x7f4d9800b040).stop 2024-02-20T00:10:07.008 INFO:teuthology.orchestra.run.smithi084.stderr:2024-02-20T00:10:07.003+0000 7f4da94a6700 1 -- 172.21.15.84:0/1602242889 shutdown_connections 2024-02-20T00:10:07.008 INFO:teuthology.orchestra.run.smithi084.stderr:2024-02-20T00:10:07.003+0000 7f4da94a6700 1 --2- 172.21.15.84:0/1602242889 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f4d8c02ff10 0x7f4d8c0323d0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:10:07.008 INFO:teuthology.orchestra.run.smithi084.stderr:2024-02-20T00:10:07.003+0000 7f4da94a6700 1 --2- 172.21.15.84:0/1602242889 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4da40f6510 0x7f4da41067b0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:10:07.008 INFO:teuthology.orchestra.run.smithi084.stderr:2024-02-20T00:10:07.003+0000 7f4da94a6700 1 -- 172.21.15.84:0/1602242889 >> 172.21.15.84:0/1602242889 conn(0x7f4da40f1c70 msgr2=0x7f4da40f26a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:10:07.008 INFO:teuthology.orchestra.run.smithi084.stderr:2024-02-20T00:10:07.004+0000 7f4da94a6700 1 -- 172.21.15.84:0/1602242889 shutdown_connections 2024-02-20T00:10:07.008 INFO:teuthology.orchestra.run.smithi084.stderr:2024-02-20T00:10:07.004+0000 7f4da94a6700 1 -- 172.21.15.84:0/1602242889 wait complete. 2024-02-20T00:10:07.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:10:07 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:10:06.030389+0000 mon.smithi084 (mon.0) 112 : audit [INF] from='mgr.14162 172.21.15.84:0/2152933216' entity='mgr.smithi084.bmqrnf' 2024-02-20T00:10:07.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:10:07 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:10:06.468070+0000 mon.smithi084 (mon.0) 113 : audit [INF] from='mgr.14162 172.21.15.84:0/2152933216' entity='mgr.smithi084.bmqrnf' 2024-02-20T00:10:07.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:10:07 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:10:06.473572+0000 mon.smithi084 (mon.0) 114 : audit [INF] from='mgr.14162 172.21.15.84:0/2152933216' entity='mgr.smithi084.bmqrnf' 2024-02-20T00:10:07.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:10:07 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:10:07.003462+0000 mon.smithi084 (mon.0) 115 : audit [INF] from='client.? 172.21.15.84:0/1602242889' entity='client.admin' 2024-02-20T00:10:07.591 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi132 2024-02-20T00:10:07.591 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-02-20T00:10:07.591 DEBUG:teuthology.orchestra.run.smithi132:> dd of=/etc/ceph/ceph.conf 2024-02-20T00:10:07.613 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-02-20T00:10:07.613 DEBUG:teuthology.orchestra.run.smithi132:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-02-20T00:10:07.674 INFO:tasks.cephadm:Adding host smithi132 to orchestrator... 2024-02-20T00:10:07.675 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph orch host add smithi132 2024-02-20T00:10:08.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:10:08 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cephadm 2024-02-20T00:10:06.479871+0000 mgr.smithi084.bmqrnf (mgr.14162) 5 : cephadm [INF] Deploying daemon alertmanager.smithi084 on smithi084 2024-02-20T00:10:18.108 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:10:17 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:10:16.838467+0000 mgr.smithi084.bmqrnf (mgr.14162) 6 : cluster [DBG] pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:10:20.107 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:10:19 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:10:18.838876+0000 mgr.smithi084.bmqrnf (mgr.14162) 7 : cluster [DBG] pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:10:22.107 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:10:21 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:10:20.839123+0000 mgr.smithi084.bmqrnf (mgr.14162) 8 : cluster [DBG] pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:10:24.107 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:10:23 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:10:22.839370+0000 mgr.smithi084.bmqrnf (mgr.14162) 9 : cluster [DBG] pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:10:26.108 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:10:25 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:10:24.839709+0000 mgr.smithi084.bmqrnf (mgr.14162) 10 : cluster [DBG] pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:10:28.107 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:10:27 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:10:26.839929+0000 mgr.smithi084.bmqrnf (mgr.14162) 11 : cluster [DBG] pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:10:30.108 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:10:29 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:10:28.840297+0000 mgr.smithi084.bmqrnf (mgr.14162) 12 : cluster [DBG] pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:10:30.156 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:30.151+0000 7efec893b700 1 -- 172.21.15.132:0/2553376574 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7efec40f51a0 msgr2=0x7efec40f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:10:30.156 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:30.151+0000 7efec893b700 1 --2- 172.21.15.132:0/2553376574 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7efec40f51a0 0x7efec40f55c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7efeb8009fb0 tx=0x7efeb8009b30).stop 2024-02-20T00:10:30.156 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:30.151+0000 7efec893b700 1 -- 172.21.15.132:0/2553376574 shutdown_connections 2024-02-20T00:10:30.156 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:30.151+0000 7efec893b700 1 --2- 172.21.15.132:0/2553376574 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7efec40f51a0 0x7efec40f55c0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:10:30.156 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:30.152+0000 7efec893b700 1 -- 172.21.15.132:0/2553376574 >> 172.21.15.132:0/2553376574 conn(0x7efec40f09a0 msgr2=0x7efec40f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:10:30.156 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:30.152+0000 7efec893b700 1 -- 172.21.15.132:0/2553376574 shutdown_connections 2024-02-20T00:10:30.157 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:30.152+0000 7efec893b700 1 -- 172.21.15.132:0/2553376574 wait complete. 2024-02-20T00:10:30.157 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:30.152+0000 7efec893b700 1 Processor -- start 2024-02-20T00:10:30.157 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:30.152+0000 7efec893b700 1 -- start start 2024-02-20T00:10:30.157 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:30.153+0000 7efec893b700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7efec40f51a0 0x7efec4109290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:10:30.157 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:30.153+0000 7efec893b700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efec4060280 con 0x7efec40f51a0 2024-02-20T00:10:30.157 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:30.153+0000 7efec2d9d700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7efec40f51a0 0x7efec4109290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:10:30.157 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:30.153+0000 7efec2d9d700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7efec40f51a0 0x7efec4109290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:40844/0 (socket says 172.21.15.132:40844) 2024-02-20T00:10:30.157 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:30.153+0000 7efec2d9d700 1 -- 172.21.15.132:0/3421548861 learned_addr learned my addr 172.21.15.132:0/3421548861 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:10:30.157 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:30.153+0000 7efec2d9d700 1 -- 172.21.15.132:0/3421548861 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efeb8005040 con 0x7efec40f51a0 2024-02-20T00:10:30.157 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:30.153+0000 7efec2d9d700 1 --2- 172.21.15.132:0/3421548861 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7efec40f51a0 0x7efec4109290 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7efeb80089f0 tx=0x7efeb800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:10:30.157 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:30.153+0000 7efeab7fe700 1 -- 172.21.15.132:0/3421548861 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efeb8009320 con 0x7efec40f51a0 2024-02-20T00:10:30.157 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:30.153+0000 7efec893b700 1 -- 172.21.15.132:0/3421548861 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efec4060480 con 0x7efec40f51a0 2024-02-20T00:10:30.157 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:30.153+0000 7efeab7fe700 1 -- 172.21.15.132:0/3421548861 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efeb8013380 con 0x7efec40f51a0 2024-02-20T00:10:30.157 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:30.153+0000 7efeab7fe700 1 -- 172.21.15.132:0/3421548861 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efeb8014e20 con 0x7efec40f51a0 2024-02-20T00:10:30.158 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:30.153+0000 7efec893b700 1 -- 172.21.15.132:0/3421548861 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efec41099c0 con 0x7efec40f51a0 2024-02-20T00:10:30.158 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:30.154+0000 7efeab7fe700 1 -- 172.21.15.132:0/3421548861 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7efeb8014420 con 0x7efec40f51a0 2024-02-20T00:10:30.158 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:30.154+0000 7efec893b700 1 -- 172.21.15.132:0/3421548861 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efec4045ad0 con 0x7efec40f51a0 2024-02-20T00:10:30.159 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:30.155+0000 7efeab7fe700 1 --2- 172.21.15.132:0/3421548861 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7efeac02ff10 0x7efeac0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:10:30.159 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:30.155+0000 7efeab7fe700 1 -- 172.21.15.132:0/3421548861 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7efeb8040f00 con 0x7efec40f51a0 2024-02-20T00:10:30.159 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:30.155+0000 7efec259c700 1 --2- 172.21.15.132:0/3421548861 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7efeac02ff10 0x7efeac0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:10:30.159 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:30.155+0000 7efec259c700 1 --2- 172.21.15.132:0/3421548861 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7efeac02ff10 0x7efeac0323d0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7efeb40096a0 tx=0x7efeb4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:10:30.161 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:30.159+0000 7efeab7fe700 1 -- 172.21.15.132:0/3421548861 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7efeb80095f0 con 0x7efec40f51a0 2024-02-20T00:10:30.334 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:30.332+0000 7efec893b700 1 -- 172.21.15.132:0/3421548861 --> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi132", "target": ["mon-mgr", ""]}) v1 -- 0x7efec40f1ac0 con 0x7efeac02ff10 2024-02-20T00:10:30.926 INFO:teuthology.orchestra.run.smithi132.stdout:Added host 'smithi132' 2024-02-20T00:10:30.926 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:30.923+0000 7efeab7fe700 1 -- 172.21.15.132:0/3421548861 <== mgr.14162 v2:172.21.15.84:6800/1303158740 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+23 (secure 0 0 0) 0x7efec40f1ac0 con 0x7efeac02ff10 2024-02-20T00:10:30.928 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:30.925+0000 7efec893b700 1 -- 172.21.15.132:0/3421548861 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7efeac02ff10 msgr2=0x7efeac0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:10:30.928 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:30.925+0000 7efec893b700 1 --2- 172.21.15.132:0/3421548861 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7efeac02ff10 0x7efeac0323d0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7efeb40096a0 tx=0x7efeb4006b40).stop 2024-02-20T00:10:30.928 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:30.925+0000 7efec893b700 1 -- 172.21.15.132:0/3421548861 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7efec40f51a0 msgr2=0x7efec4109290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:10:30.928 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:30.925+0000 7efec893b700 1 --2- 172.21.15.132:0/3421548861 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7efec40f51a0 0x7efec4109290 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7efeb80089f0 tx=0x7efeb800b040).stop 2024-02-20T00:10:30.928 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:30.925+0000 7efec893b700 1 -- 172.21.15.132:0/3421548861 shutdown_connections 2024-02-20T00:10:30.928 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:30.925+0000 7efec893b700 1 --2- 172.21.15.132:0/3421548861 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7efeac02ff10 0x7efeac0323d0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:10:30.928 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:30.925+0000 7efec893b700 1 --2- 172.21.15.132:0/3421548861 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7efec40f51a0 0x7efec4109290 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:10:30.928 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:30.925+0000 7efec893b700 1 -- 172.21.15.132:0/3421548861 >> 172.21.15.132:0/3421548861 conn(0x7efec40f09a0 msgr2=0x7efec40f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:10:30.928 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:30.925+0000 7efec893b700 1 -- 172.21.15.132:0/3421548861 shutdown_connections 2024-02-20T00:10:30.928 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:30.925+0000 7efec893b700 1 -- 172.21.15.132:0/3421548861 wait complete. 2024-02-20T00:10:31.551 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph orch host ls --format=json 2024-02-20T00:10:32.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:10:31 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:10:30.334207+0000 mgr.smithi084.bmqrnf (mgr.14162) 13 : audit [DBG] from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi132", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T00:10:32.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:10:31 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:10:30.840675+0000 mgr.smithi084.bmqrnf (mgr.14162) 14 : cluster [DBG] pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:10:32.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:10:31 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:10:30.923727+0000 mon.smithi084 (mon.0) 116 : audit [INF] from='mgr.14162 172.21.15.84:0/2152933216' entity='mgr.smithi084.bmqrnf' 2024-02-20T00:10:32.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:10:31 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cephadm 2024-02-20T00:10:30.924099+0000 mgr.smithi084.bmqrnf (mgr.14162) 15 : cephadm [INF] Added host smithi132 2024-02-20T00:10:33.320 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:33.316+0000 7f0508c09700 1 -- 172.21.15.132:0/3732218752 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f05040f5410 msgr2=0x7f05040f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:10:33.320 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:33.316+0000 7f0508c09700 1 --2- 172.21.15.132:0/3732218752 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f05040f5410 0x7f05040f5830 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f04f4009fb0 tx=0x7f04f4009b30).stop 2024-02-20T00:10:33.321 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:33.316+0000 7f0508c09700 1 -- 172.21.15.132:0/3732218752 shutdown_connections 2024-02-20T00:10:33.321 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:33.316+0000 7f0508c09700 1 --2- 172.21.15.132:0/3732218752 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f05040f5410 0x7f05040f5830 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:10:33.321 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:33.316+0000 7f0508c09700 1 -- 172.21.15.132:0/3732218752 >> 172.21.15.132:0/3732218752 conn(0x7f05040f0b70 msgr2=0x7f05040f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:10:33.321 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:33.316+0000 7f0508c09700 1 -- 172.21.15.132:0/3732218752 shutdown_connections 2024-02-20T00:10:33.321 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:33.316+0000 7f0508c09700 1 -- 172.21.15.132:0/3732218752 wait complete. 2024-02-20T00:10:33.321 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:33.316+0000 7f0508c09700 1 Processor -- start 2024-02-20T00:10:33.321 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:33.317+0000 7f0508c09700 1 -- start start 2024-02-20T00:10:33.321 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:33.317+0000 7f0508c09700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f05040f5410 0x7f05041067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:10:33.321 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:33.317+0000 7f0508c09700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0504106cd0 con 0x7f05040f5410 2024-02-20T00:10:33.321 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:33.317+0000 7f050259c700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f05040f5410 0x7f05041067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:10:33.321 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:33.317+0000 7f050259c700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f05040f5410 0x7f05041067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:40854/0 (socket says 172.21.15.132:40854) 2024-02-20T00:10:33.321 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:33.317+0000 7f050259c700 1 -- 172.21.15.132:0/2734229835 learned_addr learned my addr 172.21.15.132:0/2734229835 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:10:33.322 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:33.317+0000 7f050259c700 1 -- 172.21.15.132:0/2734229835 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04f4005040 con 0x7f05040f5410 2024-02-20T00:10:33.322 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:33.317+0000 7f050259c700 1 --2- 172.21.15.132:0/2734229835 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f05040f5410 0x7f05041067e0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f04f4005010 tx=0x7f04f400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:10:33.322 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:33.318+0000 7f04f2ffd700 1 -- 172.21.15.132:0/2734229835 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f04f4009320 con 0x7f05040f5410 2024-02-20T00:10:33.322 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:33.318+0000 7f0508c09700 1 -- 172.21.15.132:0/2734229835 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0504106ed0 con 0x7f05040f5410 2024-02-20T00:10:33.322 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:33.318+0000 7f04f2ffd700 1 -- 172.21.15.132:0/2734229835 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f04f4013380 con 0x7f05040f5410 2024-02-20T00:10:33.322 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:33.318+0000 7f04f2ffd700 1 -- 172.21.15.132:0/2734229835 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f04f4014e50 con 0x7f05040f5410 2024-02-20T00:10:33.322 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:33.318+0000 7f0508c09700 1 -- 172.21.15.132:0/2734229835 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0504107360 con 0x7f05040f5410 2024-02-20T00:10:33.323 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:33.318+0000 7f04f2ffd700 1 -- 172.21.15.132:0/2734229835 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f04f40095f0 con 0x7f05040f5410 2024-02-20T00:10:33.323 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:33.318+0000 7f0508c09700 1 -- 172.21.15.132:0/2734229835 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0504045ad0 con 0x7f05040f5410 2024-02-20T00:10:33.323 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:33.319+0000 7f04f2ffd700 1 --2- 172.21.15.132:0/2734229835 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f04ec02ff10 0x7f04ec0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:10:33.323 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:33.319+0000 7f04f2ffd700 1 -- 172.21.15.132:0/2734229835 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f04f4040fa0 con 0x7f05040f5410 2024-02-20T00:10:33.323 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:33.319+0000 7f0501d9b700 1 --2- 172.21.15.132:0/2734229835 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f04ec02ff10 0x7f04ec0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:10:33.323 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:33.319+0000 7f0501d9b700 1 --2- 172.21.15.132:0/2734229835 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f04ec02ff10 0x7f04ec0323d0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f04f80096a0 tx=0x7f04f8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:10:33.325 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:33.323+0000 7f04f2ffd700 1 -- 172.21.15.132:0/2734229835 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f04f4014420 con 0x7f05040f5410 2024-02-20T00:10:33.495 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:33.493+0000 7f0508c09700 1 -- 172.21.15.132:0/2734229835 --> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f05040f1d20 con 0x7f04ec02ff10 2024-02-20T00:10:33.496 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:33.494+0000 7f04f2ffd700 1 -- 172.21.15.132:0/2734229835 <== mgr.14162 v2:172.21.15.84:6800/1303158740 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+156 (secure 0 0 0) 0x7f05040f1d20 con 0x7f04ec02ff10 2024-02-20T00:10:33.497 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:10:33.497 INFO:teuthology.orchestra.run.smithi132.stdout:[{"addr": "172.21.15.84", "hostname": "smithi084", "labels": [], "status": ""}, {"addr": "smithi132", "hostname": "smithi132", "labels": [], "status": ""}] 2024-02-20T00:10:33.498 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:33.496+0000 7f0508c09700 1 -- 172.21.15.132:0/2734229835 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f04ec02ff10 msgr2=0x7f04ec0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:10:33.498 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:33.496+0000 7f0508c09700 1 --2- 172.21.15.132:0/2734229835 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f04ec02ff10 0x7f04ec0323d0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f04f80096a0 tx=0x7f04f8006b40).stop 2024-02-20T00:10:33.498 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:33.496+0000 7f0508c09700 1 -- 172.21.15.132:0/2734229835 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f05040f5410 msgr2=0x7f05041067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:10:33.498 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:33.496+0000 7f0508c09700 1 --2- 172.21.15.132:0/2734229835 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f05040f5410 0x7f05041067e0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f04f4005010 tx=0x7f04f400b040).stop 2024-02-20T00:10:33.498 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:33.496+0000 7f0508c09700 1 -- 172.21.15.132:0/2734229835 shutdown_connections 2024-02-20T00:10:33.498 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:33.496+0000 7f0508c09700 1 --2- 172.21.15.132:0/2734229835 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f04ec02ff10 0x7f04ec0323d0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:10:33.499 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:33.496+0000 7f0508c09700 1 --2- 172.21.15.132:0/2734229835 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f05040f5410 0x7f05041067e0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:10:33.499 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:33.496+0000 7f0508c09700 1 -- 172.21.15.132:0/2734229835 >> 172.21.15.132:0/2734229835 conn(0x7f05040f0b70 msgr2=0x7f05040f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:10:33.499 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:33.496+0000 7f0508c09700 1 -- 172.21.15.132:0/2734229835 shutdown_connections 2024-02-20T00:10:33.499 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:33.496+0000 7f0508c09700 1 -- 172.21.15.132:0/2734229835 wait complete. 2024-02-20T00:10:34.034 INFO:tasks.cephadm:Setting crush tunables to default 2024-02-20T00:10:34.035 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph osd crush tunables default 2024-02-20T00:10:34.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:10:33 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:10:32.840939+0000 mgr.smithi084.bmqrnf (mgr.14162) 16 : cluster [DBG] pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:10:35.341 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:10:34 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:10:33.495381+0000 mgr.smithi084.bmqrnf (mgr.14162) 17 : audit [DBG] from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-20T00:10:35.825 INFO:teuthology.orchestra.run.smithi084.stderr:2024-02-20T00:10:35.793+0000 7f5bcb805700 1 -- 172.21.15.84:0/4228805974 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5bc4060280 msgr2=0x7f5bc40f90d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:10:35.825 INFO:teuthology.orchestra.run.smithi084.stderr:2024-02-20T00:10:35.793+0000 7f5bcb805700 1 --2- 172.21.15.84:0/4228805974 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5bc4060280 0x7f5bc40f90d0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f5bb4009fb0 tx=0x7f5bb4009b30).stop 2024-02-20T00:10:35.825 INFO:teuthology.orchestra.run.smithi084.stderr:2024-02-20T00:10:35.793+0000 7f5bcb805700 1 -- 172.21.15.84:0/4228805974 shutdown_connections 2024-02-20T00:10:35.825 INFO:teuthology.orchestra.run.smithi084.stderr:2024-02-20T00:10:35.793+0000 7f5bcb805700 1 --2- 172.21.15.84:0/4228805974 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5bc4060280 0x7f5bc40f90d0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:10:35.825 INFO:teuthology.orchestra.run.smithi084.stderr:2024-02-20T00:10:35.793+0000 7f5bcb805700 1 -- 172.21.15.84:0/4228805974 >> 172.21.15.84:0/4228805974 conn(0x7f5bc40f0b70 msgr2=0x7f5bc40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:10:35.825 INFO:teuthology.orchestra.run.smithi084.stderr:2024-02-20T00:10:35.793+0000 7f5bcb805700 1 -- 172.21.15.84:0/4228805974 shutdown_connections 2024-02-20T00:10:35.826 INFO:teuthology.orchestra.run.smithi084.stderr:2024-02-20T00:10:35.793+0000 7f5bcb805700 1 -- 172.21.15.84:0/4228805974 wait complete. 2024-02-20T00:10:35.826 INFO:teuthology.orchestra.run.smithi084.stderr:2024-02-20T00:10:35.794+0000 7f5bcb805700 1 Processor -- start 2024-02-20T00:10:35.826 INFO:teuthology.orchestra.run.smithi084.stderr:2024-02-20T00:10:35.794+0000 7f5bcb805700 1 -- start start 2024-02-20T00:10:35.826 INFO:teuthology.orchestra.run.smithi084.stderr:2024-02-20T00:10:35.794+0000 7f5bcb805700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5bc4060280 0x7f5bc40f6b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:10:35.826 INFO:teuthology.orchestra.run.smithi084.stderr:2024-02-20T00:10:35.794+0000 7f5bcb805700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bc40f4fd0 con 0x7f5bc4060280 2024-02-20T00:10:35.826 INFO:teuthology.orchestra.run.smithi084.stderr:2024-02-20T00:10:35.794+0000 7f5bc95a1700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5bc4060280 0x7f5bc40f6b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:10:35.826 INFO:teuthology.orchestra.run.smithi084.stderr:2024-02-20T00:10:35.794+0000 7f5bc95a1700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5bc4060280 0x7f5bc40f6b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:35528/0 (socket says 172.21.15.84:35528) 2024-02-20T00:10:35.826 INFO:teuthology.orchestra.run.smithi084.stderr:2024-02-20T00:10:35.794+0000 7f5bc95a1700 1 -- 172.21.15.84:0/3043601610 learned_addr learned my addr 172.21.15.84:0/3043601610 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-02-20T00:10:35.826 INFO:teuthology.orchestra.run.smithi084.stderr:2024-02-20T00:10:35.794+0000 7f5bc95a1700 1 -- 172.21.15.84:0/3043601610 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bb4005040 con 0x7f5bc4060280 2024-02-20T00:10:35.827 INFO:teuthology.orchestra.run.smithi084.stderr:2024-02-20T00:10:35.794+0000 7f5bc95a1700 1 --2- 172.21.15.84:0/3043601610 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5bc4060280 0x7f5bc40f6b50 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f5bb40089f0 tx=0x7f5bb400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:10:35.827 INFO:teuthology.orchestra.run.smithi084.stderr:2024-02-20T00:10:35.794+0000 7f5bb9ffb700 1 -- 172.21.15.84:0/3043601610 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5bb4009320 con 0x7f5bc4060280 2024-02-20T00:10:35.827 INFO:teuthology.orchestra.run.smithi084.stderr:2024-02-20T00:10:35.794+0000 7f5bcb805700 1 -- 172.21.15.84:0/3043601610 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bc40f5170 con 0x7f5bc4060280 2024-02-20T00:10:35.827 INFO:teuthology.orchestra.run.smithi084.stderr:2024-02-20T00:10:35.794+0000 7f5bb9ffb700 1 -- 172.21.15.84:0/3043601610 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5bb4013380 con 0x7f5bc4060280 2024-02-20T00:10:35.827 INFO:teuthology.orchestra.run.smithi084.stderr:2024-02-20T00:10:35.794+0000 7f5bb9ffb700 1 -- 172.21.15.84:0/3043601610 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5bb4014e20 con 0x7f5bc4060280 2024-02-20T00:10:35.827 INFO:teuthology.orchestra.run.smithi084.stderr:2024-02-20T00:10:35.795+0000 7f5bcb805700 1 -- 172.21.15.84:0/3043601610 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bc40f5600 con 0x7f5bc4060280 2024-02-20T00:10:35.827 INFO:teuthology.orchestra.run.smithi084.stderr:2024-02-20T00:10:35.795+0000 7f5bb9ffb700 1 -- 172.21.15.84:0/3043601610 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5bb4014420 con 0x7f5bc4060280 2024-02-20T00:10:35.827 INFO:teuthology.orchestra.run.smithi084.stderr:2024-02-20T00:10:35.795+0000 7f5bb9ffb700 1 --2- 172.21.15.84:0/3043601610 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f5bb002ff10 0x7f5bb00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:10:35.827 INFO:teuthology.orchestra.run.smithi084.stderr:2024-02-20T00:10:35.795+0000 7f5bb9ffb700 1 -- 172.21.15.84:0/3043601610 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5bb4040ef0 con 0x7f5bc4060280 2024-02-20T00:10:35.828 INFO:teuthology.orchestra.run.smithi084.stderr:2024-02-20T00:10:35.795+0000 7f5bc8da0700 1 --2- 172.21.15.84:0/3043601610 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f5bb002ff10 0x7f5bb00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:10:35.828 INFO:teuthology.orchestra.run.smithi084.stderr:2024-02-20T00:10:35.796+0000 7f5bc8da0700 1 --2- 172.21.15.84:0/3043601610 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f5bb002ff10 0x7f5bb00323d0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f5bc00096a0 tx=0x7f5bc0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:10:35.828 INFO:teuthology.orchestra.run.smithi084.stderr:2024-02-20T00:10:35.796+0000 7f5bcb805700 1 -- 172.21.15.84:0/3043601610 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5bc4059100 con 0x7f5bc4060280 2024-02-20T00:10:35.828 INFO:teuthology.orchestra.run.smithi084.stderr:2024-02-20T00:10:35.800+0000 7f5bb9ffb700 1 -- 172.21.15.84:0/3043601610 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5bb4014790 con 0x7f5bc4060280 2024-02-20T00:10:35.957 INFO:teuthology.orchestra.run.smithi084.stderr:2024-02-20T00:10:35.956+0000 7f5bcb805700 1 -- 172.21.15.84:0/3043601610 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f5bc4045ad0 con 0x7f5bc4060280 2024-02-20T00:10:36.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:10:35 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:10:34.841258+0000 mgr.smithi084.bmqrnf (mgr.14162) 18 : cluster [DBG] pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:10:36.935 INFO:teuthology.orchestra.run.smithi084.stderr:2024-02-20T00:10:36.933+0000 7f5bb9ffb700 1 -- 172.21.15.84:0/3043601610 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v5) v1 ==== 124+0+0 (secure 0 0 0) 0x7f5bb401d7f0 con 0x7f5bc4060280 2024-02-20T00:10:36.937 INFO:teuthology.orchestra.run.smithi084.stderr:2024-02-20T00:10:36.935+0000 7f5bcb805700 1 -- 172.21.15.84:0/3043601610 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f5bb002ff10 msgr2=0x7f5bb00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:10:36.937 INFO:teuthology.orchestra.run.smithi084.stderr:2024-02-20T00:10:36.935+0000 7f5bcb805700 1 --2- 172.21.15.84:0/3043601610 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f5bb002ff10 0x7f5bb00323d0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f5bc00096a0 tx=0x7f5bc0006b40).stop 2024-02-20T00:10:36.937 INFO:teuthology.orchestra.run.smithi084.stderr:2024-02-20T00:10:36.935+0000 7f5bcb805700 1 -- 172.21.15.84:0/3043601610 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5bc4060280 msgr2=0x7f5bc40f6b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:10:36.937 INFO:teuthology.orchestra.run.smithi084.stderr:2024-02-20T00:10:36.935+0000 7f5bcb805700 1 --2- 172.21.15.84:0/3043601610 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5bc4060280 0x7f5bc40f6b50 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f5bb40089f0 tx=0x7f5bb400b040).stop 2024-02-20T00:10:36.937 INFO:teuthology.orchestra.run.smithi084.stderr:2024-02-20T00:10:36.935+0000 7f5bcb805700 1 -- 172.21.15.84:0/3043601610 shutdown_connections 2024-02-20T00:10:36.937 INFO:teuthology.orchestra.run.smithi084.stderr:2024-02-20T00:10:36.936+0000 7f5bcb805700 1 --2- 172.21.15.84:0/3043601610 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f5bb002ff10 0x7f5bb00323d0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:10:36.938 INFO:teuthology.orchestra.run.smithi084.stderr:2024-02-20T00:10:36.936+0000 7f5bcb805700 1 --2- 172.21.15.84:0/3043601610 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5bc4060280 0x7f5bc40f6b50 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:10:36.938 INFO:teuthology.orchestra.run.smithi084.stderr:2024-02-20T00:10:36.936+0000 7f5bcb805700 1 -- 172.21.15.84:0/3043601610 >> 172.21.15.84:0/3043601610 conn(0x7f5bc40f0b70 msgr2=0x7f5bc40f1580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:10:36.938 INFO:teuthology.orchestra.run.smithi084.stderr:2024-02-20T00:10:36.936+0000 7f5bcb805700 1 -- 172.21.15.84:0/3043601610 shutdown_connections 2024-02-20T00:10:36.938 INFO:teuthology.orchestra.run.smithi084.stderr:2024-02-20T00:10:36.936+0000 7f5bcb805700 1 -- 172.21.15.84:0/3043601610 wait complete. 2024-02-20T00:10:36.938 INFO:teuthology.orchestra.run.smithi084.stderr:adjusted tunables profile to default 2024-02-20T00:10:37.241 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:10:36 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:10:35.957173+0000 mon.smithi084 (mon.0) 117 : audit [INF] from='client.? 172.21.15.84:0/3043601610' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-02-20T00:10:37.511 INFO:tasks.cephadm:Adding mon.smithi084 on smithi084 2024-02-20T00:10:37.511 INFO:tasks.cephadm:Adding mon.smithi132 on smithi132 2024-02-20T00:10:37.511 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph orch apply mon '2;smithi084:172.21.15.84=smithi084;smithi132:172.21.15.132=smithi132' 2024-02-20T00:10:38.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:10:37 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:10:36.841528+0000 mgr.smithi084.bmqrnf (mgr.14162) 19 : cluster [DBG] pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:10:38.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:10:37 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:10:36.933922+0000 mon.smithi084 (mon.0) 118 : audit [INF] from='client.? 172.21.15.84:0/3043601610' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-02-20T00:10:38.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:10:37 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:10:36.933972+0000 mon.smithi084 (mon.0) 119 : cluster [DBG] osdmap e5: 0 total, 0 up, 0 in 2024-02-20T00:10:39.367 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:39.353+0000 7f6de6ee1700 1 -- 172.21.15.132:0/3127952970 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6de00f6510 msgr2=0x7f6de00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:10:39.367 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:39.353+0000 7f6de6ee1700 1 --2- 172.21.15.132:0/3127952970 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6de00f6510 0x7f6de00f6930 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f6dd4009fb0 tx=0x7f6dd4009b30).stop 2024-02-20T00:10:39.367 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:39.353+0000 7f6de6ee1700 1 -- 172.21.15.132:0/3127952970 shutdown_connections 2024-02-20T00:10:39.368 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:39.353+0000 7f6de6ee1700 1 --2- 172.21.15.132:0/3127952970 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6de00f6510 0x7f6de00f6930 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:10:39.368 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:39.353+0000 7f6de6ee1700 1 -- 172.21.15.132:0/3127952970 >> 172.21.15.132:0/3127952970 conn(0x7f6de00f1c70 msgr2=0x7f6de00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:10:39.368 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:39.353+0000 7f6de6ee1700 1 -- 172.21.15.132:0/3127952970 shutdown_connections 2024-02-20T00:10:39.368 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:39.353+0000 7f6de6ee1700 1 -- 172.21.15.132:0/3127952970 wait complete. 2024-02-20T00:10:39.368 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:39.353+0000 7f6de6ee1700 1 Processor -- start 2024-02-20T00:10:39.368 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:39.353+0000 7f6de6ee1700 1 -- start start 2024-02-20T00:10:39.368 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:39.354+0000 7f6de6ee1700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6de00f6510 0x7f6de01067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:10:39.368 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:39.354+0000 7f6de6ee1700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6de0106cd0 con 0x7f6de00f6510 2024-02-20T00:10:39.368 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:39.354+0000 7f6de4c7d700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6de00f6510 0x7f6de01067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:10:39.368 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:39.354+0000 7f6de4c7d700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6de00f6510 0x7f6de01067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:47342/0 (socket says 172.21.15.132:47342) 2024-02-20T00:10:39.368 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:39.354+0000 7f6de4c7d700 1 -- 172.21.15.132:0/802474421 learned_addr learned my addr 172.21.15.132:0/802474421 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:10:39.368 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:39.354+0000 7f6de4c7d700 1 -- 172.21.15.132:0/802474421 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dd4005040 con 0x7f6de00f6510 2024-02-20T00:10:39.368 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:39.354+0000 7f6de4c7d700 1 --2- 172.21.15.132:0/802474421 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6de00f6510 0x7f6de01067e0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f6dd4005010 tx=0x7f6dd400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:10:39.369 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:39.354+0000 7f6ddd7fa700 1 -- 172.21.15.132:0/802474421 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6dd4009320 con 0x7f6de00f6510 2024-02-20T00:10:39.369 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:39.354+0000 7f6de6ee1700 1 -- 172.21.15.132:0/802474421 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6de0106ed0 con 0x7f6de00f6510 2024-02-20T00:10:39.369 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:39.354+0000 7f6ddd7fa700 1 -- 172.21.15.132:0/802474421 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6dd4013380 con 0x7f6de00f6510 2024-02-20T00:10:39.369 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:39.354+0000 7f6ddd7fa700 1 -- 172.21.15.132:0/802474421 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6dd4014e50 con 0x7f6de00f6510 2024-02-20T00:10:39.369 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:39.355+0000 7f6de6ee1700 1 -- 172.21.15.132:0/802474421 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6de0107360 con 0x7f6de00f6510 2024-02-20T00:10:39.369 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:39.355+0000 7f6ddd7fa700 1 -- 172.21.15.132:0/802474421 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6dd40095f0 con 0x7f6de00f6510 2024-02-20T00:10:39.369 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:39.355+0000 7f6de6ee1700 1 -- 172.21.15.132:0/802474421 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6de0045ad0 con 0x7f6de00f6510 2024-02-20T00:10:39.369 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:39.355+0000 7f6ddd7fa700 1 --2- 172.21.15.132:0/802474421 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f6dd002ff10 0x7f6dd00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:10:39.369 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:39.355+0000 7f6ddd7fa700 1 -- 172.21.15.132:0/802474421 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6dd4040f10 con 0x7f6de00f6510 2024-02-20T00:10:39.369 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:39.356+0000 7f6ddffff700 1 --2- 172.21.15.132:0/802474421 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f6dd002ff10 0x7f6dd00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:10:39.369 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:39.356+0000 7f6ddffff700 1 --2- 172.21.15.132:0/802474421 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f6dd002ff10 0x7f6dd00323d0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f6dcc0096a0 tx=0x7f6dcc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:10:39.369 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:39.360+0000 7f6ddd7fa700 1 -- 172.21.15.132:0/802474421 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6dd4014420 con 0x7f6de00f6510 2024-02-20T00:10:39.536 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:39.534+0000 7f6de6ee1700 1 -- 172.21.15.132:0/802474421 --> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi084:172.21.15.84=smithi084;smithi132:172.21.15.132=smithi132", "target": ["mon-mgr", ""]}) v1 -- 0x7f6de00f2ed0 con 0x7f6dd002ff10 2024-02-20T00:10:39.542 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:39.539+0000 7f6ddd7fa700 1 -- 172.21.15.132:0/802474421 <== mgr.14162 v2:172.21.15.84:6800/1303158740 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f6de00f2ed0 con 0x7f6dd002ff10 2024-02-20T00:10:39.542 INFO:teuthology.orchestra.run.smithi132.stdout:Scheduled mon update... 2024-02-20T00:10:39.544 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:39.541+0000 7f6de6ee1700 1 -- 172.21.15.132:0/802474421 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f6dd002ff10 msgr2=0x7f6dd00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:10:39.544 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:39.541+0000 7f6de6ee1700 1 --2- 172.21.15.132:0/802474421 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f6dd002ff10 0x7f6dd00323d0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f6dcc0096a0 tx=0x7f6dcc006b40).stop 2024-02-20T00:10:39.544 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:39.541+0000 7f6de6ee1700 1 -- 172.21.15.132:0/802474421 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6de00f6510 msgr2=0x7f6de01067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:10:39.544 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:39.541+0000 7f6de6ee1700 1 --2- 172.21.15.132:0/802474421 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6de00f6510 0x7f6de01067e0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f6dd4005010 tx=0x7f6dd400b040).stop 2024-02-20T00:10:39.544 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:39.541+0000 7f6de6ee1700 1 -- 172.21.15.132:0/802474421 shutdown_connections 2024-02-20T00:10:39.544 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:39.541+0000 7f6de6ee1700 1 --2- 172.21.15.132:0/802474421 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f6dd002ff10 0x7f6dd00323d0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:10:39.544 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:39.541+0000 7f6de6ee1700 1 --2- 172.21.15.132:0/802474421 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6de00f6510 0x7f6de01067e0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:10:39.544 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:39.541+0000 7f6de6ee1700 1 -- 172.21.15.132:0/802474421 >> 172.21.15.132:0/802474421 conn(0x7f6de00f1c70 msgr2=0x7f6de00f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:10:39.544 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:39.541+0000 7f6de6ee1700 1 -- 172.21.15.132:0/802474421 shutdown_connections 2024-02-20T00:10:39.545 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:39.541+0000 7f6de6ee1700 1 -- 172.21.15.132:0/802474421 wait complete. 2024-02-20T00:10:40.239 DEBUG:teuthology.orchestra.run.smithi132:mon.smithi132> sudo journalctl -f -n 0 -u ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715@mon.smithi132.service 2024-02-20T00:10:40.241 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:10:40.241 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:10:40.270 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:-- Logs begin at Mon 2024-02-19 23:58:48 UTC. -- 2024-02-20T00:10:40.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:10:39 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:10:38.841763+0000 mgr.smithi084.bmqrnf (mgr.14162) 20 : cluster [DBG] pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:10:40.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:10:39 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:10:39.540744+0000 mon.smithi084 (mon.0) 120 : audit [INF] from='mgr.14162 172.21.15.84:0/2152933216' entity='mgr.smithi084.bmqrnf' 2024-02-20T00:10:41.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:10:40 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:10:39.536820+0000 mgr.smithi084.bmqrnf (mgr.14162) 21 : audit [DBG] from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi084:172.21.15.84=smithi084;smithi132:172.21.15.132=smithi132", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T00:10:41.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:10:40 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cephadm 2024-02-20T00:10:39.538165+0000 mgr.smithi084.bmqrnf (mgr.14162) 22 : cephadm [INF] Saving service mon spec with placement smithi084:172.21.15.84=smithi084;smithi132:172.21.15.132=smithi132;count:2 2024-02-20T00:10:42.118 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:42.114+0000 7f5e75ff5700 1 -- 172.21.15.132:0/2128632508 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5e70060280 msgr2=0x7f5e700f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:10:42.118 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:42.114+0000 7f5e75ff5700 1 --2- 172.21.15.132:0/2128632508 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5e70060280 0x7f5e700f90f0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f5e64009fb0 tx=0x7f5e64009b30).stop 2024-02-20T00:10:42.118 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:42.114+0000 7f5e75ff5700 1 -- 172.21.15.132:0/2128632508 shutdown_connections 2024-02-20T00:10:42.118 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:42.114+0000 7f5e75ff5700 1 --2- 172.21.15.132:0/2128632508 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5e70060280 0x7f5e700f90f0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:10:42.118 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:42.114+0000 7f5e75ff5700 1 -- 172.21.15.132:0/2128632508 >> 172.21.15.132:0/2128632508 conn(0x7f5e700f0b70 msgr2=0x7f5e700f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:10:42.118 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:42.114+0000 7f5e75ff5700 1 -- 172.21.15.132:0/2128632508 shutdown_connections 2024-02-20T00:10:42.119 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:42.114+0000 7f5e75ff5700 1 -- 172.21.15.132:0/2128632508 wait complete. 2024-02-20T00:10:42.119 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:42.114+0000 7f5e75ff5700 1 Processor -- start 2024-02-20T00:10:42.119 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:42.115+0000 7f5e75ff5700 1 -- start start 2024-02-20T00:10:42.119 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:42.115+0000 7f5e75ff5700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5e70060280 0x7f5e70106780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:10:42.119 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:42.115+0000 7f5e75ff5700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e70106c70 con 0x7f5e70060280 2024-02-20T00:10:42.119 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:42.115+0000 7f5e6f7fe700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5e70060280 0x7f5e70106780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:10:42.119 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:42.115+0000 7f5e6f7fe700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5e70060280 0x7f5e70106780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:47346/0 (socket says 172.21.15.132:47346) 2024-02-20T00:10:42.119 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:42.115+0000 7f5e6f7fe700 1 -- 172.21.15.132:0/3191393547 learned_addr learned my addr 172.21.15.132:0/3191393547 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:10:42.119 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:42.115+0000 7f5e6f7fe700 1 -- 172.21.15.132:0/3191393547 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e64005040 con 0x7f5e70060280 2024-02-20T00:10:42.119 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:42.116+0000 7f5e6f7fe700 1 --2- 172.21.15.132:0/3191393547 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5e70060280 0x7f5e70106780 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f5e640089f0 tx=0x7f5e6400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:10:42.119 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:42.116+0000 7f5e57fff700 1 -- 172.21.15.132:0/3191393547 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5e64009320 con 0x7f5e70060280 2024-02-20T00:10:42.119 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:42.116+0000 7f5e75ff5700 1 -- 172.21.15.132:0/3191393547 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e70106e70 con 0x7f5e70060280 2024-02-20T00:10:42.119 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:42.116+0000 7f5e57fff700 1 -- 172.21.15.132:0/3191393547 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5e64013380 con 0x7f5e70060280 2024-02-20T00:10:42.119 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:42.116+0000 7f5e57fff700 1 -- 172.21.15.132:0/3191393547 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5e64014e20 con 0x7f5e70060280 2024-02-20T00:10:42.119 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:42.116+0000 7f5e75ff5700 1 -- 172.21.15.132:0/3191393547 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e70107300 con 0x7f5e70060280 2024-02-20T00:10:42.120 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:42.117+0000 7f5e57fff700 1 -- 172.21.15.132:0/3191393547 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5e640095f0 con 0x7f5e70060280 2024-02-20T00:10:42.120 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:42.117+0000 7f5e75ff5700 1 -- 172.21.15.132:0/3191393547 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e70045ad0 con 0x7f5e70060280 2024-02-20T00:10:42.120 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:42.117+0000 7f5e57fff700 1 --2- 172.21.15.132:0/3191393547 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f5e5802ff10 0x7f5e580323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:10:42.120 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:42.117+0000 7f5e57fff700 1 -- 172.21.15.132:0/3191393547 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5e64040e90 con 0x7f5e70060280 2024-02-20T00:10:42.121 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:42.117+0000 7f5e6effd700 1 --2- 172.21.15.132:0/3191393547 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f5e5802ff10 0x7f5e580323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:10:42.121 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:42.118+0000 7f5e6effd700 1 --2- 172.21.15.132:0/3191393547 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f5e5802ff10 0x7f5e580323d0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f5e600096a0 tx=0x7f5e60006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:10:42.124 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:42.122+0000 7f5e57fff700 1 -- 172.21.15.132:0/3191393547 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5e64014420 con 0x7f5e70060280 2024-02-20T00:10:42.329 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:42.326+0000 7f5e75ff5700 1 -- 172.21.15.132:0/3191393547 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5e701081b0 con 0x7f5e70060280 2024-02-20T00:10:42.329 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:42.327+0000 7f5e57fff700 1 -- 172.21.15.132:0/3191393547 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5e6401d980 con 0x7f5e70060280 2024-02-20T00:10:42.329 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:10:42.330 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:10:42.331 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:42.329+0000 7f5e75ff5700 1 -- 172.21.15.132:0/3191393547 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f5e5802ff10 msgr2=0x7f5e580323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:10:42.331 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:42.329+0000 7f5e75ff5700 1 --2- 172.21.15.132:0/3191393547 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f5e5802ff10 0x7f5e580323d0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f5e600096a0 tx=0x7f5e60006b40).stop 2024-02-20T00:10:42.331 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:42.329+0000 7f5e75ff5700 1 -- 172.21.15.132:0/3191393547 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5e70060280 msgr2=0x7f5e70106780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:10:42.332 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:42.329+0000 7f5e75ff5700 1 --2- 172.21.15.132:0/3191393547 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5e70060280 0x7f5e70106780 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f5e640089f0 tx=0x7f5e6400b040).stop 2024-02-20T00:10:42.332 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:42.329+0000 7f5e75ff5700 1 -- 172.21.15.132:0/3191393547 shutdown_connections 2024-02-20T00:10:42.332 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:42.329+0000 7f5e75ff5700 1 --2- 172.21.15.132:0/3191393547 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f5e5802ff10 0x7f5e580323d0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:10:42.332 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:42.329+0000 7f5e75ff5700 1 --2- 172.21.15.132:0/3191393547 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5e70060280 0x7f5e70106780 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:10:42.332 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:42.329+0000 7f5e75ff5700 1 -- 172.21.15.132:0/3191393547 >> 172.21.15.132:0/3191393547 conn(0x7f5e700f0b70 msgr2=0x7f5e700f15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:10:42.332 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:42.329+0000 7f5e75ff5700 1 -- 172.21.15.132:0/3191393547 shutdown_connections 2024-02-20T00:10:42.332 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:42.329+0000 7f5e75ff5700 1 -- 172.21.15.132:0/3191393547 wait complete. 2024-02-20T00:10:42.332 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:10:42.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:10:41 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:10:40.842122+0000 mgr.smithi084.bmqrnf (mgr.14162) 23 : cluster [DBG] pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:10:43.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:10:42 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:10:42.328685+0000 mon.smithi084 (mon.0) 121 : audit [DBG] from='client.? 172.21.15.132:0/3191393547' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:10:43.875 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:10:43.875 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:10:44.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:10:43 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:10:42.842545+0000 mgr.smithi084.bmqrnf (mgr.14162) 24 : cluster [DBG] pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:10:45.751 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:45.747+0000 7fedc0372700 1 -- 172.21.15.132:0/3641873696 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fedb80601e0 msgr2=0x7fedb80f9190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:10:45.751 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:45.747+0000 7fedc0372700 1 --2- 172.21.15.132:0/3641873696 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fedb80601e0 0x7fedb80f9190 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7feda8009fb0 tx=0x7feda8009b30).stop 2024-02-20T00:10:45.751 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:45.747+0000 7fedc0372700 1 -- 172.21.15.132:0/3641873696 shutdown_connections 2024-02-20T00:10:45.751 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:45.747+0000 7fedc0372700 1 --2- 172.21.15.132:0/3641873696 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fedb80601e0 0x7fedb80f9190 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:10:45.752 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:45.747+0000 7fedc0372700 1 -- 172.21.15.132:0/3641873696 >> 172.21.15.132:0/3641873696 conn(0x7fedb80f0b70 msgr2=0x7fedb80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:10:45.752 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:45.747+0000 7fedc0372700 1 -- 172.21.15.132:0/3641873696 shutdown_connections 2024-02-20T00:10:45.752 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:45.747+0000 7fedc0372700 1 -- 172.21.15.132:0/3641873696 wait complete. 2024-02-20T00:10:45.752 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:45.748+0000 7fedc0372700 1 Processor -- start 2024-02-20T00:10:45.752 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:45.748+0000 7fedc0372700 1 -- start start 2024-02-20T00:10:45.752 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:45.748+0000 7fedc0372700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fedb80601e0 0x7fedb81067d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:10:45.752 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:45.748+0000 7fedc0372700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedb8106cc0 con 0x7fedb80601e0 2024-02-20T00:10:45.752 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:45.748+0000 7fedbe10e700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fedb80601e0 0x7fedb81067d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:10:45.752 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:45.748+0000 7fedbe10e700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fedb80601e0 0x7fedb81067d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:47358/0 (socket says 172.21.15.132:47358) 2024-02-20T00:10:45.752 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:45.748+0000 7fedbe10e700 1 -- 172.21.15.132:0/1940559152 learned_addr learned my addr 172.21.15.132:0/1940559152 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:10:45.752 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:45.749+0000 7fedbe10e700 1 -- 172.21.15.132:0/1940559152 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feda8005040 con 0x7fedb80601e0 2024-02-20T00:10:45.752 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:45.749+0000 7fedbe10e700 1 --2- 172.21.15.132:0/1940559152 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fedb80601e0 0x7fedb81067d0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7feda8007a10 tx=0x7feda8009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:10:45.752 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:45.749+0000 7fedaeffd700 1 -- 172.21.15.132:0/1940559152 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feda8018070 con 0x7fedb80601e0 2024-02-20T00:10:45.752 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:45.749+0000 7fedc0372700 1 -- 172.21.15.132:0/1940559152 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedb8106ec0 con 0x7fedb80601e0 2024-02-20T00:10:45.753 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:45.749+0000 7fedaeffd700 1 -- 172.21.15.132:0/1940559152 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7feda8013410 con 0x7fedb80601e0 2024-02-20T00:10:45.753 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:45.749+0000 7fedaeffd700 1 -- 172.21.15.132:0/1940559152 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feda801c3c0 con 0x7fedb80601e0 2024-02-20T00:10:45.753 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:45.749+0000 7fedc0372700 1 -- 172.21.15.132:0/1940559152 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fedb8107350 con 0x7fedb80601e0 2024-02-20T00:10:45.753 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:45.750+0000 7fedaeffd700 1 -- 172.21.15.132:0/1940559152 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7feda801d420 con 0x7fedb80601e0 2024-02-20T00:10:45.754 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:45.750+0000 7fedaeffd700 1 --2- 172.21.15.132:0/1940559152 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7feda402ff60 0x7feda4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:10:45.754 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:45.750+0000 7fedc0372700 1 -- 172.21.15.132:0/1940559152 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fedb8045ad0 con 0x7fedb80601e0 2024-02-20T00:10:45.754 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:45.750+0000 7fedaeffd700 1 -- 172.21.15.132:0/1940559152 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7feda80414b0 con 0x7fedb80601e0 2024-02-20T00:10:45.754 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:45.750+0000 7fedbd90d700 1 --2- 172.21.15.132:0/1940559152 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7feda402ff60 0x7feda4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:10:45.754 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:45.751+0000 7fedbd90d700 1 --2- 172.21.15.132:0/1940559152 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7feda402ff60 0x7feda4032420 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7fedb40096a0 tx=0x7fedb4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:10:45.757 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:45.755+0000 7fedaeffd700 1 -- 172.21.15.132:0/1940559152 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7feda801b370 con 0x7fedb80601e0 2024-02-20T00:10:45.960 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:45.958+0000 7fedc0372700 1 -- 172.21.15.132:0/1940559152 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fedb8108200 con 0x7fedb80601e0 2024-02-20T00:10:45.960 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:45.958+0000 7fedaeffd700 1 -- 172.21.15.132:0/1940559152 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7feda801b370 con 0x7fedb80601e0 2024-02-20T00:10:45.961 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:10:45.961 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:10:45.962 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:45.960+0000 7fedc0372700 1 -- 172.21.15.132:0/1940559152 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7feda402ff60 msgr2=0x7feda4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:10:45.963 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:45.960+0000 7fedc0372700 1 --2- 172.21.15.132:0/1940559152 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7feda402ff60 0x7feda4032420 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7fedb40096a0 tx=0x7fedb4006b40).stop 2024-02-20T00:10:45.963 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:45.960+0000 7fedc0372700 1 -- 172.21.15.132:0/1940559152 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fedb80601e0 msgr2=0x7fedb81067d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:10:45.963 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:45.960+0000 7fedc0372700 1 --2- 172.21.15.132:0/1940559152 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fedb80601e0 0x7fedb81067d0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7feda8007a10 tx=0x7feda8009b30).stop 2024-02-20T00:10:45.963 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:45.960+0000 7fedc0372700 1 -- 172.21.15.132:0/1940559152 shutdown_connections 2024-02-20T00:10:45.963 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:45.960+0000 7fedc0372700 1 --2- 172.21.15.132:0/1940559152 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7feda402ff60 0x7feda4032420 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:10:45.963 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:45.960+0000 7fedc0372700 1 --2- 172.21.15.132:0/1940559152 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fedb80601e0 0x7fedb81067d0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:10:45.963 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:45.960+0000 7fedc0372700 1 -- 172.21.15.132:0/1940559152 >> 172.21.15.132:0/1940559152 conn(0x7fedb80f0b70 msgr2=0x7fedb80f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:10:45.963 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:45.960+0000 7fedc0372700 1 -- 172.21.15.132:0/1940559152 shutdown_connections 2024-02-20T00:10:45.963 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:45.960+0000 7fedc0372700 1 -- 172.21.15.132:0/1940559152 wait complete. 2024-02-20T00:10:45.964 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:10:46.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:10:45 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:10:44.842864+0000 mgr.smithi084.bmqrnf (mgr.14162) 25 : cluster [DBG] pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:10:47.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:10:46 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:10:45.960064+0000 mon.smithi084 (mon.0) 122 : audit [DBG] from='client.? 172.21.15.132:0/1940559152' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:10:47.582 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:10:47.582 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:10:48.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:10:47 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:10:46.843244+0000 mgr.smithi084.bmqrnf (mgr.14162) 26 : cluster [DBG] pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:10:49.403 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:49.383+0000 7f6e1f243700 1 -- 172.21.15.132:0/2676296285 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6e180f5410 msgr2=0x7f6e180f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:10:49.403 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:49.383+0000 7f6e1f243700 1 --2- 172.21.15.132:0/2676296285 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6e180f5410 0x7f6e180f5830 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f6e0c009fb0 tx=0x7f6e0c009b30).stop 2024-02-20T00:10:49.403 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:49.383+0000 7f6e1f243700 1 -- 172.21.15.132:0/2676296285 shutdown_connections 2024-02-20T00:10:49.403 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:49.383+0000 7f6e1f243700 1 --2- 172.21.15.132:0/2676296285 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6e180f5410 0x7f6e180f5830 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:10:49.403 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:49.383+0000 7f6e1f243700 1 -- 172.21.15.132:0/2676296285 >> 172.21.15.132:0/2676296285 conn(0x7f6e180f0b70 msgr2=0x7f6e180f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:10:49.403 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:49.383+0000 7f6e1f243700 1 -- 172.21.15.132:0/2676296285 shutdown_connections 2024-02-20T00:10:49.403 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:49.383+0000 7f6e1f243700 1 -- 172.21.15.132:0/2676296285 wait complete. 2024-02-20T00:10:49.404 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:49.383+0000 7f6e1f243700 1 Processor -- start 2024-02-20T00:10:49.404 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:49.384+0000 7f6e1f243700 1 -- start start 2024-02-20T00:10:49.404 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:49.384+0000 7f6e1f243700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6e180f5410 0x7f6e181094e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:10:49.404 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:49.384+0000 7f6e1f243700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e18060240 con 0x7f6e180f5410 2024-02-20T00:10:49.404 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:49.384+0000 7f6e1cfdf700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6e180f5410 0x7f6e181094e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:10:49.404 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:49.384+0000 7f6e1cfdf700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6e180f5410 0x7f6e181094e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:50870/0 (socket says 172.21.15.132:50870) 2024-02-20T00:10:49.404 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:49.384+0000 7f6e1cfdf700 1 -- 172.21.15.132:0/3859372099 learned_addr learned my addr 172.21.15.132:0/3859372099 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:10:49.404 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:49.385+0000 7f6e1cfdf700 1 -- 172.21.15.132:0/3859372099 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e0c005040 con 0x7f6e180f5410 2024-02-20T00:10:49.404 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:49.385+0000 7f6e1cfdf700 1 --2- 172.21.15.132:0/3859372099 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6e180f5410 0x7f6e181094e0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f6e0c005010 tx=0x7f6e0c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:10:49.404 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:49.385+0000 7f6e157fa700 1 -- 172.21.15.132:0/3859372099 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6e0c009320 con 0x7f6e180f5410 2024-02-20T00:10:49.404 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:49.385+0000 7f6e1f243700 1 -- 172.21.15.132:0/3859372099 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e18060440 con 0x7f6e180f5410 2024-02-20T00:10:49.404 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:49.385+0000 7f6e157fa700 1 -- 172.21.15.132:0/3859372099 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6e0c013380 con 0x7f6e180f5410 2024-02-20T00:10:49.404 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:49.385+0000 7f6e157fa700 1 -- 172.21.15.132:0/3859372099 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6e0c014e50 con 0x7f6e180f5410 2024-02-20T00:10:49.404 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:49.385+0000 7f6e1f243700 1 -- 172.21.15.132:0/3859372099 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e18109b90 con 0x7f6e180f5410 2024-02-20T00:10:49.405 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:49.386+0000 7f6e157fa700 1 -- 172.21.15.132:0/3859372099 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6e0c0095f0 con 0x7f6e180f5410 2024-02-20T00:10:49.405 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:49.386+0000 7f6e1f243700 1 -- 172.21.15.132:0/3859372099 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e18045ad0 con 0x7f6e180f5410 2024-02-20T00:10:49.405 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:49.386+0000 7f6e157fa700 1 --2- 172.21.15.132:0/3859372099 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f6e0802ff10 0x7f6e080323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:10:49.405 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:49.386+0000 7f6e157fa700 1 -- 172.21.15.132:0/3859372099 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6e0c040fa0 con 0x7f6e180f5410 2024-02-20T00:10:49.405 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:49.386+0000 7f6e17fff700 1 --2- 172.21.15.132:0/3859372099 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f6e0802ff10 0x7f6e080323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:10:49.405 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:49.387+0000 7f6e17fff700 1 --2- 172.21.15.132:0/3859372099 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f6e0802ff10 0x7f6e080323d0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f6e040096a0 tx=0x7f6e04006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:10:49.405 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:49.391+0000 7f6e157fa700 1 -- 172.21.15.132:0/3859372099 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6e0c014420 con 0x7f6e180f5410 2024-02-20T00:10:49.600 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:49.598+0000 7f6e1f243700 1 -- 172.21.15.132:0/3859372099 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6e1810a3b0 con 0x7f6e180f5410 2024-02-20T00:10:49.600 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:49.598+0000 7f6e157fa700 1 -- 172.21.15.132:0/3859372099 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6e0c010230 con 0x7f6e180f5410 2024-02-20T00:10:49.601 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:10:49.601 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:10:49.602 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:49.601+0000 7f6e1f243700 1 -- 172.21.15.132:0/3859372099 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f6e0802ff10 msgr2=0x7f6e080323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:10:49.602 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:49.601+0000 7f6e1f243700 1 --2- 172.21.15.132:0/3859372099 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f6e0802ff10 0x7f6e080323d0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f6e040096a0 tx=0x7f6e04006b40).stop 2024-02-20T00:10:49.602 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:49.601+0000 7f6e1f243700 1 -- 172.21.15.132:0/3859372099 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6e180f5410 msgr2=0x7f6e181094e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:10:49.602 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:49.601+0000 7f6e1f243700 1 --2- 172.21.15.132:0/3859372099 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6e180f5410 0x7f6e181094e0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f6e0c005010 tx=0x7f6e0c00b040).stop 2024-02-20T00:10:49.602 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:49.601+0000 7f6e1f243700 1 -- 172.21.15.132:0/3859372099 shutdown_connections 2024-02-20T00:10:49.602 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:49.601+0000 7f6e1f243700 1 --2- 172.21.15.132:0/3859372099 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f6e0802ff10 0x7f6e080323d0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:10:49.602 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:49.601+0000 7f6e1f243700 1 --2- 172.21.15.132:0/3859372099 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6e180f5410 0x7f6e181094e0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:10:49.602 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:49.601+0000 7f6e1f243700 1 -- 172.21.15.132:0/3859372099 >> 172.21.15.132:0/3859372099 conn(0x7f6e180f0b70 msgr2=0x7f6e180f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:10:49.602 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:49.601+0000 7f6e1f243700 1 -- 172.21.15.132:0/3859372099 shutdown_connections 2024-02-20T00:10:49.603 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:49.601+0000 7f6e1f243700 1 -- 172.21.15.132:0/3859372099 wait complete. 2024-02-20T00:10:49.603 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:10:50.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:10:49 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:10:48.843530+0000 mgr.smithi084.bmqrnf (mgr.14162) 27 : cluster [DBG] pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:10:50.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:10:49 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:10:49.599738+0000 mon.smithi084 (mon.0) 123 : audit [DBG] from='client.? 172.21.15.132:0/3859372099' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:10:51.159 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:10:51.160 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:10:52.107 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:10:51 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:10:50.843817+0000 mgr.smithi084.bmqrnf (mgr.14162) 28 : cluster [DBG] pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:10:52.940 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:52.901+0000 7f13136cc700 1 -- 172.21.15.132:0/95180703 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f130c0f5410 msgr2=0x7f130c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:10:52.940 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:52.901+0000 7f13136cc700 1 --2- 172.21.15.132:0/95180703 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f130c0f5410 0x7f130c0f5830 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f12fc009fb0 tx=0x7f12fc009b30).stop 2024-02-20T00:10:52.940 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:52.901+0000 7f13136cc700 1 -- 172.21.15.132:0/95180703 shutdown_connections 2024-02-20T00:10:52.940 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:52.901+0000 7f13136cc700 1 --2- 172.21.15.132:0/95180703 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f130c0f5410 0x7f130c0f5830 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:10:52.940 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:52.901+0000 7f13136cc700 1 -- 172.21.15.132:0/95180703 >> 172.21.15.132:0/95180703 conn(0x7f130c0f0b70 msgr2=0x7f130c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:10:52.941 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:52.901+0000 7f13136cc700 1 -- 172.21.15.132:0/95180703 shutdown_connections 2024-02-20T00:10:52.941 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:52.901+0000 7f13136cc700 1 -- 172.21.15.132:0/95180703 wait complete. 2024-02-20T00:10:52.941 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:52.902+0000 7f13136cc700 1 Processor -- start 2024-02-20T00:10:52.941 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:52.902+0000 7f13136cc700 1 -- start start 2024-02-20T00:10:52.941 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:52.902+0000 7f13136cc700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f130c0f5410 0x7f130c1056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:10:52.941 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:52.902+0000 7f13136cc700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f130c105bd0 con 0x7f130c0f5410 2024-02-20T00:10:52.941 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:52.902+0000 7f1311468700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f130c0f5410 0x7f130c1056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:10:52.941 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:52.902+0000 7f1311468700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f130c0f5410 0x7f130c1056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:50888/0 (socket says 172.21.15.132:50888) 2024-02-20T00:10:52.941 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:52.902+0000 7f1311468700 1 -- 172.21.15.132:0/527967714 learned_addr learned my addr 172.21.15.132:0/527967714 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:10:52.941 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:52.903+0000 7f1311468700 1 -- 172.21.15.132:0/527967714 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12fc005040 con 0x7f130c0f5410 2024-02-20T00:10:52.941 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:52.903+0000 7f1311468700 1 --2- 172.21.15.132:0/527967714 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f130c0f5410 0x7f130c1056e0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f12fc005010 tx=0x7f12fc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:10:52.941 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:52.903+0000 7f1301ffb700 1 -- 172.21.15.132:0/527967714 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f12fc009320 con 0x7f130c0f5410 2024-02-20T00:10:52.941 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:52.903+0000 7f13136cc700 1 -- 172.21.15.132:0/527967714 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f130c105dd0 con 0x7f130c0f5410 2024-02-20T00:10:52.941 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:52.903+0000 7f1301ffb700 1 -- 172.21.15.132:0/527967714 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f12fc013380 con 0x7f130c0f5410 2024-02-20T00:10:52.941 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:52.903+0000 7f1301ffb700 1 -- 172.21.15.132:0/527967714 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f12fc014e50 con 0x7f130c0f5410 2024-02-20T00:10:52.942 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:52.903+0000 7f13136cc700 1 -- 172.21.15.132:0/527967714 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f130c106260 con 0x7f130c0f5410 2024-02-20T00:10:52.942 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:52.904+0000 7f13136cc700 1 -- 172.21.15.132:0/527967714 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f130c045ad0 con 0x7f130c0f5410 2024-02-20T00:10:52.942 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:52.904+0000 7f1301ffb700 1 -- 172.21.15.132:0/527967714 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f12fc0095f0 con 0x7f130c0f5410 2024-02-20T00:10:52.942 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:52.904+0000 7f1301ffb700 1 --2- 172.21.15.132:0/527967714 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f12f802ff10 0x7f12f80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:10:52.942 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:52.904+0000 7f1301ffb700 1 -- 172.21.15.132:0/527967714 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f12fc040e90 con 0x7f130c0f5410 2024-02-20T00:10:52.942 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:52.905+0000 7f1310c67700 1 --2- 172.21.15.132:0/527967714 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f12f802ff10 0x7f12f80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:10:52.942 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:52.905+0000 7f1310c67700 1 --2- 172.21.15.132:0/527967714 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f12f802ff10 0x7f12f80323d0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f13080096a0 tx=0x7f1308006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:10:52.942 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:52.909+0000 7f1301ffb700 1 -- 172.21.15.132:0/527967714 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f12fc014420 con 0x7f130c0f5410 2024-02-20T00:10:53.119 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:53.117+0000 7f13136cc700 1 -- 172.21.15.132:0/527967714 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f130c1070b0 con 0x7f130c0f5410 2024-02-20T00:10:53.120 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:53.118+0000 7f1301ffb700 1 -- 172.21.15.132:0/527967714 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f12fc010230 con 0x7f130c0f5410 2024-02-20T00:10:53.120 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:10:53.121 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:10:53.121 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:53.120+0000 7f13136cc700 1 -- 172.21.15.132:0/527967714 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f12f802ff10 msgr2=0x7f12f80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:10:53.121 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:53.120+0000 7f13136cc700 1 --2- 172.21.15.132:0/527967714 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f12f802ff10 0x7f12f80323d0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f13080096a0 tx=0x7f1308006b40).stop 2024-02-20T00:10:53.121 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:53.120+0000 7f13136cc700 1 -- 172.21.15.132:0/527967714 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f130c0f5410 msgr2=0x7f130c1056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:10:53.122 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:53.120+0000 7f13136cc700 1 --2- 172.21.15.132:0/527967714 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f130c0f5410 0x7f130c1056e0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f12fc005010 tx=0x7f12fc00b040).stop 2024-02-20T00:10:53.122 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:53.120+0000 7f13136cc700 1 -- 172.21.15.132:0/527967714 shutdown_connections 2024-02-20T00:10:53.122 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:53.120+0000 7f13136cc700 1 --2- 172.21.15.132:0/527967714 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f12f802ff10 0x7f12f80323d0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:10:53.122 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:53.120+0000 7f13136cc700 1 --2- 172.21.15.132:0/527967714 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f130c0f5410 0x7f130c1056e0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:10:53.122 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:53.120+0000 7f13136cc700 1 -- 172.21.15.132:0/527967714 >> 172.21.15.132:0/527967714 conn(0x7f130c0f0b70 msgr2=0x7f130c0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:10:53.122 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:53.120+0000 7f13136cc700 1 -- 172.21.15.132:0/527967714 shutdown_connections 2024-02-20T00:10:53.122 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:10:53.120+0000 7f13136cc700 1 -- 172.21.15.132:0/527967714 wait complete. 2024-02-20T00:10:53.122 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:10:54.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:10:53 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:10:52.844065+0000 mgr.smithi084.bmqrnf (mgr.14162) 29 : cluster [DBG] pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:10:54.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:10:53 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:10:53.118775+0000 mon.smithi084 (mon.0) 124 : audit [DBG] from='client.? 172.21.15.132:0/527967714' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:10:56.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:10:55 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:10:54.844341+0000 mgr.smithi084.bmqrnf (mgr.14162) 30 : cluster [DBG] pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:10:58.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:10:57 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:10:56.844644+0000 mgr.smithi084.bmqrnf (mgr.14162) 31 : cluster [DBG] pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:10:58.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:10:57 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:10:56.970030+0000 mon.smithi084 (mon.0) 125 : audit [INF] from='mgr.14162 172.21.15.84:0/2152933216' entity='mgr.smithi084.bmqrnf' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi084.bmqrnf/mirror_snapshot_schedule"}]: dispatch 2024-02-20T00:10:58.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:10:57 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:10:56.976693+0000 mon.smithi084 (mon.0) 126 : audit [INF] from='mgr.14162 172.21.15.84:0/2152933216' entity='mgr.smithi084.bmqrnf' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi084.bmqrnf/trash_purge_schedule"}]: dispatch 2024-02-20T00:10:58.959 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:10:58.960 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:11:00.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:10:59 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:10:58.844935+0000 mgr.smithi084.bmqrnf (mgr.14162) 32 : cluster [DBG] pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:11:02.107 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:11:01 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:11:00.845285+0000 mgr.smithi084.bmqrnf (mgr.14162) 33 : cluster [DBG] pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:11:04.128 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:04.126+0000 7fe9c4257700 1 -- 172.21.15.132:0/1906044759 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe9bc0f6510 msgr2=0x7fe9bc0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:11:04.128 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:04.126+0000 7fe9c4257700 1 --2- 172.21.15.132:0/1906044759 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe9bc0f6510 0x7fe9bc0f6930 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7fe9b8009fb0 tx=0x7fe9b8009b30).stop 2024-02-20T00:11:04.129 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:04.126+0000 7fe9c4257700 1 -- 172.21.15.132:0/1906044759 shutdown_connections 2024-02-20T00:11:04.129 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:04.126+0000 7fe9c4257700 1 --2- 172.21.15.132:0/1906044759 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe9bc0f6510 0x7fe9bc0f6930 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:11:04.129 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:04.126+0000 7fe9c4257700 1 -- 172.21.15.132:0/1906044759 >> 172.21.15.132:0/1906044759 conn(0x7fe9bc0f1c70 msgr2=0x7fe9bc0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:11:04.129 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:04.126+0000 7fe9c4257700 1 -- 172.21.15.132:0/1906044759 shutdown_connections 2024-02-20T00:11:04.129 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:04.126+0000 7fe9c4257700 1 -- 172.21.15.132:0/1906044759 wait complete. 2024-02-20T00:11:04.129 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:04.127+0000 7fe9c4257700 1 Processor -- start 2024-02-20T00:11:04.129 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:04.127+0000 7fe9c4257700 1 -- start start 2024-02-20T00:11:04.129 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:04.127+0000 7fe9c4257700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe9bc0f6510 0x7fe9bc1067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:11:04.129 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:04.127+0000 7fe9c4257700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9bc106cd0 con 0x7fe9bc0f6510 2024-02-20T00:11:04.129 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:04.127+0000 7fe9c1ff3700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe9bc0f6510 0x7fe9bc1067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:11:04.129 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:04.127+0000 7fe9c1ff3700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe9bc0f6510 0x7fe9bc1067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:33866/0 (socket says 172.21.15.132:33866) 2024-02-20T00:11:04.129 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:04.127+0000 7fe9c1ff3700 1 -- 172.21.15.132:0/3669442951 learned_addr learned my addr 172.21.15.132:0/3669442951 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:11:04.131 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:04.128+0000 7fe9c1ff3700 1 -- 172.21.15.132:0/3669442951 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9b8005040 con 0x7fe9bc0f6510 2024-02-20T00:11:04.131 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:04.128+0000 7fe9c1ff3700 1 --2- 172.21.15.132:0/3669442951 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe9bc0f6510 0x7fe9bc1067e0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7fe9b8005010 tx=0x7fe9b800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:11:04.131 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:04.128+0000 7fe9b27fc700 1 -- 172.21.15.132:0/3669442951 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe9b8009320 con 0x7fe9bc0f6510 2024-02-20T00:11:04.131 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:04.128+0000 7fe9c4257700 1 -- 172.21.15.132:0/3669442951 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9bc106ed0 con 0x7fe9bc0f6510 2024-02-20T00:11:04.131 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:04.128+0000 7fe9b27fc700 1 -- 172.21.15.132:0/3669442951 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe9b8013380 con 0x7fe9bc0f6510 2024-02-20T00:11:04.131 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:04.128+0000 7fe9b27fc700 1 -- 172.21.15.132:0/3669442951 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe9b8014e50 con 0x7fe9bc0f6510 2024-02-20T00:11:04.131 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:04.128+0000 7fe9c4257700 1 -- 172.21.15.132:0/3669442951 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9bc107360 con 0x7fe9bc0f6510 2024-02-20T00:11:04.131 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:04.129+0000 7fe9b27fc700 1 -- 172.21.15.132:0/3669442951 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe9b8014420 con 0x7fe9bc0f6510 2024-02-20T00:11:04.131 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:04.129+0000 7fe9b27fc700 1 --2- 172.21.15.132:0/3669442951 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fe9a802ff10 0x7fe9a80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:11:04.131 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:04.129+0000 7fe9c4257700 1 -- 172.21.15.132:0/3669442951 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9bc045ad0 con 0x7fe9bc0f6510 2024-02-20T00:11:04.132 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:04.129+0000 7fe9b27fc700 1 -- 172.21.15.132:0/3669442951 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe9b8040f90 con 0x7fe9bc0f6510 2024-02-20T00:11:04.132 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:04.129+0000 7fe9c17f2700 1 --2- 172.21.15.132:0/3669442951 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fe9a802ff10 0x7fe9a80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:11:04.132 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:04.129+0000 7fe9c17f2700 1 --2- 172.21.15.132:0/3669442951 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fe9a802ff10 0x7fe9a80323d0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7fe9ac0096a0 tx=0x7fe9ac006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:11:04.135 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:04.134+0000 7fe9b27fc700 1 -- 172.21.15.132:0/3669442951 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe9b8014790 con 0x7fe9bc0f6510 2024-02-20T00:11:04.339 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:04.338+0000 7fe9c4257700 1 -- 172.21.15.132:0/3669442951 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe9bc108210 con 0x7fe9bc0f6510 2024-02-20T00:11:04.340 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:04.339+0000 7fe9b27fc700 1 -- 172.21.15.132:0/3669442951 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe9b801d380 con 0x7fe9bc0f6510 2024-02-20T00:11:04.340 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:11:04.340 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:11:04.342 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:04.340+0000 7fe9c4257700 1 -- 172.21.15.132:0/3669442951 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fe9a802ff10 msgr2=0x7fe9a80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:11:04.342 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:04.340+0000 7fe9c4257700 1 --2- 172.21.15.132:0/3669442951 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fe9a802ff10 0x7fe9a80323d0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7fe9ac0096a0 tx=0x7fe9ac006b40).stop 2024-02-20T00:11:04.342 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:04.340+0000 7fe9c4257700 1 -- 172.21.15.132:0/3669442951 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe9bc0f6510 msgr2=0x7fe9bc1067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:11:04.342 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:04.340+0000 7fe9c4257700 1 --2- 172.21.15.132:0/3669442951 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe9bc0f6510 0x7fe9bc1067e0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7fe9b8005010 tx=0x7fe9b800b040).stop 2024-02-20T00:11:04.342 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:04.340+0000 7fe9c4257700 1 -- 172.21.15.132:0/3669442951 shutdown_connections 2024-02-20T00:11:04.342 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:04.340+0000 7fe9c4257700 1 --2- 172.21.15.132:0/3669442951 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fe9a802ff10 0x7fe9a80323d0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:11:04.342 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:04.340+0000 7fe9c4257700 1 --2- 172.21.15.132:0/3669442951 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe9bc0f6510 0x7fe9bc1067e0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:11:04.342 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:04.340+0000 7fe9c4257700 1 -- 172.21.15.132:0/3669442951 >> 172.21.15.132:0/3669442951 conn(0x7fe9bc0f1c70 msgr2=0x7fe9bc0f26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:11:04.342 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:04.340+0000 7fe9c4257700 1 -- 172.21.15.132:0/3669442951 shutdown_connections 2024-02-20T00:11:04.342 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:04.340+0000 7fe9c4257700 1 -- 172.21.15.132:0/3669442951 wait complete. 2024-02-20T00:11:04.343 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:11:04.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:11:03 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:11:02.845648+0000 mgr.smithi084.bmqrnf (mgr.14162) 34 : cluster [DBG] pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:11:05.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:11:04 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:11:04.339310+0000 mon.smithi084 (mon.0) 127 : audit [DBG] from='client.? 172.21.15.132:0/3669442951' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:11:05.959 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:11:05.960 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:11:06.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:11:05 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:11:04.846038+0000 mgr.smithi084.bmqrnf (mgr.14162) 35 : cluster [DBG] pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:11:07.910 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:07.907+0000 7f8e234a5700 1 -- 172.21.15.132:0/3211725846 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8e1c0f5410 msgr2=0x7f8e1c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:11:07.910 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:07.907+0000 7f8e234a5700 1 --2- 172.21.15.132:0/3211725846 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8e1c0f5410 0x7f8e1c0f5830 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7f8e18009fb0 tx=0x7f8e18009b30).stop 2024-02-20T00:11:07.911 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:07.907+0000 7f8e234a5700 1 -- 172.21.15.132:0/3211725846 shutdown_connections 2024-02-20T00:11:07.911 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:07.907+0000 7f8e234a5700 1 --2- 172.21.15.132:0/3211725846 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8e1c0f5410 0x7f8e1c0f5830 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:11:07.911 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:07.907+0000 7f8e234a5700 1 -- 172.21.15.132:0/3211725846 >> 172.21.15.132:0/3211725846 conn(0x7f8e1c0f0b70 msgr2=0x7f8e1c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:11:07.911 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:07.907+0000 7f8e234a5700 1 -- 172.21.15.132:0/3211725846 shutdown_connections 2024-02-20T00:11:07.911 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:07.907+0000 7f8e234a5700 1 -- 172.21.15.132:0/3211725846 wait complete. 2024-02-20T00:11:07.911 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:07.908+0000 7f8e234a5700 1 Processor -- start 2024-02-20T00:11:07.911 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:07.908+0000 7f8e234a5700 1 -- start start 2024-02-20T00:11:07.911 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:07.908+0000 7f8e234a5700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8e1c0f5410 0x7f8e1c1094e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:11:07.911 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:07.908+0000 7f8e234a5700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e1c060240 con 0x7f8e1c0f5410 2024-02-20T00:11:07.911 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:07.908+0000 7f8e21241700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8e1c0f5410 0x7f8e1c1094e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:11:07.911 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:07.908+0000 7f8e21241700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8e1c0f5410 0x7f8e1c1094e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:50962/0 (socket says 172.21.15.132:50962) 2024-02-20T00:11:07.911 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:07.908+0000 7f8e21241700 1 -- 172.21.15.132:0/2914751907 learned_addr learned my addr 172.21.15.132:0/2914751907 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:11:07.911 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:07.908+0000 7f8e21241700 1 -- 172.21.15.132:0/2914751907 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e18005040 con 0x7f8e1c0f5410 2024-02-20T00:11:07.913 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:07.909+0000 7f8e21241700 1 --2- 172.21.15.132:0/2914751907 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8e1c0f5410 0x7f8e1c1094e0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f8e18005010 tx=0x7f8e1800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:11:07.913 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:07.909+0000 7f8e11ffb700 1 -- 172.21.15.132:0/2914751907 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8e18009320 con 0x7f8e1c0f5410 2024-02-20T00:11:07.913 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:07.909+0000 7f8e234a5700 1 -- 172.21.15.132:0/2914751907 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e1c060440 con 0x7f8e1c0f5410 2024-02-20T00:11:07.913 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:07.909+0000 7f8e11ffb700 1 -- 172.21.15.132:0/2914751907 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8e18013380 con 0x7f8e1c0f5410 2024-02-20T00:11:07.913 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:07.909+0000 7f8e11ffb700 1 -- 172.21.15.132:0/2914751907 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8e18014e50 con 0x7f8e1c0f5410 2024-02-20T00:11:07.913 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:07.909+0000 7f8e234a5700 1 -- 172.21.15.132:0/2914751907 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e1c109b90 con 0x7f8e1c0f5410 2024-02-20T00:11:07.913 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:07.910+0000 7f8e11ffb700 1 -- 172.21.15.132:0/2914751907 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8e180263a0 con 0x7f8e1c0f5410 2024-02-20T00:11:07.913 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:07.910+0000 7f8e11ffb700 1 --2- 172.21.15.132:0/2914751907 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f8e0802ff10 0x7f8e080323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:11:07.913 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:07.910+0000 7f8e11ffb700 1 -- 172.21.15.132:0/2914751907 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8e18040f70 con 0x7f8e1c0f5410 2024-02-20T00:11:07.913 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:07.910+0000 7f8e234a5700 1 -- 172.21.15.132:0/2914751907 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e1c045ad0 con 0x7f8e1c0f5410 2024-02-20T00:11:07.913 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:07.910+0000 7f8e20a40700 1 --2- 172.21.15.132:0/2914751907 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f8e0802ff10 0x7f8e080323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:11:07.913 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:07.911+0000 7f8e20a40700 1 --2- 172.21.15.132:0/2914751907 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f8e0802ff10 0x7f8e080323d0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7f8e0c0096a0 tx=0x7f8e0c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:11:07.917 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:07.915+0000 7f8e11ffb700 1 -- 172.21.15.132:0/2914751907 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8e18012920 con 0x7f8e1c0f5410 2024-02-20T00:11:08.120 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:08.118+0000 7f8e234a5700 1 -- 172.21.15.132:0/2914751907 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8e1c0fafa0 con 0x7f8e1c0f5410 2024-02-20T00:11:08.121 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:08.119+0000 7f8e11ffb700 1 -- 172.21.15.132:0/2914751907 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8e1802b020 con 0x7f8e1c0f5410 2024-02-20T00:11:08.121 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:11:08.123 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:11:08.124 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:08.120+0000 7f8e234a5700 1 -- 172.21.15.132:0/2914751907 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f8e0802ff10 msgr2=0x7f8e080323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:11:08.124 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:08.120+0000 7f8e234a5700 1 --2- 172.21.15.132:0/2914751907 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f8e0802ff10 0x7f8e080323d0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7f8e0c0096a0 tx=0x7f8e0c006b40).stop 2024-02-20T00:11:08.125 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:08.120+0000 7f8e234a5700 1 -- 172.21.15.132:0/2914751907 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8e1c0f5410 msgr2=0x7f8e1c1094e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:11:08.125 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:08.120+0000 7f8e234a5700 1 --2- 172.21.15.132:0/2914751907 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8e1c0f5410 0x7f8e1c1094e0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f8e18005010 tx=0x7f8e1800b040).stop 2024-02-20T00:11:08.125 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:08.120+0000 7f8e234a5700 1 -- 172.21.15.132:0/2914751907 shutdown_connections 2024-02-20T00:11:08.125 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:08.120+0000 7f8e234a5700 1 --2- 172.21.15.132:0/2914751907 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f8e0802ff10 0x7f8e080323d0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:11:08.125 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:08.120+0000 7f8e234a5700 1 --2- 172.21.15.132:0/2914751907 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8e1c0f5410 0x7f8e1c1094e0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:11:08.125 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:08.120+0000 7f8e234a5700 1 -- 172.21.15.132:0/2914751907 >> 172.21.15.132:0/2914751907 conn(0x7f8e1c0f0b70 msgr2=0x7f8e1c0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:11:08.125 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:08.121+0000 7f8e234a5700 1 -- 172.21.15.132:0/2914751907 shutdown_connections 2024-02-20T00:11:08.125 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:08.121+0000 7f8e234a5700 1 -- 172.21.15.132:0/2914751907 wait complete. 2024-02-20T00:11:08.125 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:11:08.359 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:11:07 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:11:06.846473+0000 mgr.smithi084.bmqrnf (mgr.14162) 36 : cluster [DBG] pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:11:09.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:11:08 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:11:08.120571+0000 mon.smithi084 (mon.0) 128 : audit [DBG] from='client.? 172.21.15.132:0/2914751907' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:11:09.764 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:11:09.764 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:11:10.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:11:09 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:11:08.846902+0000 mgr.smithi084.bmqrnf (mgr.14162) 37 : cluster [DBG] pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:11:11.651 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:11.647+0000 7f3b5ba42700 1 -- 172.21.15.132:0/221800944 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3b540f5410 msgr2=0x7f3b540f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:11:11.651 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:11.647+0000 7f3b5ba42700 1 --2- 172.21.15.132:0/221800944 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3b540f5410 0x7f3b540f5830 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f3b50009fb0 tx=0x7f3b50009b30).stop 2024-02-20T00:11:11.651 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:11.647+0000 7f3b5ba42700 1 -- 172.21.15.132:0/221800944 shutdown_connections 2024-02-20T00:11:11.651 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:11.647+0000 7f3b5ba42700 1 --2- 172.21.15.132:0/221800944 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3b540f5410 0x7f3b540f5830 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:11:11.651 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:11.647+0000 7f3b5ba42700 1 -- 172.21.15.132:0/221800944 >> 172.21.15.132:0/221800944 conn(0x7f3b540f0b70 msgr2=0x7f3b540f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:11:11.651 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:11.647+0000 7f3b5ba42700 1 -- 172.21.15.132:0/221800944 shutdown_connections 2024-02-20T00:11:11.651 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:11.647+0000 7f3b5ba42700 1 -- 172.21.15.132:0/221800944 wait complete. 2024-02-20T00:11:11.651 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:11.647+0000 7f3b5ba42700 1 Processor -- start 2024-02-20T00:11:11.651 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:11.647+0000 7f3b5ba42700 1 -- start start 2024-02-20T00:11:11.651 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:11.647+0000 7f3b5ba42700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3b540f5410 0x7f3b541056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:11:11.652 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:11.647+0000 7f3b5ba42700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b54105bd0 con 0x7f3b540f5410 2024-02-20T00:11:11.652 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:11.648+0000 7f3b597de700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3b540f5410 0x7f3b541056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:11:11.652 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:11.648+0000 7f3b597de700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3b540f5410 0x7f3b541056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:50978/0 (socket says 172.21.15.132:50978) 2024-02-20T00:11:11.652 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:11.648+0000 7f3b597de700 1 -- 172.21.15.132:0/2573177842 learned_addr learned my addr 172.21.15.132:0/2573177842 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:11:11.652 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:11.648+0000 7f3b597de700 1 -- 172.21.15.132:0/2573177842 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b50005040 con 0x7f3b540f5410 2024-02-20T00:11:11.652 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:11.648+0000 7f3b597de700 1 --2- 172.21.15.132:0/2573177842 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3b540f5410 0x7f3b541056e0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f3b50005010 tx=0x7f3b5000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:11:11.652 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:11.648+0000 7f3b49ffb700 1 -- 172.21.15.132:0/2573177842 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3b50009320 con 0x7f3b540f5410 2024-02-20T00:11:11.652 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:11.648+0000 7f3b49ffb700 1 -- 172.21.15.132:0/2573177842 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3b50013380 con 0x7f3b540f5410 2024-02-20T00:11:11.652 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:11.648+0000 7f3b5ba42700 1 -- 172.21.15.132:0/2573177842 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b54105dd0 con 0x7f3b540f5410 2024-02-20T00:11:11.652 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:11.648+0000 7f3b49ffb700 1 -- 172.21.15.132:0/2573177842 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3b50014e50 con 0x7f3b540f5410 2024-02-20T00:11:11.652 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:11.649+0000 7f3b5ba42700 1 -- 172.21.15.132:0/2573177842 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b54106260 con 0x7f3b540f5410 2024-02-20T00:11:11.653 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:11.649+0000 7f3b49ffb700 1 -- 172.21.15.132:0/2573177842 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3b50014420 con 0x7f3b540f5410 2024-02-20T00:11:11.653 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:11.650+0000 7f3b49ffb700 1 --2- 172.21.15.132:0/2573177842 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f3b4002ff10 0x7f3b400323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:11:11.653 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:11.650+0000 7f3b49ffb700 1 -- 172.21.15.132:0/2573177842 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3b50040e60 con 0x7f3b540f5410 2024-02-20T00:11:11.653 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:11.650+0000 7f3b5ba42700 1 -- 172.21.15.132:0/2573177842 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b54045ad0 con 0x7f3b540f5410 2024-02-20T00:11:11.653 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:11.650+0000 7f3b58fdd700 1 --2- 172.21.15.132:0/2573177842 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f3b4002ff10 0x7f3b400323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:11:11.654 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:11.650+0000 7f3b58fdd700 1 --2- 172.21.15.132:0/2573177842 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f3b4002ff10 0x7f3b400323d0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f3b440096a0 tx=0x7f3b44006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:11:11.657 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:11.655+0000 7f3b49ffb700 1 -- 172.21.15.132:0/2573177842 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3b500095f0 con 0x7f3b540f5410 2024-02-20T00:11:11.860 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:11.858+0000 7f3b5ba42700 1 -- 172.21.15.132:0/2573177842 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3b54107110 con 0x7f3b540f5410 2024-02-20T00:11:11.861 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:11.859+0000 7f3b49ffb700 1 -- 172.21.15.132:0/2573177842 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3b5001db70 con 0x7f3b540f5410 2024-02-20T00:11:11.861 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:11:11.862 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:11:11.863 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:11.861+0000 7f3b5ba42700 1 -- 172.21.15.132:0/2573177842 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f3b4002ff10 msgr2=0x7f3b400323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:11:11.864 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:11.861+0000 7f3b5ba42700 1 --2- 172.21.15.132:0/2573177842 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f3b4002ff10 0x7f3b400323d0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f3b440096a0 tx=0x7f3b44006b40).stop 2024-02-20T00:11:11.864 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:11.861+0000 7f3b5ba42700 1 -- 172.21.15.132:0/2573177842 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3b540f5410 msgr2=0x7f3b541056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:11:11.864 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:11.861+0000 7f3b5ba42700 1 --2- 172.21.15.132:0/2573177842 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3b540f5410 0x7f3b541056e0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f3b50005010 tx=0x7f3b5000b040).stop 2024-02-20T00:11:11.864 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:11.861+0000 7f3b5ba42700 1 -- 172.21.15.132:0/2573177842 shutdown_connections 2024-02-20T00:11:11.864 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:11.861+0000 7f3b5ba42700 1 --2- 172.21.15.132:0/2573177842 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f3b4002ff10 0x7f3b400323d0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:11:11.864 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:11.861+0000 7f3b5ba42700 1 --2- 172.21.15.132:0/2573177842 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3b540f5410 0x7f3b541056e0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:11:11.864 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:11.861+0000 7f3b5ba42700 1 -- 172.21.15.132:0/2573177842 >> 172.21.15.132:0/2573177842 conn(0x7f3b540f0b70 msgr2=0x7f3b540f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:11:11.864 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:11.861+0000 7f3b5ba42700 1 -- 172.21.15.132:0/2573177842 shutdown_connections 2024-02-20T00:11:11.864 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:11.861+0000 7f3b5ba42700 1 -- 172.21.15.132:0/2573177842 wait complete. 2024-02-20T00:11:11.864 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:11:12.107 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:11:11 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:11:10.847225+0000 mgr.smithi084.bmqrnf (mgr.14162) 38 : cluster [DBG] pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:11:13.107 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:11:12 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:11:11.860630+0000 mon.smithi084 (mon.0) 129 : audit [DBG] from='client.? 172.21.15.132:0/2573177842' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:11:13.451 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:11:13.452 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:11:14.107 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:11:13 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:11:12.847456+0000 mgr.smithi084.bmqrnf (mgr.14162) 39 : cluster [DBG] pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:11:15.270 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:15.266+0000 7f983a163700 1 -- 172.21.15.132:0/625122804 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9834060280 msgr2=0x7f98340f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:11:15.270 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:15.266+0000 7f983a163700 1 --2- 172.21.15.132:0/625122804 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9834060280 0x7f98340f90f0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7f9828009fb0 tx=0x7f9828009b30).stop 2024-02-20T00:11:15.270 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:15.266+0000 7f983a163700 1 -- 172.21.15.132:0/625122804 shutdown_connections 2024-02-20T00:11:15.270 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:15.266+0000 7f983a163700 1 --2- 172.21.15.132:0/625122804 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9834060280 0x7f98340f90f0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:11:15.270 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:15.266+0000 7f983a163700 1 -- 172.21.15.132:0/625122804 >> 172.21.15.132:0/625122804 conn(0x7f98340f0b70 msgr2=0x7f98340f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:11:15.271 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:15.266+0000 7f983a163700 1 -- 172.21.15.132:0/625122804 shutdown_connections 2024-02-20T00:11:15.271 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:15.266+0000 7f983a163700 1 -- 172.21.15.132:0/625122804 wait complete. 2024-02-20T00:11:15.271 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:15.267+0000 7f983a163700 1 Processor -- start 2024-02-20T00:11:15.271 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:15.267+0000 7f983a163700 1 -- start start 2024-02-20T00:11:15.271 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:15.267+0000 7f983a163700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9834060280 0x7f9834108990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:11:15.271 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:15.267+0000 7f983a163700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9834108e80 con 0x7f9834060280 2024-02-20T00:11:15.271 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:15.267+0000 7f98337fe700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9834060280 0x7f9834108990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:11:15.271 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:15.267+0000 7f98337fe700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9834060280 0x7f9834108990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:50990/0 (socket says 172.21.15.132:50990) 2024-02-20T00:11:15.271 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:15.267+0000 7f98337fe700 1 -- 172.21.15.132:0/2039800223 learned_addr learned my addr 172.21.15.132:0/2039800223 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:11:15.271 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:15.268+0000 7f98337fe700 1 -- 172.21.15.132:0/2039800223 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9828005040 con 0x7f9834060280 2024-02-20T00:11:15.271 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:15.268+0000 7f98337fe700 1 --2- 172.21.15.132:0/2039800223 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9834060280 0x7f9834108990 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f98280089f0 tx=0x7f982800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:11:15.271 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:15.268+0000 7f981bfff700 1 -- 172.21.15.132:0/2039800223 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9828009320 con 0x7f9834060280 2024-02-20T00:11:15.271 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:15.268+0000 7f983a163700 1 -- 172.21.15.132:0/2039800223 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9834109080 con 0x7f9834060280 2024-02-20T00:11:15.271 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:15.268+0000 7f981bfff700 1 -- 172.21.15.132:0/2039800223 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9828013380 con 0x7f9834060280 2024-02-20T00:11:15.272 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:15.268+0000 7f981bfff700 1 -- 172.21.15.132:0/2039800223 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9828014e20 con 0x7f9834060280 2024-02-20T00:11:15.272 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:15.268+0000 7f983a163700 1 -- 172.21.15.132:0/2039800223 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9834109510 con 0x7f9834060280 2024-02-20T00:11:15.272 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:15.269+0000 7f981bfff700 1 -- 172.21.15.132:0/2039800223 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9828014420 con 0x7f9834060280 2024-02-20T00:11:15.273 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:15.269+0000 7f983a163700 1 -- 172.21.15.132:0/2039800223 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9834045ad0 con 0x7f9834060280 2024-02-20T00:11:15.273 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:15.269+0000 7f981bfff700 1 --2- 172.21.15.132:0/2039800223 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f981c02ff10 0x7f981c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:11:15.273 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:15.269+0000 7f981bfff700 1 -- 172.21.15.132:0/2039800223 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9828040f50 con 0x7f9834060280 2024-02-20T00:11:15.273 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:15.269+0000 7f9832ffd700 1 --2- 172.21.15.132:0/2039800223 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f981c02ff10 0x7f981c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:11:15.273 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:15.270+0000 7f9832ffd700 1 --2- 172.21.15.132:0/2039800223 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f981c02ff10 0x7f981c0323d0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f98240096a0 tx=0x7f9824006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:11:15.276 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:15.274+0000 7f981bfff700 1 -- 172.21.15.132:0/2039800223 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f982806d2b0 con 0x7f9834060280 2024-02-20T00:11:15.481 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:15.479+0000 7f983a163700 1 -- 172.21.15.132:0/2039800223 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f983410a3c0 con 0x7f9834060280 2024-02-20T00:11:15.482 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:15.480+0000 7f981bfff700 1 -- 172.21.15.132:0/2039800223 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f98280095f0 con 0x7f9834060280 2024-02-20T00:11:15.482 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:11:15.483 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:11:15.484 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:15.482+0000 7f983a163700 1 -- 172.21.15.132:0/2039800223 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f981c02ff10 msgr2=0x7f981c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:11:15.484 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:15.482+0000 7f983a163700 1 --2- 172.21.15.132:0/2039800223 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f981c02ff10 0x7f981c0323d0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f98240096a0 tx=0x7f9824006b40).stop 2024-02-20T00:11:15.485 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:15.482+0000 7f983a163700 1 -- 172.21.15.132:0/2039800223 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9834060280 msgr2=0x7f9834108990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:11:15.485 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:15.482+0000 7f983a163700 1 --2- 172.21.15.132:0/2039800223 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9834060280 0x7f9834108990 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f98280089f0 tx=0x7f982800b040).stop 2024-02-20T00:11:15.485 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:15.482+0000 7f983a163700 1 -- 172.21.15.132:0/2039800223 shutdown_connections 2024-02-20T00:11:15.485 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:15.482+0000 7f983a163700 1 --2- 172.21.15.132:0/2039800223 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f981c02ff10 0x7f981c0323d0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:11:15.485 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:15.482+0000 7f983a163700 1 --2- 172.21.15.132:0/2039800223 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9834060280 0x7f9834108990 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:11:15.485 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:15.482+0000 7f983a163700 1 -- 172.21.15.132:0/2039800223 >> 172.21.15.132:0/2039800223 conn(0x7f98340f0b70 msgr2=0x7f98340f1520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:11:15.485 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:15.482+0000 7f983a163700 1 -- 172.21.15.132:0/2039800223 shutdown_connections 2024-02-20T00:11:15.485 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:15.482+0000 7f983a163700 1 -- 172.21.15.132:0/2039800223 wait complete. 2024-02-20T00:11:15.485 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:11:16.107 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:11:15 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:11:14.847646+0000 mgr.smithi084.bmqrnf (mgr.14162) 40 : cluster [DBG] pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:11:16.107 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:11:15 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:11:15.481709+0000 mon.smithi084 (mon.0) 130 : audit [DBG] from='client.? 172.21.15.132:0/2039800223' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:11:17.088 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:11:17.089 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:11:18.107 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:11:17 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:11:16.847853+0000 mgr.smithi084.bmqrnf (mgr.14162) 41 : cluster [DBG] pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:11:18.924 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:18.921+0000 7f273111f700 1 -- 172.21.15.132:0/3959570111 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f272c0f62a0 msgr2=0x7f272c0f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:11:18.924 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:18.921+0000 7f273111f700 1 --2- 172.21.15.132:0/3959570111 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f272c0f62a0 0x7f272c0f66c0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f271c009fb0 tx=0x7f271c009b30).stop 2024-02-20T00:11:18.925 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:18.921+0000 7f273111f700 1 -- 172.21.15.132:0/3959570111 shutdown_connections 2024-02-20T00:11:18.925 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:18.921+0000 7f273111f700 1 --2- 172.21.15.132:0/3959570111 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f272c0f62a0 0x7f272c0f66c0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:11:18.925 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:18.921+0000 7f273111f700 1 -- 172.21.15.132:0/3959570111 >> 172.21.15.132:0/3959570111 conn(0x7f272c0f1aa0 msgr2=0x7f272c0f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:11:18.925 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:18.921+0000 7f273111f700 1 -- 172.21.15.132:0/3959570111 shutdown_connections 2024-02-20T00:11:18.925 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:18.921+0000 7f273111f700 1 -- 172.21.15.132:0/3959570111 wait complete. 2024-02-20T00:11:18.925 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:18.922+0000 7f273111f700 1 Processor -- start 2024-02-20T00:11:18.925 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:18.922+0000 7f273111f700 1 -- start start 2024-02-20T00:11:18.925 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:18.922+0000 7f273111f700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f272c0f62a0 0x7f272c1065d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:11:18.925 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:18.922+0000 7f273111f700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f272c106ac0 con 0x7f272c0f62a0 2024-02-20T00:11:18.925 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:18.922+0000 7f272bfff700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f272c0f62a0 0x7f272c1065d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:11:18.925 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:18.922+0000 7f272bfff700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f272c0f62a0 0x7f272c1065d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:42596/0 (socket says 172.21.15.132:42596) 2024-02-20T00:11:18.925 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:18.922+0000 7f272bfff700 1 -- 172.21.15.132:0/2596656724 learned_addr learned my addr 172.21.15.132:0/2596656724 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:11:18.927 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:18.923+0000 7f272bfff700 1 -- 172.21.15.132:0/2596656724 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f271c005040 con 0x7f272c0f62a0 2024-02-20T00:11:18.927 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:18.923+0000 7f272bfff700 1 --2- 172.21.15.132:0/2596656724 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f272c0f62a0 0x7f272c1065d0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f271c0089f0 tx=0x7f271c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:11:18.927 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:18.923+0000 7f2728ff9700 1 -- 172.21.15.132:0/2596656724 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f271c009320 con 0x7f272c0f62a0 2024-02-20T00:11:18.927 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:18.923+0000 7f273111f700 1 -- 172.21.15.132:0/2596656724 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f272c106cc0 con 0x7f272c0f62a0 2024-02-20T00:11:18.927 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:18.923+0000 7f2728ff9700 1 -- 172.21.15.132:0/2596656724 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f271c013380 con 0x7f272c0f62a0 2024-02-20T00:11:18.927 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:18.923+0000 7f2728ff9700 1 -- 172.21.15.132:0/2596656724 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f271c014e20 con 0x7f272c0f62a0 2024-02-20T00:11:18.928 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:18.923+0000 7f273111f700 1 -- 172.21.15.132:0/2596656724 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f272c107150 con 0x7f272c0f62a0 2024-02-20T00:11:18.928 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:18.924+0000 7f2728ff9700 1 -- 172.21.15.132:0/2596656724 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f271c0095f0 con 0x7f272c0f62a0 2024-02-20T00:11:18.928 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:18.924+0000 7f273111f700 1 -- 172.21.15.132:0/2596656724 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f272c045ad0 con 0x7f272c0f62a0 2024-02-20T00:11:18.928 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:18.924+0000 7f2728ff9700 1 --2- 172.21.15.132:0/2596656724 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f271402ff10 0x7f27140323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:11:18.928 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:18.924+0000 7f2728ff9700 1 -- 172.21.15.132:0/2596656724 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f271c040f80 con 0x7f272c0f62a0 2024-02-20T00:11:18.928 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:18.924+0000 7f272b7fe700 1 --2- 172.21.15.132:0/2596656724 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f271402ff10 0x7f27140323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:11:18.928 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:18.925+0000 7f272b7fe700 1 --2- 172.21.15.132:0/2596656724 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f271402ff10 0x7f27140323d0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f27200096a0 tx=0x7f2720006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:11:18.931 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:18.929+0000 7f2728ff9700 1 -- 172.21.15.132:0/2596656724 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f271c014420 con 0x7f272c0f62a0 2024-02-20T00:11:19.135 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:19.133+0000 7f273111f700 1 -- 172.21.15.132:0/2596656724 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f272c107fa0 con 0x7f272c0f62a0 2024-02-20T00:11:19.136 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:19.133+0000 7f2728ff9700 1 -- 172.21.15.132:0/2596656724 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f271c010230 con 0x7f272c0f62a0 2024-02-20T00:11:19.136 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:11:19.136 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:11:19.137 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:19.135+0000 7f273111f700 1 -- 172.21.15.132:0/2596656724 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f271402ff10 msgr2=0x7f27140323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:11:19.138 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:19.135+0000 7f273111f700 1 --2- 172.21.15.132:0/2596656724 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f271402ff10 0x7f27140323d0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f27200096a0 tx=0x7f2720006b40).stop 2024-02-20T00:11:19.138 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:19.135+0000 7f273111f700 1 -- 172.21.15.132:0/2596656724 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f272c0f62a0 msgr2=0x7f272c1065d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:11:19.138 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:19.135+0000 7f273111f700 1 --2- 172.21.15.132:0/2596656724 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f272c0f62a0 0x7f272c1065d0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f271c0089f0 tx=0x7f271c00b040).stop 2024-02-20T00:11:19.138 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:19.135+0000 7f273111f700 1 -- 172.21.15.132:0/2596656724 shutdown_connections 2024-02-20T00:11:19.138 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:19.135+0000 7f273111f700 1 --2- 172.21.15.132:0/2596656724 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f271402ff10 0x7f27140323d0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:11:19.138 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:19.135+0000 7f273111f700 1 --2- 172.21.15.132:0/2596656724 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f272c0f62a0 0x7f272c1065d0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:11:19.138 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:19.135+0000 7f273111f700 1 -- 172.21.15.132:0/2596656724 >> 172.21.15.132:0/2596656724 conn(0x7f272c0f1aa0 msgr2=0x7f272c0f2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:11:19.138 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:19.135+0000 7f273111f700 1 -- 172.21.15.132:0/2596656724 shutdown_connections 2024-02-20T00:11:19.138 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:19.135+0000 7f273111f700 1 -- 172.21.15.132:0/2596656724 wait complete. 2024-02-20T00:11:19.139 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:11:20.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:11:19 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:11:18.848094+0000 mgr.smithi084.bmqrnf (mgr.14162) 42 : cluster [DBG] pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:11:20.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:11:19 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:11:19.135148+0000 mon.smithi084 (mon.0) 131 : audit [DBG] from='client.? 172.21.15.132:0/2596656724' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:11:20.702 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:11:20.702 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:11:22.107 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:11:21 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:11:20.848395+0000 mgr.smithi084.bmqrnf (mgr.14162) 43 : cluster [DBG] pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:11:22.511 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:22.508+0000 7fa8cec41700 1 -- 172.21.15.132:0/2961688689 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa8c80f5410 msgr2=0x7fa8c80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:11:22.511 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:22.508+0000 7fa8cec41700 1 --2- 172.21.15.132:0/2961688689 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa8c80f5410 0x7fa8c80f5830 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7fa8bc009fb0 tx=0x7fa8bc009b30).stop 2024-02-20T00:11:22.511 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:22.508+0000 7fa8cec41700 1 -- 172.21.15.132:0/2961688689 shutdown_connections 2024-02-20T00:11:22.511 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:22.508+0000 7fa8cec41700 1 --2- 172.21.15.132:0/2961688689 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa8c80f5410 0x7fa8c80f5830 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:11:22.511 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:22.508+0000 7fa8cec41700 1 -- 172.21.15.132:0/2961688689 >> 172.21.15.132:0/2961688689 conn(0x7fa8c80f0b70 msgr2=0x7fa8c80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:11:22.511 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:22.508+0000 7fa8cec41700 1 -- 172.21.15.132:0/2961688689 shutdown_connections 2024-02-20T00:11:22.511 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:22.508+0000 7fa8cec41700 1 -- 172.21.15.132:0/2961688689 wait complete. 2024-02-20T00:11:22.511 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:22.508+0000 7fa8cec41700 1 Processor -- start 2024-02-20T00:11:22.511 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:22.508+0000 7fa8cec41700 1 -- start start 2024-02-20T00:11:22.512 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:22.509+0000 7fa8cec41700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa8c80f5410 0x7fa8c8109500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:11:22.512 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:22.509+0000 7fa8cec41700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8c8060220 con 0x7fa8c80f5410 2024-02-20T00:11:22.512 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:22.509+0000 7fa8cc9dd700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa8c80f5410 0x7fa8c8109500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:11:22.513 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:22.509+0000 7fa8cc9dd700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa8c80f5410 0x7fa8c8109500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:42608/0 (socket says 172.21.15.132:42608) 2024-02-20T00:11:22.513 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:22.509+0000 7fa8cc9dd700 1 -- 172.21.15.132:0/1630263781 learned_addr learned my addr 172.21.15.132:0/1630263781 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:11:22.513 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:22.510+0000 7fa8cc9dd700 1 -- 172.21.15.132:0/1630263781 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8bc005040 con 0x7fa8c80f5410 2024-02-20T00:11:22.513 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:22.510+0000 7fa8cc9dd700 1 --2- 172.21.15.132:0/1630263781 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa8c80f5410 0x7fa8c8109500 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7fa8bc005010 tx=0x7fa8bc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:11:22.513 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:22.510+0000 7fa8c57fa700 1 -- 172.21.15.132:0/1630263781 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa8bc009320 con 0x7fa8c80f5410 2024-02-20T00:11:22.513 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:22.510+0000 7fa8cec41700 1 -- 172.21.15.132:0/1630263781 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8c8060420 con 0x7fa8c80f5410 2024-02-20T00:11:22.513 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:22.510+0000 7fa8c57fa700 1 -- 172.21.15.132:0/1630263781 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa8bc013380 con 0x7fa8c80f5410 2024-02-20T00:11:22.513 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:22.510+0000 7fa8c57fa700 1 -- 172.21.15.132:0/1630263781 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa8bc014e50 con 0x7fa8c80f5410 2024-02-20T00:11:22.513 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:22.510+0000 7fa8cec41700 1 -- 172.21.15.132:0/1630263781 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8c8109bb0 con 0x7fa8c80f5410 2024-02-20T00:11:22.514 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:22.511+0000 7fa8c57fa700 1 -- 172.21.15.132:0/1630263781 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa8bc014420 con 0x7fa8c80f5410 2024-02-20T00:11:22.514 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:22.511+0000 7fa8cec41700 1 -- 172.21.15.132:0/1630263781 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8c8059100 con 0x7fa8c80f5410 2024-02-20T00:11:22.514 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:22.511+0000 7fa8c57fa700 1 --2- 172.21.15.132:0/1630263781 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fa8b802ff10 0x7fa8b80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:11:22.514 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:22.511+0000 7fa8c57fa700 1 -- 172.21.15.132:0/1630263781 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa8bc040e30 con 0x7fa8c80f5410 2024-02-20T00:11:22.514 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:22.511+0000 7fa8c7fff700 1 --2- 172.21.15.132:0/1630263781 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fa8b802ff10 0x7fa8b80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:11:22.514 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:22.512+0000 7fa8c7fff700 1 --2- 172.21.15.132:0/1630263781 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fa8b802ff10 0x7fa8b80323d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7fa8b40096a0 tx=0x7fa8b4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:11:22.518 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:22.516+0000 7fa8c57fa700 1 -- 172.21.15.132:0/1630263781 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa8bc0095f0 con 0x7fa8c80f5410 2024-02-20T00:11:22.722 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:22.720+0000 7fa8cec41700 1 -- 172.21.15.132:0/1630263781 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa8c8045ad0 con 0x7fa8c80f5410 2024-02-20T00:11:22.723 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:22.721+0000 7fa8c57fa700 1 -- 172.21.15.132:0/1630263781 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa8bc01db70 con 0x7fa8c80f5410 2024-02-20T00:11:22.723 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:11:22.724 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:11:22.725 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:22.723+0000 7fa8cec41700 1 -- 172.21.15.132:0/1630263781 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fa8b802ff10 msgr2=0x7fa8b80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:11:22.725 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:22.723+0000 7fa8cec41700 1 --2- 172.21.15.132:0/1630263781 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fa8b802ff10 0x7fa8b80323d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7fa8b40096a0 tx=0x7fa8b4006b40).stop 2024-02-20T00:11:22.725 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:22.723+0000 7fa8cec41700 1 -- 172.21.15.132:0/1630263781 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa8c80f5410 msgr2=0x7fa8c8109500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:11:22.725 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:22.723+0000 7fa8cec41700 1 --2- 172.21.15.132:0/1630263781 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa8c80f5410 0x7fa8c8109500 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7fa8bc005010 tx=0x7fa8bc00b040).stop 2024-02-20T00:11:22.725 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:22.723+0000 7fa8cec41700 1 -- 172.21.15.132:0/1630263781 shutdown_connections 2024-02-20T00:11:22.725 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:22.723+0000 7fa8cec41700 1 --2- 172.21.15.132:0/1630263781 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fa8b802ff10 0x7fa8b80323d0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:11:22.726 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:22.723+0000 7fa8cec41700 1 --2- 172.21.15.132:0/1630263781 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa8c80f5410 0x7fa8c8109500 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:11:22.726 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:22.723+0000 7fa8cec41700 1 -- 172.21.15.132:0/1630263781 >> 172.21.15.132:0/1630263781 conn(0x7fa8c80f0b70 msgr2=0x7fa8c80f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:11:22.726 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:22.723+0000 7fa8cec41700 1 -- 172.21.15.132:0/1630263781 shutdown_connections 2024-02-20T00:11:22.726 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:22.723+0000 7fa8cec41700 1 -- 172.21.15.132:0/1630263781 wait complete. 2024-02-20T00:11:22.726 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:11:23.107 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:11:22 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:11:22.722663+0000 mon.smithi084 (mon.0) 132 : audit [DBG] from='client.? 172.21.15.132:0/1630263781' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:11:24.107 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:11:23 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:11:22.848650+0000 mgr.smithi084.bmqrnf (mgr.14162) 44 : cluster [DBG] pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:11:24.353 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:11:24.353 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:11:26.074 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:26.070+0000 7fc7450a8700 1 -- 172.21.15.132:0/2181245967 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc7400f5410 msgr2=0x7fc7400f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:11:26.074 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:26.070+0000 7fc7450a8700 1 --2- 172.21.15.132:0/2181245967 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc7400f5410 0x7fc7400f5830 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7fc730009fb0 tx=0x7fc730009b30).stop 2024-02-20T00:11:26.074 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:26.070+0000 7fc7450a8700 1 -- 172.21.15.132:0/2181245967 shutdown_connections 2024-02-20T00:11:26.074 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:26.070+0000 7fc7450a8700 1 --2- 172.21.15.132:0/2181245967 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc7400f5410 0x7fc7400f5830 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:11:26.074 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:26.070+0000 7fc7450a8700 1 -- 172.21.15.132:0/2181245967 >> 172.21.15.132:0/2181245967 conn(0x7fc7400f0b70 msgr2=0x7fc7400f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:11:26.075 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:26.070+0000 7fc7450a8700 1 -- 172.21.15.132:0/2181245967 shutdown_connections 2024-02-20T00:11:26.075 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:26.071+0000 7fc7450a8700 1 -- 172.21.15.132:0/2181245967 wait complete. 2024-02-20T00:11:26.075 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:26.071+0000 7fc7450a8700 1 Processor -- start 2024-02-20T00:11:26.075 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:26.071+0000 7fc7450a8700 1 -- start start 2024-02-20T00:11:26.075 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:26.071+0000 7fc7450a8700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc7400f5410 0x7fc740105720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:11:26.075 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:26.071+0000 7fc7450a8700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc740105c10 con 0x7fc7400f5410 2024-02-20T00:11:26.075 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:26.071+0000 7fc73ed9d700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc7400f5410 0x7fc740105720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:11:26.075 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:26.071+0000 7fc73ed9d700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc7400f5410 0x7fc740105720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:42634/0 (socket says 172.21.15.132:42634) 2024-02-20T00:11:26.075 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:26.071+0000 7fc73ed9d700 1 -- 172.21.15.132:0/1065762436 learned_addr learned my addr 172.21.15.132:0/1065762436 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:11:26.075 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:26.072+0000 7fc73ed9d700 1 -- 172.21.15.132:0/1065762436 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc730005040 con 0x7fc7400f5410 2024-02-20T00:11:26.075 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:26.072+0000 7fc73ed9d700 1 --2- 172.21.15.132:0/1065762436 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc7400f5410 0x7fc740105720 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7fc730005010 tx=0x7fc73000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:11:26.075 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:26.072+0000 7fc7277fe700 1 -- 172.21.15.132:0/1065762436 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc730009320 con 0x7fc7400f5410 2024-02-20T00:11:26.075 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:26.072+0000 7fc7450a8700 1 -- 172.21.15.132:0/1065762436 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc740105e10 con 0x7fc7400f5410 2024-02-20T00:11:26.075 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:26.072+0000 7fc7277fe700 1 -- 172.21.15.132:0/1065762436 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc730013380 con 0x7fc7400f5410 2024-02-20T00:11:26.076 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:26.072+0000 7fc7277fe700 1 -- 172.21.15.132:0/1065762436 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc730014e50 con 0x7fc7400f5410 2024-02-20T00:11:26.076 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:26.072+0000 7fc7450a8700 1 -- 172.21.15.132:0/1065762436 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7401062a0 con 0x7fc7400f5410 2024-02-20T00:11:26.076 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:26.073+0000 7fc7277fe700 1 -- 172.21.15.132:0/1065762436 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc7300095f0 con 0x7fc7400f5410 2024-02-20T00:11:26.077 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:26.073+0000 7fc7450a8700 1 -- 172.21.15.132:0/1065762436 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc740045ad0 con 0x7fc7400f5410 2024-02-20T00:11:26.077 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:26.073+0000 7fc7277fe700 1 --2- 172.21.15.132:0/1065762436 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fc72802ff10 0x7fc7280323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:11:26.077 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:26.073+0000 7fc7277fe700 1 -- 172.21.15.132:0/1065762436 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc730040e90 con 0x7fc7400f5410 2024-02-20T00:11:26.077 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:26.073+0000 7fc73e59c700 1 --2- 172.21.15.132:0/1065762436 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fc72802ff10 0x7fc7280323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:11:26.077 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:26.074+0000 7fc73e59c700 1 --2- 172.21.15.132:0/1065762436 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fc72802ff10 0x7fc7280323d0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7fc7340096a0 tx=0x7fc734006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:11:26.080 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:26.078+0000 7fc7277fe700 1 -- 172.21.15.132:0/1065762436 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc730014420 con 0x7fc7400f5410 2024-02-20T00:11:26.286 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:26.284+0000 7fc7450a8700 1 -- 172.21.15.132:0/1065762436 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc740107150 con 0x7fc7400f5410 2024-02-20T00:11:26.287 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:26.286+0000 7fc7277fe700 1 -- 172.21.15.132:0/1065762436 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc730010230 con 0x7fc7400f5410 2024-02-20T00:11:26.287 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:11:26.288 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:11:26.289 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:26.286+0000 7fc7450a8700 1 -- 172.21.15.132:0/1065762436 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fc72802ff10 msgr2=0x7fc7280323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:11:26.289 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:26.286+0000 7fc7450a8700 1 --2- 172.21.15.132:0/1065762436 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fc72802ff10 0x7fc7280323d0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7fc7340096a0 tx=0x7fc734006b40).stop 2024-02-20T00:11:26.289 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:26.286+0000 7fc7450a8700 1 -- 172.21.15.132:0/1065762436 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc7400f5410 msgr2=0x7fc740105720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:11:26.289 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:26.286+0000 7fc7450a8700 1 --2- 172.21.15.132:0/1065762436 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc7400f5410 0x7fc740105720 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7fc730005010 tx=0x7fc73000b040).stop 2024-02-20T00:11:26.289 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:26.287+0000 7fc7450a8700 1 -- 172.21.15.132:0/1065762436 shutdown_connections 2024-02-20T00:11:26.289 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:26.287+0000 7fc7450a8700 1 --2- 172.21.15.132:0/1065762436 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fc72802ff10 0x7fc7280323d0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:11:26.289 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:26.287+0000 7fc7450a8700 1 --2- 172.21.15.132:0/1065762436 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc7400f5410 0x7fc740105720 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:11:26.289 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:26.287+0000 7fc7450a8700 1 -- 172.21.15.132:0/1065762436 >> 172.21.15.132:0/1065762436 conn(0x7fc7400f0b70 msgr2=0x7fc7400f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:11:26.289 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:26.287+0000 7fc7450a8700 1 -- 172.21.15.132:0/1065762436 shutdown_connections 2024-02-20T00:11:26.290 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:26.287+0000 7fc7450a8700 1 -- 172.21.15.132:0/1065762436 wait complete. 2024-02-20T00:11:26.290 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:11:26.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:11:25 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:11:24.848960+0000 mgr.smithi084.bmqrnf (mgr.14162) 45 : cluster [DBG] pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:11:27.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:11:26 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:11:26.286427+0000 mon.smithi084 (mon.0) 133 : audit [DBG] from='client.? 172.21.15.132:0/1065762436' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:11:27.973 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:11:27.974 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:11:28.359 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:11:27 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:11:26.849224+0000 mgr.smithi084.bmqrnf (mgr.14162) 46 : cluster [DBG] pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:11:29.862 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:29.860+0000 7f1e561b5700 1 -- 172.21.15.132:0/2342799464 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1e500f5410 msgr2=0x7f1e500f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:11:29.862 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:29.860+0000 7f1e561b5700 1 --2- 172.21.15.132:0/2342799464 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1e500f5410 0x7f1e500f5830 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7f1e40009fb0 tx=0x7f1e40009b30).stop 2024-02-20T00:11:29.862 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:29.860+0000 7f1e561b5700 1 -- 172.21.15.132:0/2342799464 shutdown_connections 2024-02-20T00:11:29.862 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:29.860+0000 7f1e561b5700 1 --2- 172.21.15.132:0/2342799464 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1e500f5410 0x7f1e500f5830 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:11:29.862 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:29.860+0000 7f1e561b5700 1 -- 172.21.15.132:0/2342799464 >> 172.21.15.132:0/2342799464 conn(0x7f1e500f0b70 msgr2=0x7f1e500f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:11:29.862 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:29.860+0000 7f1e561b5700 1 -- 172.21.15.132:0/2342799464 shutdown_connections 2024-02-20T00:11:29.862 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:29.860+0000 7f1e561b5700 1 -- 172.21.15.132:0/2342799464 wait complete. 2024-02-20T00:11:29.864 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:29.861+0000 7f1e561b5700 1 Processor -- start 2024-02-20T00:11:29.864 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:29.861+0000 7f1e561b5700 1 -- start start 2024-02-20T00:11:29.865 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:29.862+0000 7f1e561b5700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1e500f5410 0x7f1e501056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:11:29.865 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:29.862+0000 7f1e561b5700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e50105bd0 con 0x7f1e500f5410 2024-02-20T00:11:29.865 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:29.862+0000 7f1e4f7fe700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1e500f5410 0x7f1e501056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:11:29.865 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:29.862+0000 7f1e4f7fe700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1e500f5410 0x7f1e501056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:46194/0 (socket says 172.21.15.132:46194) 2024-02-20T00:11:29.865 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:29.862+0000 7f1e4f7fe700 1 -- 172.21.15.132:0/645188306 learned_addr learned my addr 172.21.15.132:0/645188306 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:11:29.865 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:29.862+0000 7f1e4f7fe700 1 -- 172.21.15.132:0/645188306 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e40005040 con 0x7f1e500f5410 2024-02-20T00:11:29.865 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:29.862+0000 7f1e4f7fe700 1 --2- 172.21.15.132:0/645188306 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1e500f5410 0x7f1e501056e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f1e40005010 tx=0x7f1e4000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:11:29.865 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:29.862+0000 7f1e37fff700 1 -- 172.21.15.132:0/645188306 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1e40009320 con 0x7f1e500f5410 2024-02-20T00:11:29.865 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:29.862+0000 7f1e561b5700 1 -- 172.21.15.132:0/645188306 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e50105dd0 con 0x7f1e500f5410 2024-02-20T00:11:29.865 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:29.862+0000 7f1e37fff700 1 -- 172.21.15.132:0/645188306 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1e40013380 con 0x7f1e500f5410 2024-02-20T00:11:29.865 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:29.862+0000 7f1e37fff700 1 -- 172.21.15.132:0/645188306 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1e40014e50 con 0x7f1e500f5410 2024-02-20T00:11:29.865 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:29.862+0000 7f1e561b5700 1 -- 172.21.15.132:0/645188306 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e50106260 con 0x7f1e500f5410 2024-02-20T00:11:29.865 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:29.863+0000 7f1e37fff700 1 -- 172.21.15.132:0/645188306 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1e400095f0 con 0x7f1e500f5410 2024-02-20T00:11:29.865 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:29.863+0000 7f1e561b5700 1 -- 172.21.15.132:0/645188306 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e50045ad0 con 0x7f1e500f5410 2024-02-20T00:11:29.866 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:29.863+0000 7f1e37fff700 1 --2- 172.21.15.132:0/645188306 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f1e3802ff10 0x7f1e380323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:11:29.866 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:29.863+0000 7f1e37fff700 1 -- 172.21.15.132:0/645188306 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1e40040ed0 con 0x7f1e500f5410 2024-02-20T00:11:29.866 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:29.863+0000 7f1e4effd700 1 --2- 172.21.15.132:0/645188306 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f1e3802ff10 0x7f1e380323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:11:29.866 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:29.864+0000 7f1e4effd700 1 --2- 172.21.15.132:0/645188306 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f1e3802ff10 0x7f1e380323d0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7f1e440096a0 tx=0x7f1e44006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:11:29.869 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:29.868+0000 7f1e37fff700 1 -- 172.21.15.132:0/645188306 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1e40014420 con 0x7f1e500f5410 2024-02-20T00:11:30.073 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:30.072+0000 7f1e561b5700 1 -- 172.21.15.132:0/645188306 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1e50107110 con 0x7f1e500f5410 2024-02-20T00:11:30.073 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:30.072+0000 7f1e37fff700 1 -- 172.21.15.132:0/645188306 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1e40010230 con 0x7f1e500f5410 2024-02-20T00:11:30.074 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:11:30.074 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:11:30.075 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:30.074+0000 7f1e561b5700 1 -- 172.21.15.132:0/645188306 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f1e3802ff10 msgr2=0x7f1e380323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:11:30.075 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:30.074+0000 7f1e561b5700 1 --2- 172.21.15.132:0/645188306 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f1e3802ff10 0x7f1e380323d0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7f1e440096a0 tx=0x7f1e44006b40).stop 2024-02-20T00:11:30.075 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:30.074+0000 7f1e561b5700 1 -- 172.21.15.132:0/645188306 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1e500f5410 msgr2=0x7f1e501056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:11:30.075 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:30.074+0000 7f1e561b5700 1 --2- 172.21.15.132:0/645188306 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1e500f5410 0x7f1e501056e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f1e40005010 tx=0x7f1e4000b040).stop 2024-02-20T00:11:30.075 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:30.074+0000 7f1e561b5700 1 -- 172.21.15.132:0/645188306 shutdown_connections 2024-02-20T00:11:30.075 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:30.074+0000 7f1e561b5700 1 --2- 172.21.15.132:0/645188306 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f1e3802ff10 0x7f1e380323d0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:11:30.075 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:30.074+0000 7f1e561b5700 1 --2- 172.21.15.132:0/645188306 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1e500f5410 0x7f1e501056e0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:11:30.075 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:30.074+0000 7f1e561b5700 1 -- 172.21.15.132:0/645188306 >> 172.21.15.132:0/645188306 conn(0x7f1e500f0b70 msgr2=0x7f1e500f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:11:30.076 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:30.074+0000 7f1e561b5700 1 -- 172.21.15.132:0/645188306 shutdown_connections 2024-02-20T00:11:30.076 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:30.074+0000 7f1e561b5700 1 -- 172.21.15.132:0/645188306 wait complete. 2024-02-20T00:11:30.076 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:11:30.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:11:29 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:11:28.849469+0000 mgr.smithi084.bmqrnf (mgr.14162) 47 : cluster [DBG] pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:11:31.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:11:30 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:11:30.072827+0000 mon.smithi084 (mon.0) 134 : audit [DBG] from='client.? 172.21.15.132:0/645188306' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:11:31.693 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:11:31.694 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:11:32.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:11:31 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:11:30.849803+0000 mgr.smithi084.bmqrnf (mgr.14162) 48 : cluster [DBG] pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:11:33.594 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:33.590+0000 7fb35326c700 1 -- 172.21.15.132:0/2941816193 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb34c0f6510 msgr2=0x7fb34c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:11:33.595 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:33.590+0000 7fb35326c700 1 --2- 172.21.15.132:0/2941816193 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb34c0f6510 0x7fb34c0f6930 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7fb33c009fb0 tx=0x7fb33c009b30).stop 2024-02-20T00:11:33.595 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:33.590+0000 7fb35326c700 1 -- 172.21.15.132:0/2941816193 shutdown_connections 2024-02-20T00:11:33.595 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:33.590+0000 7fb35326c700 1 --2- 172.21.15.132:0/2941816193 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb34c0f6510 0x7fb34c0f6930 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:11:33.595 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:33.590+0000 7fb35326c700 1 -- 172.21.15.132:0/2941816193 >> 172.21.15.132:0/2941816193 conn(0x7fb34c0f1c70 msgr2=0x7fb34c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:11:33.595 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:33.590+0000 7fb35326c700 1 -- 172.21.15.132:0/2941816193 shutdown_connections 2024-02-20T00:11:33.595 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:33.590+0000 7fb35326c700 1 -- 172.21.15.132:0/2941816193 wait complete. 2024-02-20T00:11:33.595 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:33.591+0000 7fb35326c700 1 Processor -- start 2024-02-20T00:11:33.595 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:33.591+0000 7fb35326c700 1 -- start start 2024-02-20T00:11:33.596 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:33.591+0000 7fb35326c700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb34c0f6510 0x7fb34c06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:11:33.596 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:33.591+0000 7fb35326c700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb34c069e20 con 0x7fb34c0f6510 2024-02-20T00:11:33.596 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:33.591+0000 7fb351008700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb34c0f6510 0x7fb34c06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:11:33.596 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:33.592+0000 7fb351008700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb34c0f6510 0x7fb34c06b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:46218/0 (socket says 172.21.15.132:46218) 2024-02-20T00:11:33.596 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:33.592+0000 7fb351008700 1 -- 172.21.15.132:0/3949481012 learned_addr learned my addr 172.21.15.132:0/3949481012 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:11:33.597 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:33.592+0000 7fb351008700 1 -- 172.21.15.132:0/3949481012 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb33c005040 con 0x7fb34c0f6510 2024-02-20T00:11:33.597 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:33.592+0000 7fb351008700 1 --2- 172.21.15.132:0/3949481012 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb34c0f6510 0x7fb34c06b9a0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7fb33c005010 tx=0x7fb33c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:11:33.597 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:33.592+0000 7fb341ffb700 1 -- 172.21.15.132:0/3949481012 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb33c009320 con 0x7fb34c0f6510 2024-02-20T00:11:33.597 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:33.592+0000 7fb35326c700 1 -- 172.21.15.132:0/3949481012 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb34c069fc0 con 0x7fb34c0f6510 2024-02-20T00:11:33.597 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:33.592+0000 7fb341ffb700 1 -- 172.21.15.132:0/3949481012 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb33c013380 con 0x7fb34c0f6510 2024-02-20T00:11:33.597 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:33.592+0000 7fb341ffb700 1 -- 172.21.15.132:0/3949481012 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb33c014e50 con 0x7fb34c0f6510 2024-02-20T00:11:33.597 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:33.592+0000 7fb35326c700 1 -- 172.21.15.132:0/3949481012 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb34c06a450 con 0x7fb34c0f6510 2024-02-20T00:11:33.597 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:33.593+0000 7fb341ffb700 1 -- 172.21.15.132:0/3949481012 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb33c0095f0 con 0x7fb34c0f6510 2024-02-20T00:11:33.597 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:33.593+0000 7fb35326c700 1 -- 172.21.15.132:0/3949481012 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb34c059100 con 0x7fb34c0f6510 2024-02-20T00:11:33.598 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:33.593+0000 7fb341ffb700 1 --2- 172.21.15.132:0/3949481012 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fb33802ff10 0x7fb3380323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:11:33.598 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:33.593+0000 7fb341ffb700 1 -- 172.21.15.132:0/3949481012 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb33c040f70 con 0x7fb34c0f6510 2024-02-20T00:11:33.598 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:33.593+0000 7fb350807700 1 --2- 172.21.15.132:0/3949481012 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fb33802ff10 0x7fb3380323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:11:33.598 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:33.594+0000 7fb350807700 1 --2- 172.21.15.132:0/3949481012 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fb33802ff10 0x7fb3380323d0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7fb3480096a0 tx=0x7fb348006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:11:33.599 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:33.598+0000 7fb341ffb700 1 -- 172.21.15.132:0/3949481012 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb33c014420 con 0x7fb34c0f6510 2024-02-20T00:11:33.804 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:33.802+0000 7fb35326c700 1 -- 172.21.15.132:0/3949481012 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb34c0fc0a0 con 0x7fb34c0f6510 2024-02-20T00:11:33.804 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:33.803+0000 7fb341ffb700 1 -- 172.21.15.132:0/3949481012 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb33c01d980 con 0x7fb34c0f6510 2024-02-20T00:11:33.805 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:11:33.805 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:11:33.807 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:33.804+0000 7fb35326c700 1 -- 172.21.15.132:0/3949481012 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fb33802ff10 msgr2=0x7fb3380323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:11:33.807 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:33.804+0000 7fb35326c700 1 --2- 172.21.15.132:0/3949481012 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fb33802ff10 0x7fb3380323d0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7fb3480096a0 tx=0x7fb348006b40).stop 2024-02-20T00:11:33.807 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:33.805+0000 7fb35326c700 1 -- 172.21.15.132:0/3949481012 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb34c0f6510 msgr2=0x7fb34c06b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:11:33.807 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:33.805+0000 7fb35326c700 1 --2- 172.21.15.132:0/3949481012 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb34c0f6510 0x7fb34c06b9a0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7fb33c005010 tx=0x7fb33c00b040).stop 2024-02-20T00:11:33.808 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:33.805+0000 7fb35326c700 1 -- 172.21.15.132:0/3949481012 shutdown_connections 2024-02-20T00:11:33.808 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:33.805+0000 7fb35326c700 1 --2- 172.21.15.132:0/3949481012 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fb33802ff10 0x7fb3380323d0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:11:33.808 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:33.805+0000 7fb35326c700 1 --2- 172.21.15.132:0/3949481012 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb34c0f6510 0x7fb34c06b9a0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:11:33.808 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:33.805+0000 7fb35326c700 1 -- 172.21.15.132:0/3949481012 >> 172.21.15.132:0/3949481012 conn(0x7fb34c0f1c70 msgr2=0x7fb34c0f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:11:33.808 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:33.805+0000 7fb35326c700 1 -- 172.21.15.132:0/3949481012 shutdown_connections 2024-02-20T00:11:33.808 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:33.805+0000 7fb35326c700 1 -- 172.21.15.132:0/3949481012 wait complete. 2024-02-20T00:11:33.809 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:11:34.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:11:33 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:11:32.850053+0000 mgr.smithi084.bmqrnf (mgr.14162) 49 : cluster [DBG] pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:11:34.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:11:33 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:11:33.803453+0000 mon.smithi084 (mon.0) 135 : audit [DBG] from='client.? 172.21.15.132:0/3949481012' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:11:35.598 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:11:35.598 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:11:36.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:11:35 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:11:34.850291+0000 mgr.smithi084.bmqrnf (mgr.14162) 50 : cluster [DBG] pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:11:37.385 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:37.384+0000 7fb968a26700 1 -- 172.21.15.132:0/1143502627 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb9640f78e0 msgr2=0x7fb9640f7d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:11:37.385 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:37.384+0000 7fb968a26700 1 --2- 172.21.15.132:0/1143502627 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb9640f78e0 0x7fb9640f7d00 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7fb954009fb0 tx=0x7fb954009b30).stop 2024-02-20T00:11:37.385 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:37.384+0000 7fb968a26700 1 -- 172.21.15.132:0/1143502627 shutdown_connections 2024-02-20T00:11:37.385 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:37.384+0000 7fb968a26700 1 --2- 172.21.15.132:0/1143502627 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb9640f78e0 0x7fb9640f7d00 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:11:37.385 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:37.384+0000 7fb968a26700 1 -- 172.21.15.132:0/1143502627 >> 172.21.15.132:0/1143502627 conn(0x7fb9640f0120 msgr2=0x7fb9640f2540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:11:37.385 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:37.384+0000 7fb968a26700 1 -- 172.21.15.132:0/1143502627 shutdown_connections 2024-02-20T00:11:37.385 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:37.384+0000 7fb968a26700 1 -- 172.21.15.132:0/1143502627 wait complete. 2024-02-20T00:11:37.387 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:37.384+0000 7fb968a26700 1 Processor -- start 2024-02-20T00:11:37.387 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:37.385+0000 7fb968a26700 1 -- start start 2024-02-20T00:11:37.387 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:37.385+0000 7fb968a26700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb9640f78e0 0x7fb9641065e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:11:37.387 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:37.385+0000 7fb968a26700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb964106ad0 con 0x7fb9640f78e0 2024-02-20T00:11:37.387 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:37.385+0000 7fb9637fe700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb9640f78e0 0x7fb9641065e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:11:37.387 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:37.385+0000 7fb9637fe700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb9640f78e0 0x7fb9641065e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:45600/0 (socket says 172.21.15.132:45600) 2024-02-20T00:11:37.388 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:37.385+0000 7fb9637fe700 1 -- 172.21.15.132:0/1226475305 learned_addr learned my addr 172.21.15.132:0/1226475305 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:11:37.388 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:37.385+0000 7fb9637fe700 1 -- 172.21.15.132:0/1226475305 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb954005040 con 0x7fb9640f78e0 2024-02-20T00:11:37.388 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:37.385+0000 7fb9637fe700 1 --2- 172.21.15.132:0/1226475305 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb9640f78e0 0x7fb9641065e0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7fb954009830 tx=0x7fb95400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:11:37.388 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:37.386+0000 7fb94bfff700 1 -- 172.21.15.132:0/1226475305 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb954009320 con 0x7fb9640f78e0 2024-02-20T00:11:37.388 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:37.386+0000 7fb968a26700 1 -- 172.21.15.132:0/1226475305 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb964106cd0 con 0x7fb9640f78e0 2024-02-20T00:11:37.388 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:37.386+0000 7fb94bfff700 1 -- 172.21.15.132:0/1226475305 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb954013380 con 0x7fb9640f78e0 2024-02-20T00:11:37.388 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:37.386+0000 7fb94bfff700 1 -- 172.21.15.132:0/1226475305 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb954014420 con 0x7fb9640f78e0 2024-02-20T00:11:37.388 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:37.386+0000 7fb968a26700 1 -- 172.21.15.132:0/1226475305 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb964107160 con 0x7fb9640f78e0 2024-02-20T00:11:37.389 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:37.386+0000 7fb94bfff700 1 -- 172.21.15.132:0/1226475305 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb954014580 con 0x7fb9640f78e0 2024-02-20T00:11:37.389 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:37.387+0000 7fb94bfff700 1 --2- 172.21.15.132:0/1226475305 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fb94c02ff60 0x7fb94c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:11:37.389 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:37.387+0000 7fb94bfff700 1 -- 172.21.15.132:0/1226475305 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb954040ed0 con 0x7fb9640f78e0 2024-02-20T00:11:37.389 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:37.387+0000 7fb968a26700 1 -- 172.21.15.132:0/1226475305 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb964045ad0 con 0x7fb9640f78e0 2024-02-20T00:11:37.389 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:37.387+0000 7fb962ffd700 1 --2- 172.21.15.132:0/1226475305 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fb94c02ff60 0x7fb94c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:11:37.389 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:37.387+0000 7fb962ffd700 1 --2- 172.21.15.132:0/1226475305 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fb94c02ff60 0x7fb94c032420 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7fb9580096a0 tx=0x7fb958006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:11:37.392 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:37.391+0000 7fb94bfff700 1 -- 172.21.15.132:0/1226475305 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb9540273f0 con 0x7fb9640f78e0 2024-02-20T00:11:37.599 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:37.597+0000 7fb968a26700 1 -- 172.21.15.132:0/1226475305 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb9640fbe20 con 0x7fb9640f78e0 2024-02-20T00:11:37.599 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:37.598+0000 7fb94bfff700 1 -- 172.21.15.132:0/1226475305 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb95401c890 con 0x7fb9640f78e0 2024-02-20T00:11:37.599 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:11:37.599 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:11:37.601 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:37.599+0000 7fb968a26700 1 -- 172.21.15.132:0/1226475305 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fb94c02ff60 msgr2=0x7fb94c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:11:37.601 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:37.599+0000 7fb968a26700 1 --2- 172.21.15.132:0/1226475305 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fb94c02ff60 0x7fb94c032420 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7fb9580096a0 tx=0x7fb958006b40).stop 2024-02-20T00:11:37.601 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:37.599+0000 7fb968a26700 1 -- 172.21.15.132:0/1226475305 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb9640f78e0 msgr2=0x7fb9641065e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:11:37.601 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:37.599+0000 7fb968a26700 1 --2- 172.21.15.132:0/1226475305 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb9640f78e0 0x7fb9641065e0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7fb954009830 tx=0x7fb95400b040).stop 2024-02-20T00:11:37.601 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:37.600+0000 7fb968a26700 1 -- 172.21.15.132:0/1226475305 shutdown_connections 2024-02-20T00:11:37.601 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:37.600+0000 7fb968a26700 1 --2- 172.21.15.132:0/1226475305 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fb94c02ff60 0x7fb94c032420 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:11:37.601 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:37.600+0000 7fb968a26700 1 --2- 172.21.15.132:0/1226475305 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb9640f78e0 0x7fb9641065e0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:11:37.601 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:37.600+0000 7fb968a26700 1 -- 172.21.15.132:0/1226475305 >> 172.21.15.132:0/1226475305 conn(0x7fb9640f0120 msgr2=0x7fb9640f0b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:11:37.601 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:37.600+0000 7fb968a26700 1 -- 172.21.15.132:0/1226475305 shutdown_connections 2024-02-20T00:11:37.602 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:37.600+0000 7fb968a26700 1 -- 172.21.15.132:0/1226475305 wait complete. 2024-02-20T00:11:37.602 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:11:38.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:11:37 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:11:36.850535+0000 mgr.smithi084.bmqrnf (mgr.14162) 51 : cluster [DBG] pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:11:38.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:11:37 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:11:37.598639+0000 mon.smithi084 (mon.0) 136 : audit [DBG] from='client.? 172.21.15.132:0/1226475305' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:11:39.142 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:11:39.143 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:11:40.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:11:39 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:11:38.850770+0000 mgr.smithi084.bmqrnf (mgr.14162) 52 : cluster [DBG] pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:11:40.873 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:40.870+0000 7f7c390a8700 1 -- 172.21.15.132:0/3950001088 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7c340f6510 msgr2=0x7f7c340f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:11:40.873 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:40.870+0000 7f7c390a8700 1 --2- 172.21.15.132:0/3950001088 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7c340f6510 0x7f7c340f6930 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7f7c24009fb0 tx=0x7f7c24009b30).stop 2024-02-20T00:11:40.873 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:40.870+0000 7f7c390a8700 1 -- 172.21.15.132:0/3950001088 shutdown_connections 2024-02-20T00:11:40.873 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:40.870+0000 7f7c390a8700 1 --2- 172.21.15.132:0/3950001088 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7c340f6510 0x7f7c340f6930 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:11:40.873 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:40.870+0000 7f7c390a8700 1 -- 172.21.15.132:0/3950001088 >> 172.21.15.132:0/3950001088 conn(0x7f7c340f1c70 msgr2=0x7f7c340f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:11:40.874 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:40.870+0000 7f7c390a8700 1 -- 172.21.15.132:0/3950001088 shutdown_connections 2024-02-20T00:11:40.874 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:40.870+0000 7f7c390a8700 1 -- 172.21.15.132:0/3950001088 wait complete. 2024-02-20T00:11:40.874 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:40.871+0000 7f7c390a8700 1 Processor -- start 2024-02-20T00:11:40.874 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:40.871+0000 7f7c390a8700 1 -- start start 2024-02-20T00:11:40.874 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:40.871+0000 7f7c390a8700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7c340f6510 0x7f7c341067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:11:40.874 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:40.871+0000 7f7c390a8700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c34106cd0 con 0x7f7c340f6510 2024-02-20T00:11:40.874 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:40.871+0000 7f7c32d9d700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7c340f6510 0x7f7c341067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:11:40.874 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:40.871+0000 7f7c32d9d700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7c340f6510 0x7f7c341067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:45622/0 (socket says 172.21.15.132:45622) 2024-02-20T00:11:40.874 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:40.871+0000 7f7c32d9d700 1 -- 172.21.15.132:0/1458756961 learned_addr learned my addr 172.21.15.132:0/1458756961 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:11:40.874 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:40.871+0000 7f7c32d9d700 1 -- 172.21.15.132:0/1458756961 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c24005040 con 0x7f7c340f6510 2024-02-20T00:11:40.874 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:40.872+0000 7f7c32d9d700 1 --2- 172.21.15.132:0/1458756961 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7c340f6510 0x7f7c341067e0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f7c24005010 tx=0x7f7c2400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:11:40.874 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:40.872+0000 7f7c2b7fe700 1 -- 172.21.15.132:0/1458756961 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7c24009320 con 0x7f7c340f6510 2024-02-20T00:11:40.875 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:40.872+0000 7f7c390a8700 1 -- 172.21.15.132:0/1458756961 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c34106ed0 con 0x7f7c340f6510 2024-02-20T00:11:40.875 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:40.872+0000 7f7c2b7fe700 1 -- 172.21.15.132:0/1458756961 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7c24013380 con 0x7f7c340f6510 2024-02-20T00:11:40.875 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:40.872+0000 7f7c2b7fe700 1 -- 172.21.15.132:0/1458756961 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7c24014e50 con 0x7f7c340f6510 2024-02-20T00:11:40.875 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:40.872+0000 7f7c390a8700 1 -- 172.21.15.132:0/1458756961 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c34107360 con 0x7f7c340f6510 2024-02-20T00:11:40.876 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:40.872+0000 7f7c2b7fe700 1 -- 172.21.15.132:0/1458756961 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7c240095f0 con 0x7f7c340f6510 2024-02-20T00:11:40.876 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:40.873+0000 7f7c390a8700 1 -- 172.21.15.132:0/1458756961 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c34045ad0 con 0x7f7c340f6510 2024-02-20T00:11:40.876 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:40.873+0000 7f7c2b7fe700 1 --2- 172.21.15.132:0/1458756961 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f7c2002ff10 0x7f7c200323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:11:40.876 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:40.873+0000 7f7c2b7fe700 1 -- 172.21.15.132:0/1458756961 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7c24040e90 con 0x7f7c340f6510 2024-02-20T00:11:40.876 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:40.873+0000 7f7c3259c700 1 --2- 172.21.15.132:0/1458756961 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f7c2002ff10 0x7f7c200323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:11:40.876 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:40.873+0000 7f7c3259c700 1 --2- 172.21.15.132:0/1458756961 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f7c2002ff10 0x7f7c200323d0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7f7c1c0096a0 tx=0x7f7c1c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:11:40.878 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:40.877+0000 7f7c2b7fe700 1 -- 172.21.15.132:0/1458756961 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7c24014420 con 0x7f7c340f6510 2024-02-20T00:11:41.084 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:41.083+0000 7f7c390a8700 1 -- 172.21.15.132:0/1458756961 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7c34108210 con 0x7f7c340f6510 2024-02-20T00:11:41.085 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:41.084+0000 7f7c2b7fe700 1 -- 172.21.15.132:0/1458756961 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7c24010230 con 0x7f7c340f6510 2024-02-20T00:11:41.086 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:11:41.086 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:11:41.087 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:41.085+0000 7f7c390a8700 1 -- 172.21.15.132:0/1458756961 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f7c2002ff10 msgr2=0x7f7c200323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:11:41.087 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:41.085+0000 7f7c390a8700 1 --2- 172.21.15.132:0/1458756961 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f7c2002ff10 0x7f7c200323d0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7f7c1c0096a0 tx=0x7f7c1c006b40).stop 2024-02-20T00:11:41.087 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:41.086+0000 7f7c390a8700 1 -- 172.21.15.132:0/1458756961 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7c340f6510 msgr2=0x7f7c341067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:11:41.087 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:41.086+0000 7f7c390a8700 1 --2- 172.21.15.132:0/1458756961 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7c340f6510 0x7f7c341067e0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f7c24005010 tx=0x7f7c2400b040).stop 2024-02-20T00:11:41.087 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:41.086+0000 7f7c390a8700 1 -- 172.21.15.132:0/1458756961 shutdown_connections 2024-02-20T00:11:41.087 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:41.086+0000 7f7c390a8700 1 --2- 172.21.15.132:0/1458756961 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f7c2002ff10 0x7f7c200323d0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:11:41.087 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:41.086+0000 7f7c390a8700 1 --2- 172.21.15.132:0/1458756961 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7c340f6510 0x7f7c341067e0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:11:41.087 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:41.086+0000 7f7c390a8700 1 -- 172.21.15.132:0/1458756961 >> 172.21.15.132:0/1458756961 conn(0x7f7c340f1c70 msgr2=0x7f7c340f26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:11:41.087 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:41.086+0000 7f7c390a8700 1 -- 172.21.15.132:0/1458756961 shutdown_connections 2024-02-20T00:11:41.088 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:41.086+0000 7f7c390a8700 1 -- 172.21.15.132:0/1458756961 wait complete. 2024-02-20T00:11:41.088 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:11:42.107 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:11:41 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:11:40.850976+0000 mgr.smithi084.bmqrnf (mgr.14162) 53 : cluster [DBG] pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:11:42.108 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:11:41 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:11:41.084473+0000 mon.smithi084 (mon.0) 137 : audit [DBG] from='client.? 172.21.15.132:0/1458756961' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:11:42.662 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:11:42.662 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:11:44.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:11:43 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:11:42.851309+0000 mgr.smithi084.bmqrnf (mgr.14162) 54 : cluster [DBG] pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:11:44.376 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:44.370+0000 7fd9d50d7700 1 -- 172.21.15.132:0/2680168115 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd9d00f6510 msgr2=0x7fd9d00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:11:44.377 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:44.370+0000 7fd9d50d7700 1 --2- 172.21.15.132:0/2680168115 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd9d00f6510 0x7fd9d00f6930 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7fd9c4009fb0 tx=0x7fd9c4009b30).stop 2024-02-20T00:11:44.377 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:44.370+0000 7fd9d50d7700 1 -- 172.21.15.132:0/2680168115 shutdown_connections 2024-02-20T00:11:44.377 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:44.370+0000 7fd9d50d7700 1 --2- 172.21.15.132:0/2680168115 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd9d00f6510 0x7fd9d00f6930 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:11:44.377 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:44.370+0000 7fd9d50d7700 1 -- 172.21.15.132:0/2680168115 >> 172.21.15.132:0/2680168115 conn(0x7fd9d00f1c70 msgr2=0x7fd9d00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:11:44.377 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:44.370+0000 7fd9d50d7700 1 -- 172.21.15.132:0/2680168115 shutdown_connections 2024-02-20T00:11:44.377 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:44.370+0000 7fd9d50d7700 1 -- 172.21.15.132:0/2680168115 wait complete. 2024-02-20T00:11:44.377 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:44.370+0000 7fd9d50d7700 1 Processor -- start 2024-02-20T00:11:44.377 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:44.371+0000 7fd9d50d7700 1 -- start start 2024-02-20T00:11:44.377 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:44.371+0000 7fd9d50d7700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd9d00f6510 0x7fd9d0106810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:11:44.377 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:44.371+0000 7fd9d50d7700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9d0106d00 con 0x7fd9d00f6510 2024-02-20T00:11:44.377 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:44.371+0000 7fd9ced9d700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd9d00f6510 0x7fd9d0106810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:11:44.377 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:44.371+0000 7fd9ced9d700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd9d00f6510 0x7fd9d0106810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:45646/0 (socket says 172.21.15.132:45646) 2024-02-20T00:11:44.377 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:44.371+0000 7fd9ced9d700 1 -- 172.21.15.132:0/1783333455 learned_addr learned my addr 172.21.15.132:0/1783333455 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:11:44.378 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:44.371+0000 7fd9ced9d700 1 -- 172.21.15.132:0/1783333455 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9c4005040 con 0x7fd9d00f6510 2024-02-20T00:11:44.378 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:44.371+0000 7fd9ced9d700 1 --2- 172.21.15.132:0/1783333455 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd9d00f6510 0x7fd9d0106810 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7fd9c4005010 tx=0x7fd9c400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:11:44.378 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:44.372+0000 7fd9b77fe700 1 -- 172.21.15.132:0/1783333455 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd9c4009320 con 0x7fd9d00f6510 2024-02-20T00:11:44.378 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:44.372+0000 7fd9d50d7700 1 -- 172.21.15.132:0/1783333455 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9d0106f00 con 0x7fd9d00f6510 2024-02-20T00:11:44.378 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:44.372+0000 7fd9b77fe700 1 -- 172.21.15.132:0/1783333455 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd9c4013380 con 0x7fd9d00f6510 2024-02-20T00:11:44.378 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:44.372+0000 7fd9b77fe700 1 -- 172.21.15.132:0/1783333455 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd9c4014e50 con 0x7fd9d00f6510 2024-02-20T00:11:44.378 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:44.372+0000 7fd9d50d7700 1 -- 172.21.15.132:0/1783333455 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9d0107390 con 0x7fd9d00f6510 2024-02-20T00:11:44.378 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:44.372+0000 7fd9d50d7700 1 -- 172.21.15.132:0/1783333455 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9d0045ad0 con 0x7fd9d00f6510 2024-02-20T00:11:44.378 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:44.372+0000 7fd9b77fe700 1 -- 172.21.15.132:0/1783333455 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd9c40095f0 con 0x7fd9d00f6510 2024-02-20T00:11:44.378 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:44.373+0000 7fd9b77fe700 1 --2- 172.21.15.132:0/1783333455 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fd9b802ff10 0x7fd9b80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:11:44.378 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:44.373+0000 7fd9b77fe700 1 -- 172.21.15.132:0/1783333455 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd9c4040f80 con 0x7fd9d00f6510 2024-02-20T00:11:44.378 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:44.373+0000 7fd9ce59c700 1 --2- 172.21.15.132:0/1783333455 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fd9b802ff10 0x7fd9b80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:11:44.378 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:44.373+0000 7fd9ce59c700 1 --2- 172.21.15.132:0/1783333455 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fd9b802ff10 0x7fd9b80323d0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7fd9c00096a0 tx=0x7fd9c0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:11:44.379 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:44.377+0000 7fd9b77fe700 1 -- 172.21.15.132:0/1783333455 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd9c4014420 con 0x7fd9d00f6510 2024-02-20T00:11:44.582 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:44.580+0000 7fd9d50d7700 1 -- 172.21.15.132:0/1783333455 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd9d01081e0 con 0x7fd9d00f6510 2024-02-20T00:11:44.582 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:44.581+0000 7fd9b77fe700 1 -- 172.21.15.132:0/1783333455 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd9c4010230 con 0x7fd9d00f6510 2024-02-20T00:11:44.582 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:11:44.582 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:11:44.584 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:44.583+0000 7fd9d50d7700 1 -- 172.21.15.132:0/1783333455 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fd9b802ff10 msgr2=0x7fd9b80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:11:44.585 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:44.583+0000 7fd9d50d7700 1 --2- 172.21.15.132:0/1783333455 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fd9b802ff10 0x7fd9b80323d0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7fd9c00096a0 tx=0x7fd9c0006b40).stop 2024-02-20T00:11:44.585 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:44.583+0000 7fd9d50d7700 1 -- 172.21.15.132:0/1783333455 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd9d00f6510 msgr2=0x7fd9d0106810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:11:44.585 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:44.583+0000 7fd9d50d7700 1 --2- 172.21.15.132:0/1783333455 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd9d00f6510 0x7fd9d0106810 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7fd9c4005010 tx=0x7fd9c400b040).stop 2024-02-20T00:11:44.585 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:44.583+0000 7fd9d50d7700 1 -- 172.21.15.132:0/1783333455 shutdown_connections 2024-02-20T00:11:44.585 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:44.583+0000 7fd9d50d7700 1 --2- 172.21.15.132:0/1783333455 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fd9b802ff10 0x7fd9b80323d0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:11:44.585 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:44.583+0000 7fd9d50d7700 1 --2- 172.21.15.132:0/1783333455 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd9d00f6510 0x7fd9d0106810 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:11:44.585 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:44.583+0000 7fd9d50d7700 1 -- 172.21.15.132:0/1783333455 >> 172.21.15.132:0/1783333455 conn(0x7fd9d00f1c70 msgr2=0x7fd9d00f26f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:11:44.585 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:44.583+0000 7fd9d50d7700 1 -- 172.21.15.132:0/1783333455 shutdown_connections 2024-02-20T00:11:44.585 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:44.583+0000 7fd9d50d7700 1 -- 172.21.15.132:0/1783333455 wait complete. 2024-02-20T00:11:44.585 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:11:45.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:11:44 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:11:44.581667+0000 mon.smithi084 (mon.0) 138 : audit [DBG] from='client.? 172.21.15.132:0/1783333455' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:11:46.230 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:11:46.230 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:11:46.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:11:45 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:11:44.851585+0000 mgr.smithi084.bmqrnf (mgr.14162) 55 : cluster [DBG] pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:11:47.989 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:47.986+0000 7f97f45aa700 1 -- 172.21.15.132:0/1995336382 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f97ec060220 msgr2=0x7f97ec0f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:11:47.990 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:47.986+0000 7f97f45aa700 1 --2- 172.21.15.132:0/1995336382 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f97ec060220 0x7f97ec0f9150 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7f97dc009fb0 tx=0x7f97dc009b30).stop 2024-02-20T00:11:47.990 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:47.986+0000 7f97f45aa700 1 -- 172.21.15.132:0/1995336382 shutdown_connections 2024-02-20T00:11:47.990 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:47.986+0000 7f97f45aa700 1 --2- 172.21.15.132:0/1995336382 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f97ec060220 0x7f97ec0f9150 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:11:47.990 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:47.986+0000 7f97f45aa700 1 -- 172.21.15.132:0/1995336382 >> 172.21.15.132:0/1995336382 conn(0x7f97ec0f0b70 msgr2=0x7f97ec0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:11:47.990 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:47.986+0000 7f97f45aa700 1 -- 172.21.15.132:0/1995336382 shutdown_connections 2024-02-20T00:11:47.990 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:47.986+0000 7f97f45aa700 1 -- 172.21.15.132:0/1995336382 wait complete. 2024-02-20T00:11:47.990 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:47.987+0000 7f97f45aa700 1 Processor -- start 2024-02-20T00:11:47.990 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:47.987+0000 7f97f45aa700 1 -- start start 2024-02-20T00:11:47.990 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:47.987+0000 7f97f45aa700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f97ec060220 0x7f97ec0f6bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:11:47.990 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:47.987+0000 7f97f45aa700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97ec0f5050 con 0x7f97ec060220 2024-02-20T00:11:47.990 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:47.987+0000 7f97f2346700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f97ec060220 0x7f97ec0f6bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:11:47.990 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:47.987+0000 7f97f2346700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f97ec060220 0x7f97ec0f6bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:36336/0 (socket says 172.21.15.132:36336) 2024-02-20T00:11:47.990 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:47.987+0000 7f97f2346700 1 -- 172.21.15.132:0/2207019603 learned_addr learned my addr 172.21.15.132:0/2207019603 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:11:47.992 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:47.988+0000 7f97f2346700 1 -- 172.21.15.132:0/2207019603 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97dc005040 con 0x7f97ec060220 2024-02-20T00:11:47.992 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:47.988+0000 7f97f2346700 1 --2- 172.21.15.132:0/2207019603 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f97ec060220 0x7f97ec0f6bd0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f97dc0089f0 tx=0x7f97dc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:11:47.992 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:47.988+0000 7f97e2ffd700 1 -- 172.21.15.132:0/2207019603 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f97dc009320 con 0x7f97ec060220 2024-02-20T00:11:47.992 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:47.988+0000 7f97f45aa700 1 -- 172.21.15.132:0/2207019603 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97ec0f51f0 con 0x7f97ec060220 2024-02-20T00:11:47.992 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:47.988+0000 7f97f45aa700 1 -- 172.21.15.132:0/2207019603 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97ec0f5680 con 0x7f97ec060220 2024-02-20T00:11:47.992 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:47.988+0000 7f97e2ffd700 1 -- 172.21.15.132:0/2207019603 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f97dc013380 con 0x7f97ec060220 2024-02-20T00:11:47.992 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:47.988+0000 7f97e2ffd700 1 -- 172.21.15.132:0/2207019603 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f97dc014e20 con 0x7f97ec060220 2024-02-20T00:11:47.993 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:47.989+0000 7f97e2ffd700 1 -- 172.21.15.132:0/2207019603 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f97dc0263a0 con 0x7f97ec060220 2024-02-20T00:11:47.993 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:47.989+0000 7f97e2ffd700 1 --2- 172.21.15.132:0/2207019603 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f97d802ff10 0x7f97d80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:11:47.993 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:47.989+0000 7f97f45aa700 1 -- 172.21.15.132:0/2207019603 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97d4004fa0 con 0x7f97ec060220 2024-02-20T00:11:47.993 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:47.989+0000 7f97e2ffd700 1 -- 172.21.15.132:0/2207019603 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f97dc042060 con 0x7f97ec060220 2024-02-20T00:11:47.993 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:47.989+0000 7f97f1b45700 1 --2- 172.21.15.132:0/2207019603 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f97d802ff10 0x7f97d80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:11:47.993 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:47.990+0000 7f97f1b45700 1 --2- 172.21.15.132:0/2207019603 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f97d802ff10 0x7f97d80323d0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f97e80096a0 tx=0x7f97e8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:11:47.995 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:47.993+0000 7f97e2ffd700 1 -- 172.21.15.132:0/2207019603 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f97dc012920 con 0x7f97ec060220 2024-02-20T00:11:48.200 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:48.198+0000 7f97f45aa700 1 -- 172.21.15.132:0/2207019603 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f97d4005670 con 0x7f97ec060220 2024-02-20T00:11:48.201 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:48.199+0000 7f97e2ffd700 1 -- 172.21.15.132:0/2207019603 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f97dc02b020 con 0x7f97ec060220 2024-02-20T00:11:48.201 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:11:48.201 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:11:48.203 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:48.201+0000 7f97f45aa700 1 -- 172.21.15.132:0/2207019603 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f97d802ff10 msgr2=0x7f97d80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:11:48.203 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:48.201+0000 7f97f45aa700 1 --2- 172.21.15.132:0/2207019603 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f97d802ff10 0x7f97d80323d0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f97e80096a0 tx=0x7f97e8006b40).stop 2024-02-20T00:11:48.203 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:48.201+0000 7f97f45aa700 1 -- 172.21.15.132:0/2207019603 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f97ec060220 msgr2=0x7f97ec0f6bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:11:48.203 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:48.201+0000 7f97f45aa700 1 --2- 172.21.15.132:0/2207019603 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f97ec060220 0x7f97ec0f6bd0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f97dc0089f0 tx=0x7f97dc00b040).stop 2024-02-20T00:11:48.203 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:48.201+0000 7f97f45aa700 1 -- 172.21.15.132:0/2207019603 shutdown_connections 2024-02-20T00:11:48.204 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:48.201+0000 7f97f45aa700 1 --2- 172.21.15.132:0/2207019603 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f97d802ff10 0x7f97d80323d0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:11:48.204 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:48.201+0000 7f97f45aa700 1 --2- 172.21.15.132:0/2207019603 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f97ec060220 0x7f97ec0f6bd0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:11:48.204 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:48.201+0000 7f97f45aa700 1 -- 172.21.15.132:0/2207019603 >> 172.21.15.132:0/2207019603 conn(0x7f97ec0f0b70 msgr2=0x7f97ec0f15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:11:48.204 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:48.201+0000 7f97f45aa700 1 -- 172.21.15.132:0/2207019603 shutdown_connections 2024-02-20T00:11:48.204 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:48.201+0000 7f97f45aa700 1 -- 172.21.15.132:0/2207019603 wait complete. 2024-02-20T00:11:48.204 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:11:48.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:11:47 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:11:46.851824+0000 mgr.smithi084.bmqrnf (mgr.14162) 56 : cluster [DBG] pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:11:49.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:11:48 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:11:48.200378+0000 mon.smithi084 (mon.0) 139 : audit [DBG] from='client.? 172.21.15.132:0/2207019603' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:11:49.772 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:11:49.772 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:11:50.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:11:49 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:11:48.852046+0000 mgr.smithi084.bmqrnf (mgr.14162) 57 : cluster [DBG] pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:11:51.584 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:51.580+0000 7f35d40da700 1 -- 172.21.15.132:0/1130430176 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f35cc0f5410 msgr2=0x7f35cc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:11:51.584 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:51.580+0000 7f35d40da700 1 --2- 172.21.15.132:0/1130430176 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f35cc0f5410 0x7f35cc0f5830 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7f35bc009fb0 tx=0x7f35bc009b30).stop 2024-02-20T00:11:51.585 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:51.580+0000 7f35d40da700 1 -- 172.21.15.132:0/1130430176 shutdown_connections 2024-02-20T00:11:51.585 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:51.580+0000 7f35d40da700 1 --2- 172.21.15.132:0/1130430176 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f35cc0f5410 0x7f35cc0f5830 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:11:51.585 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:51.580+0000 7f35d40da700 1 -- 172.21.15.132:0/1130430176 >> 172.21.15.132:0/1130430176 conn(0x7f35cc0f0b70 msgr2=0x7f35cc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:11:51.585 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:51.581+0000 7f35d40da700 1 -- 172.21.15.132:0/1130430176 shutdown_connections 2024-02-20T00:11:51.585 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:51.581+0000 7f35d40da700 1 -- 172.21.15.132:0/1130430176 wait complete. 2024-02-20T00:11:51.585 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:51.581+0000 7f35d40da700 1 Processor -- start 2024-02-20T00:11:51.585 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:51.581+0000 7f35d40da700 1 -- start start 2024-02-20T00:11:51.585 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:51.581+0000 7f35d40da700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f35cc0f5410 0x7f35cc1094e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:11:51.585 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:51.581+0000 7f35d40da700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35cc060240 con 0x7f35cc0f5410 2024-02-20T00:11:51.585 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:51.581+0000 7f35d1e76700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f35cc0f5410 0x7f35cc1094e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:11:51.585 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:51.581+0000 7f35d1e76700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f35cc0f5410 0x7f35cc1094e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:36350/0 (socket says 172.21.15.132:36350) 2024-02-20T00:11:51.585 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:51.581+0000 7f35d1e76700 1 -- 172.21.15.132:0/1716730421 learned_addr learned my addr 172.21.15.132:0/1716730421 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:11:51.585 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:51.582+0000 7f35d1e76700 1 -- 172.21.15.132:0/1716730421 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35bc005040 con 0x7f35cc0f5410 2024-02-20T00:11:51.586 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:51.582+0000 7f35d1e76700 1 --2- 172.21.15.132:0/1716730421 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f35cc0f5410 0x7f35cc1094e0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f35bc005010 tx=0x7f35bc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:11:51.586 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:51.582+0000 7f35c27fc700 1 -- 172.21.15.132:0/1716730421 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f35bc009320 con 0x7f35cc0f5410 2024-02-20T00:11:51.586 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:51.582+0000 7f35d40da700 1 -- 172.21.15.132:0/1716730421 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35cc060440 con 0x7f35cc0f5410 2024-02-20T00:11:51.586 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:51.582+0000 7f35c27fc700 1 -- 172.21.15.132:0/1716730421 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f35bc013380 con 0x7f35cc0f5410 2024-02-20T00:11:51.586 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:51.582+0000 7f35c27fc700 1 -- 172.21.15.132:0/1716730421 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f35bc014e50 con 0x7f35cc0f5410 2024-02-20T00:11:51.586 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:51.582+0000 7f35d40da700 1 -- 172.21.15.132:0/1716730421 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35cc109b90 con 0x7f35cc0f5410 2024-02-20T00:11:51.587 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:51.583+0000 7f35d40da700 1 -- 172.21.15.132:0/1716730421 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35cc045ad0 con 0x7f35cc0f5410 2024-02-20T00:11:51.587 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:51.583+0000 7f35c27fc700 1 -- 172.21.15.132:0/1716730421 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f35bc0095f0 con 0x7f35cc0f5410 2024-02-20T00:11:51.587 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:51.583+0000 7f35c27fc700 1 --2- 172.21.15.132:0/1716730421 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f35b802ff10 0x7f35b80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:11:51.587 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:51.583+0000 7f35c27fc700 1 -- 172.21.15.132:0/1716730421 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f35bc040fa0 con 0x7f35cc0f5410 2024-02-20T00:11:51.587 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:51.583+0000 7f35d1675700 1 --2- 172.21.15.132:0/1716730421 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f35b802ff10 0x7f35b80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:11:51.587 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:51.584+0000 7f35d1675700 1 --2- 172.21.15.132:0/1716730421 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f35b802ff10 0x7f35b80323d0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f35c80096a0 tx=0x7f35c8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:11:51.590 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:51.588+0000 7f35c27fc700 1 -- 172.21.15.132:0/1716730421 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f35bc014420 con 0x7f35cc0f5410 2024-02-20T00:11:51.793 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:51.791+0000 7f35d40da700 1 -- 172.21.15.132:0/1716730421 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f35cc10a3b0 con 0x7f35cc0f5410 2024-02-20T00:11:51.794 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:51.792+0000 7f35c27fc700 1 -- 172.21.15.132:0/1716730421 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f35bc010230 con 0x7f35cc0f5410 2024-02-20T00:11:51.794 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:11:51.794 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:11:51.796 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:51.793+0000 7f35d40da700 1 -- 172.21.15.132:0/1716730421 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f35b802ff10 msgr2=0x7f35b80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:11:51.796 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:51.793+0000 7f35d40da700 1 --2- 172.21.15.132:0/1716730421 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f35b802ff10 0x7f35b80323d0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f35c80096a0 tx=0x7f35c8006b40).stop 2024-02-20T00:11:51.796 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:51.793+0000 7f35d40da700 1 -- 172.21.15.132:0/1716730421 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f35cc0f5410 msgr2=0x7f35cc1094e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:11:51.796 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:51.793+0000 7f35d40da700 1 --2- 172.21.15.132:0/1716730421 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f35cc0f5410 0x7f35cc1094e0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f35bc005010 tx=0x7f35bc00b040).stop 2024-02-20T00:11:51.796 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:51.793+0000 7f35d40da700 1 -- 172.21.15.132:0/1716730421 shutdown_connections 2024-02-20T00:11:51.796 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:51.793+0000 7f35d40da700 1 --2- 172.21.15.132:0/1716730421 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f35b802ff10 0x7f35b80323d0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:11:51.796 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:51.793+0000 7f35d40da700 1 --2- 172.21.15.132:0/1716730421 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f35cc0f5410 0x7f35cc1094e0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:11:51.797 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:51.793+0000 7f35d40da700 1 -- 172.21.15.132:0/1716730421 >> 172.21.15.132:0/1716730421 conn(0x7f35cc0f0b70 msgr2=0x7f35cc0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:11:51.797 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:51.793+0000 7f35d40da700 1 -- 172.21.15.132:0/1716730421 shutdown_connections 2024-02-20T00:11:51.797 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:51.793+0000 7f35d40da700 1 -- 172.21.15.132:0/1716730421 wait complete. 2024-02-20T00:11:51.797 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:11:52.107 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:11:51 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:11:50.852433+0000 mgr.smithi084.bmqrnf (mgr.14162) 58 : cluster [DBG] pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:11:52.107 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:11:51 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:11:51.793317+0000 mon.smithi084 (mon.0) 140 : audit [DBG] from='client.? 172.21.15.132:0/1716730421' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:11:53.457 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:11:53.458 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:11:54.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:11:53 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:11:52.852874+0000 mgr.smithi084.bmqrnf (mgr.14162) 59 : cluster [DBG] pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:11:55.307 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:55.301+0000 7f409b59e700 1 -- 172.21.15.132:0/620202492 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f409c0601e0 msgr2=0x7f409c0f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:11:55.307 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:55.301+0000 7f409b59e700 1 --2- 172.21.15.132:0/620202492 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f409c0601e0 0x7f409c0f8f20 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f4084009fb0 tx=0x7f4084009b30).stop 2024-02-20T00:11:55.307 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:55.302+0000 7f409b59e700 1 -- 172.21.15.132:0/620202492 shutdown_connections 2024-02-20T00:11:55.307 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:55.302+0000 7f409b59e700 1 --2- 172.21.15.132:0/620202492 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f409c0601e0 0x7f409c0f8f20 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:11:55.307 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:55.302+0000 7f409b59e700 1 -- 172.21.15.132:0/620202492 >> 172.21.15.132:0/620202492 conn(0x7f409c0f09a0 msgr2=0x7f409c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:11:55.307 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:55.302+0000 7f409b59e700 1 -- 172.21.15.132:0/620202492 shutdown_connections 2024-02-20T00:11:55.308 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:55.302+0000 7f409b59e700 1 -- 172.21.15.132:0/620202492 wait complete. 2024-02-20T00:11:55.308 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:55.302+0000 7f409b59e700 1 Processor -- start 2024-02-20T00:11:55.308 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:55.302+0000 7f409b59e700 1 -- start start 2024-02-20T00:11:55.308 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:55.303+0000 7f409b59e700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f409c0601e0 0x7f409c106600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:11:55.308 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:55.303+0000 7f409b59e700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f409c106af0 con 0x7f409c0601e0 2024-02-20T00:11:55.308 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:55.303+0000 7f409a59c700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f409c0601e0 0x7f409c106600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:11:55.308 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:55.303+0000 7f409a59c700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f409c0601e0 0x7f409c106600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:36362/0 (socket says 172.21.15.132:36362) 2024-02-20T00:11:55.308 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:55.303+0000 7f409a59c700 1 -- 172.21.15.132:0/219858438 learned_addr learned my addr 172.21.15.132:0/219858438 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:11:55.308 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:55.303+0000 7f409a59c700 1 -- 172.21.15.132:0/219858438 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4084005040 con 0x7f409c0601e0 2024-02-20T00:11:55.308 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:55.303+0000 7f409a59c700 1 --2- 172.21.15.132:0/219858438 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f409c0601e0 0x7f409c106600 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7f4084005010 tx=0x7f408400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:11:55.308 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:55.303+0000 7f4092ffd700 1 -- 172.21.15.132:0/219858438 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4084009320 con 0x7f409c0601e0 2024-02-20T00:11:55.308 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:55.303+0000 7f409b59e700 1 -- 172.21.15.132:0/219858438 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f409c106cf0 con 0x7f409c0601e0 2024-02-20T00:11:55.308 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:55.303+0000 7f4092ffd700 1 -- 172.21.15.132:0/219858438 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4084013380 con 0x7f409c0601e0 2024-02-20T00:11:55.309 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:55.303+0000 7f4092ffd700 1 -- 172.21.15.132:0/219858438 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4084014e50 con 0x7f409c0601e0 2024-02-20T00:11:55.309 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:55.303+0000 7f409b59e700 1 -- 172.21.15.132:0/219858438 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f409c107180 con 0x7f409c0601e0 2024-02-20T00:11:55.309 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:55.304+0000 7f4092ffd700 1 -- 172.21.15.132:0/219858438 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f40840095f0 con 0x7f409c0601e0 2024-02-20T00:11:55.309 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:55.304+0000 7f409b59e700 1 -- 172.21.15.132:0/219858438 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f409c045ad0 con 0x7f409c0601e0 2024-02-20T00:11:55.309 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:55.304+0000 7f4092ffd700 1 --2- 172.21.15.132:0/219858438 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f408802ff10 0x7f40880323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:11:55.309 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:55.305+0000 7f4092ffd700 1 -- 172.21.15.132:0/219858438 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4084040e90 con 0x7f409c0601e0 2024-02-20T00:11:55.309 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:55.305+0000 7f4099d9b700 1 --2- 172.21.15.132:0/219858438 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f408802ff10 0x7f40880323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:11:55.309 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:55.305+0000 7f4099d9b700 1 --2- 172.21.15.132:0/219858438 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f408802ff10 0x7f40880323d0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f408c0096a0 tx=0x7f408c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:11:55.311 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:55.309+0000 7f4092ffd700 1 -- 172.21.15.132:0/219858438 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4084014420 con 0x7f409c0601e0 2024-02-20T00:11:55.515 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:55.513+0000 7f409b59e700 1 -- 172.21.15.132:0/219858438 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f409c108030 con 0x7f409c0601e0 2024-02-20T00:11:55.516 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:55.513+0000 7f4092ffd700 1 -- 172.21.15.132:0/219858438 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4084010230 con 0x7f409c0601e0 2024-02-20T00:11:55.516 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:11:55.516 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:11:55.518 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:55.515+0000 7f409b59e700 1 -- 172.21.15.132:0/219858438 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f408802ff10 msgr2=0x7f40880323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:11:55.518 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:55.515+0000 7f409b59e700 1 --2- 172.21.15.132:0/219858438 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f408802ff10 0x7f40880323d0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f408c0096a0 tx=0x7f408c006b40).stop 2024-02-20T00:11:55.518 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:55.515+0000 7f409b59e700 1 -- 172.21.15.132:0/219858438 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f409c0601e0 msgr2=0x7f409c106600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:11:55.518 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:55.515+0000 7f409b59e700 1 --2- 172.21.15.132:0/219858438 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f409c0601e0 0x7f409c106600 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7f4084005010 tx=0x7f408400b040).stop 2024-02-20T00:11:55.518 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:55.515+0000 7f409b59e700 1 -- 172.21.15.132:0/219858438 shutdown_connections 2024-02-20T00:11:55.518 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:55.515+0000 7f409b59e700 1 --2- 172.21.15.132:0/219858438 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f408802ff10 0x7f40880323d0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:11:55.518 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:55.515+0000 7f409b59e700 1 --2- 172.21.15.132:0/219858438 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f409c0601e0 0x7f409c106600 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:11:55.518 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:55.515+0000 7f409b59e700 1 -- 172.21.15.132:0/219858438 >> 172.21.15.132:0/219858438 conn(0x7f409c0f09a0 msgr2=0x7f409c0f13d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:11:55.518 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:55.515+0000 7f409b59e700 1 -- 172.21.15.132:0/219858438 shutdown_connections 2024-02-20T00:11:55.518 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:55.515+0000 7f409b59e700 1 -- 172.21.15.132:0/219858438 wait complete. 2024-02-20T00:11:55.518 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:11:56.359 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:11:55 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:11:54.853232+0000 mgr.smithi084.bmqrnf (mgr.14162) 60 : cluster [DBG] pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:11:56.359 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:11:55 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:11:55.515185+0000 mon.smithi084 (mon.0) 141 : audit [DBG] from='client.? 172.21.15.132:0/219858438' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:11:57.198 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:11:57.198 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:11:58.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:11:57 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:11:56.853501+0000 mgr.smithi084.bmqrnf (mgr.14162) 61 : cluster [DBG] pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:11:58.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:11:57 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:11:56.971183+0000 mon.smithi084 (mon.0) 142 : audit [INF] from='mgr.14162 172.21.15.84:0/2152933216' entity='mgr.smithi084.bmqrnf' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi084.bmqrnf/mirror_snapshot_schedule"}]: dispatch 2024-02-20T00:11:58.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:11:57 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:11:56.977593+0000 mon.smithi084 (mon.0) 143 : audit [INF] from='mgr.14162 172.21.15.84:0/2152933216' entity='mgr.smithi084.bmqrnf' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi084.bmqrnf/trash_purge_schedule"}]: dispatch 2024-02-20T00:11:58.986 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:58.982+0000 7fe0b4dc5700 1 -- 172.21.15.132:0/927736883 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe0b00f5410 msgr2=0x7fe0b00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:11:58.986 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:58.982+0000 7fe0b4dc5700 1 --2- 172.21.15.132:0/927736883 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe0b00f5410 0x7fe0b00f5830 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7fe098009fb0 tx=0x7fe098009b30).stop 2024-02-20T00:11:58.987 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:58.982+0000 7fe0b4dc5700 1 -- 172.21.15.132:0/927736883 shutdown_connections 2024-02-20T00:11:58.987 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:58.982+0000 7fe0b4dc5700 1 --2- 172.21.15.132:0/927736883 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe0b00f5410 0x7fe0b00f5830 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:11:58.987 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:58.982+0000 7fe0b4dc5700 1 -- 172.21.15.132:0/927736883 >> 172.21.15.132:0/927736883 conn(0x7fe0b00f0b70 msgr2=0x7fe0b00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:11:58.987 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:58.982+0000 7fe0b4dc5700 1 -- 172.21.15.132:0/927736883 shutdown_connections 2024-02-20T00:11:58.987 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:58.982+0000 7fe0b4dc5700 1 -- 172.21.15.132:0/927736883 wait complete. 2024-02-20T00:11:58.987 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:58.983+0000 7fe0b4dc5700 1 Processor -- start 2024-02-20T00:11:58.987 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:58.983+0000 7fe0b4dc5700 1 -- start start 2024-02-20T00:11:58.987 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:58.983+0000 7fe0b4dc5700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe0b00f5410 0x7fe0b01056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:11:58.987 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:58.983+0000 7fe0b4dc5700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0b0105bd0 con 0x7fe0b00f5410 2024-02-20T00:11:58.987 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:58.983+0000 7fe0ae59c700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe0b00f5410 0x7fe0b01056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:11:58.987 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:58.983+0000 7fe0ae59c700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe0b00f5410 0x7fe0b01056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:53614/0 (socket says 172.21.15.132:53614) 2024-02-20T00:11:58.987 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:58.983+0000 7fe0ae59c700 1 -- 172.21.15.132:0/3989592783 learned_addr learned my addr 172.21.15.132:0/3989592783 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:11:58.987 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:58.984+0000 7fe0ae59c700 1 -- 172.21.15.132:0/3989592783 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe098005040 con 0x7fe0b00f5410 2024-02-20T00:11:58.988 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:58.984+0000 7fe0ae59c700 1 --2- 172.21.15.132:0/3989592783 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe0b00f5410 0x7fe0b01056e0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7fe098005010 tx=0x7fe09800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:11:58.988 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:58.984+0000 7fe0a6ffd700 1 -- 172.21.15.132:0/3989592783 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe098009320 con 0x7fe0b00f5410 2024-02-20T00:11:58.988 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:58.984+0000 7fe0b4dc5700 1 -- 172.21.15.132:0/3989592783 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0b0105dd0 con 0x7fe0b00f5410 2024-02-20T00:11:58.988 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:58.984+0000 7fe0a6ffd700 1 -- 172.21.15.132:0/3989592783 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe098013380 con 0x7fe0b00f5410 2024-02-20T00:11:58.988 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:58.984+0000 7fe0a6ffd700 1 -- 172.21.15.132:0/3989592783 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe098014e50 con 0x7fe0b00f5410 2024-02-20T00:11:58.988 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:58.984+0000 7fe0b4dc5700 1 -- 172.21.15.132:0/3989592783 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0b0106260 con 0x7fe0b00f5410 2024-02-20T00:11:58.989 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:58.985+0000 7fe0a6ffd700 1 -- 172.21.15.132:0/3989592783 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe098014420 con 0x7fe0b00f5410 2024-02-20T00:11:58.989 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:58.985+0000 7fe0b4dc5700 1 -- 172.21.15.132:0/3989592783 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0b0045ad0 con 0x7fe0b00f5410 2024-02-20T00:11:58.989 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:58.985+0000 7fe0a6ffd700 1 --2- 172.21.15.132:0/3989592783 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fe09c02ff10 0x7fe09c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:11:58.989 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:58.985+0000 7fe0a6ffd700 1 -- 172.21.15.132:0/3989592783 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe098040f90 con 0x7fe0b00f5410 2024-02-20T00:11:58.989 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:58.985+0000 7fe0add9b700 1 --2- 172.21.15.132:0/3989592783 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fe09c02ff10 0x7fe09c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:11:58.989 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:58.986+0000 7fe0add9b700 1 --2- 172.21.15.132:0/3989592783 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fe09c02ff10 0x7fe09c0323d0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7fe0a00096a0 tx=0x7fe0a0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:11:58.991 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:58.989+0000 7fe0a6ffd700 1 -- 172.21.15.132:0/3989592783 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe098014790 con 0x7fe0b00f5410 2024-02-20T00:11:59.198 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:59.196+0000 7fe0b4dc5700 1 -- 172.21.15.132:0/3989592783 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe0b0107110 con 0x7fe0b00f5410 2024-02-20T00:11:59.199 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:59.197+0000 7fe0a6ffd700 1 -- 172.21.15.132:0/3989592783 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe09801d380 con 0x7fe0b00f5410 2024-02-20T00:11:59.199 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:11:59.199 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:11:59.200 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:59.198+0000 7fe0b4dc5700 1 -- 172.21.15.132:0/3989592783 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fe09c02ff10 msgr2=0x7fe09c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:11:59.200 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:59.198+0000 7fe0b4dc5700 1 --2- 172.21.15.132:0/3989592783 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fe09c02ff10 0x7fe09c0323d0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7fe0a00096a0 tx=0x7fe0a0006b40).stop 2024-02-20T00:11:59.201 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:59.198+0000 7fe0b4dc5700 1 -- 172.21.15.132:0/3989592783 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe0b00f5410 msgr2=0x7fe0b01056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:11:59.201 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:59.198+0000 7fe0b4dc5700 1 --2- 172.21.15.132:0/3989592783 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe0b00f5410 0x7fe0b01056e0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7fe098005010 tx=0x7fe09800b040).stop 2024-02-20T00:11:59.201 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:59.198+0000 7fe0b4dc5700 1 -- 172.21.15.132:0/3989592783 shutdown_connections 2024-02-20T00:11:59.201 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:59.198+0000 7fe0b4dc5700 1 --2- 172.21.15.132:0/3989592783 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fe09c02ff10 0x7fe09c0323d0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:11:59.201 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:59.198+0000 7fe0b4dc5700 1 --2- 172.21.15.132:0/3989592783 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe0b00f5410 0x7fe0b01056e0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:11:59.201 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:59.198+0000 7fe0b4dc5700 1 -- 172.21.15.132:0/3989592783 >> 172.21.15.132:0/3989592783 conn(0x7fe0b00f0b70 msgr2=0x7fe0b00f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:11:59.201 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:59.198+0000 7fe0b4dc5700 1 -- 172.21.15.132:0/3989592783 shutdown_connections 2024-02-20T00:11:59.201 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:11:59.198+0000 7fe0b4dc5700 1 -- 172.21.15.132:0/3989592783 wait complete. 2024-02-20T00:11:59.201 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:12:00.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:11:59 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:11:58.853843+0000 mgr.smithi084.bmqrnf (mgr.14162) 62 : cluster [DBG] pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:12:00.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:11:59 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:11:59.198249+0000 mon.smithi084 (mon.0) 144 : audit [DBG] from='client.? 172.21.15.132:0/3989592783' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:12:00.848 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:12:00.849 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:12:02.107 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:12:01 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:12:00.854481+0000 mgr.smithi084.bmqrnf (mgr.14162) 63 : cluster [DBG] pgmap v56: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:12:02.588 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:02.585+0000 7f53294e0700 1 -- 172.21.15.132:0/1861870124 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f53240f6510 msgr2=0x7f53240f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:12:02.588 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:02.585+0000 7f53294e0700 1 --2- 172.21.15.132:0/1861870124 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f53240f6510 0x7f53240f6930 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7f530c009fb0 tx=0x7f530c009b30).stop 2024-02-20T00:12:02.588 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:02.585+0000 7f53294e0700 1 -- 172.21.15.132:0/1861870124 shutdown_connections 2024-02-20T00:12:02.588 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:02.585+0000 7f53294e0700 1 --2- 172.21.15.132:0/1861870124 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f53240f6510 0x7f53240f6930 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:12:02.588 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:02.585+0000 7f53294e0700 1 -- 172.21.15.132:0/1861870124 >> 172.21.15.132:0/1861870124 conn(0x7f53240f1c70 msgr2=0x7f53240f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:12:02.588 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:02.585+0000 7f53294e0700 1 -- 172.21.15.132:0/1861870124 shutdown_connections 2024-02-20T00:12:02.588 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:02.585+0000 7f53294e0700 1 -- 172.21.15.132:0/1861870124 wait complete. 2024-02-20T00:12:02.590 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:02.586+0000 7f53294e0700 1 Processor -- start 2024-02-20T00:12:02.590 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:02.586+0000 7f53294e0700 1 -- start start 2024-02-20T00:12:02.590 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:02.586+0000 7f53294e0700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f53240f6510 0x7f53241067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:12:02.590 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:02.586+0000 7f53294e0700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5324106cd0 con 0x7f53240f6510 2024-02-20T00:12:02.590 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:02.586+0000 7f5322ffd700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f53240f6510 0x7f53241067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:12:02.591 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:02.586+0000 7f5322ffd700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f53240f6510 0x7f53241067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:53624/0 (socket says 172.21.15.132:53624) 2024-02-20T00:12:02.591 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:02.586+0000 7f5322ffd700 1 -- 172.21.15.132:0/4070570237 learned_addr learned my addr 172.21.15.132:0/4070570237 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:12:02.591 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:02.587+0000 7f5322ffd700 1 -- 172.21.15.132:0/4070570237 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f530c005040 con 0x7f53240f6510 2024-02-20T00:12:02.591 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:02.587+0000 7f5322ffd700 1 --2- 172.21.15.132:0/4070570237 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f53240f6510 0x7f53241067e0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f530c005010 tx=0x7f530c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:12:02.591 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:02.587+0000 7f531b7fe700 1 -- 172.21.15.132:0/4070570237 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f530c009320 con 0x7f53240f6510 2024-02-20T00:12:02.591 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:02.587+0000 7f53294e0700 1 -- 172.21.15.132:0/4070570237 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5324106ed0 con 0x7f53240f6510 2024-02-20T00:12:02.591 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:02.587+0000 7f531b7fe700 1 -- 172.21.15.132:0/4070570237 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f530c013380 con 0x7f53240f6510 2024-02-20T00:12:02.591 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:02.587+0000 7f531b7fe700 1 -- 172.21.15.132:0/4070570237 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f530c014e50 con 0x7f53240f6510 2024-02-20T00:12:02.591 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:02.587+0000 7f53294e0700 1 -- 172.21.15.132:0/4070570237 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5324107360 con 0x7f53240f6510 2024-02-20T00:12:02.591 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:02.588+0000 7f531b7fe700 1 -- 172.21.15.132:0/4070570237 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f530c0095f0 con 0x7f53240f6510 2024-02-20T00:12:02.591 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:02.588+0000 7f531b7fe700 1 --2- 172.21.15.132:0/4070570237 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f531002ff10 0x7f53100323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:12:02.591 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:02.588+0000 7f53294e0700 1 -- 172.21.15.132:0/4070570237 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5324045ad0 con 0x7f53240f6510 2024-02-20T00:12:02.591 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:02.588+0000 7f531b7fe700 1 -- 172.21.15.132:0/4070570237 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f530c040e90 con 0x7f53240f6510 2024-02-20T00:12:02.591 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:02.588+0000 7f53227fc700 1 --2- 172.21.15.132:0/4070570237 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f531002ff10 0x7f53100323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:12:02.592 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:02.589+0000 7f53227fc700 1 --2- 172.21.15.132:0/4070570237 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f531002ff10 0x7f53100323d0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f53140096a0 tx=0x7f5314006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:12:02.595 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:02.593+0000 7f531b7fe700 1 -- 172.21.15.132:0/4070570237 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f530c014420 con 0x7f53240f6510 2024-02-20T00:12:02.799 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:02.797+0000 7f53294e0700 1 -- 172.21.15.132:0/4070570237 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5324108210 con 0x7f53240f6510 2024-02-20T00:12:02.800 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:02.798+0000 7f531b7fe700 1 -- 172.21.15.132:0/4070570237 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f530c010230 con 0x7f53240f6510 2024-02-20T00:12:02.800 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:12:02.800 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:12:02.802 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:02.799+0000 7f53294e0700 1 -- 172.21.15.132:0/4070570237 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f531002ff10 msgr2=0x7f53100323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:12:02.802 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:02.799+0000 7f53294e0700 1 --2- 172.21.15.132:0/4070570237 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f531002ff10 0x7f53100323d0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f53140096a0 tx=0x7f5314006b40).stop 2024-02-20T00:12:02.802 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:02.799+0000 7f53294e0700 1 -- 172.21.15.132:0/4070570237 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f53240f6510 msgr2=0x7f53241067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:12:02.802 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:02.799+0000 7f53294e0700 1 --2- 172.21.15.132:0/4070570237 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f53240f6510 0x7f53241067e0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f530c005010 tx=0x7f530c00b040).stop 2024-02-20T00:12:02.802 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:02.799+0000 7f53294e0700 1 -- 172.21.15.132:0/4070570237 shutdown_connections 2024-02-20T00:12:02.802 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:02.799+0000 7f53294e0700 1 --2- 172.21.15.132:0/4070570237 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f531002ff10 0x7f53100323d0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:12:02.802 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:02.799+0000 7f53294e0700 1 --2- 172.21.15.132:0/4070570237 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f53240f6510 0x7f53241067e0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:12:02.802 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:02.799+0000 7f53294e0700 1 -- 172.21.15.132:0/4070570237 >> 172.21.15.132:0/4070570237 conn(0x7f53240f1c70 msgr2=0x7f53240f26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:12:02.802 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:02.799+0000 7f53294e0700 1 -- 172.21.15.132:0/4070570237 shutdown_connections 2024-02-20T00:12:02.802 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:02.799+0000 7f53294e0700 1 -- 172.21.15.132:0/4070570237 wait complete. 2024-02-20T00:12:02.803 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:12:03.107 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:12:02 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:12:02.799258+0000 mon.smithi084 (mon.0) 145 : audit [DBG] from='client.? 172.21.15.132:0/4070570237' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:12:04.107 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:12:03 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:12:02.854758+0000 mgr.smithi084.bmqrnf (mgr.14162) 64 : cluster [DBG] pgmap v57: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:12:04.457 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:12:04.458 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:12:06.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:12:05 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:12:04.855192+0000 mgr.smithi084.bmqrnf (mgr.14162) 65 : cluster [DBG] pgmap v58: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:12:06.479 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:06.475+0000 7f23934d1700 1 -- 172.21.15.132:0/4189775625 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f238c0f6510 msgr2=0x7f238c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:12:06.479 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:06.475+0000 7f23934d1700 1 --2- 172.21.15.132:0/4189775625 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f238c0f6510 0x7f238c0f6930 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f2388009fb0 tx=0x7f2388009b30).stop 2024-02-20T00:12:06.479 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:06.475+0000 7f23934d1700 1 -- 172.21.15.132:0/4189775625 shutdown_connections 2024-02-20T00:12:06.480 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:06.475+0000 7f23934d1700 1 --2- 172.21.15.132:0/4189775625 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f238c0f6510 0x7f238c0f6930 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:12:06.480 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:06.475+0000 7f23934d1700 1 -- 172.21.15.132:0/4189775625 >> 172.21.15.132:0/4189775625 conn(0x7f238c0f1c70 msgr2=0x7f238c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:12:06.480 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:06.475+0000 7f23934d1700 1 -- 172.21.15.132:0/4189775625 shutdown_connections 2024-02-20T00:12:06.480 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:06.475+0000 7f23934d1700 1 -- 172.21.15.132:0/4189775625 wait complete. 2024-02-20T00:12:06.480 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:06.476+0000 7f23934d1700 1 Processor -- start 2024-02-20T00:12:06.480 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:06.476+0000 7f23934d1700 1 -- start start 2024-02-20T00:12:06.480 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:06.476+0000 7f23934d1700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f238c0f6510 0x7f238c1067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:12:06.480 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:06.476+0000 7f23934d1700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f238c106cd0 con 0x7f238c0f6510 2024-02-20T00:12:06.480 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:06.476+0000 7f239126d700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f238c0f6510 0x7f238c1067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:12:06.480 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:06.476+0000 7f239126d700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f238c0f6510 0x7f238c1067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:53646/0 (socket says 172.21.15.132:53646) 2024-02-20T00:12:06.480 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:06.476+0000 7f239126d700 1 -- 172.21.15.132:0/1688363995 learned_addr learned my addr 172.21.15.132:0/1688363995 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:12:06.480 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:06.477+0000 7f239126d700 1 -- 172.21.15.132:0/1688363995 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2388005040 con 0x7f238c0f6510 2024-02-20T00:12:06.481 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:06.477+0000 7f239126d700 1 --2- 172.21.15.132:0/1688363995 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f238c0f6510 0x7f238c1067e0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f2388005010 tx=0x7f238800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:12:06.481 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:06.477+0000 7f237dffb700 1 -- 172.21.15.132:0/1688363995 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2388009320 con 0x7f238c0f6510 2024-02-20T00:12:06.481 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:06.477+0000 7f23934d1700 1 -- 172.21.15.132:0/1688363995 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f238c106ed0 con 0x7f238c0f6510 2024-02-20T00:12:06.481 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:06.477+0000 7f237dffb700 1 -- 172.21.15.132:0/1688363995 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2388013380 con 0x7f238c0f6510 2024-02-20T00:12:06.481 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:06.477+0000 7f237dffb700 1 -- 172.21.15.132:0/1688363995 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2388014e50 con 0x7f238c0f6510 2024-02-20T00:12:06.481 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:06.477+0000 7f23934d1700 1 -- 172.21.15.132:0/1688363995 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f238c107360 con 0x7f238c0f6510 2024-02-20T00:12:06.482 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:06.478+0000 7f237dffb700 1 -- 172.21.15.132:0/1688363995 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f23880095f0 con 0x7f238c0f6510 2024-02-20T00:12:06.482 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:06.478+0000 7f23934d1700 1 -- 172.21.15.132:0/1688363995 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f238c045ad0 con 0x7f238c0f6510 2024-02-20T00:12:06.482 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:06.478+0000 7f237dffb700 1 --2- 172.21.15.132:0/1688363995 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f237802ff10 0x7f23780323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:12:06.482 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:06.478+0000 7f237dffb700 1 -- 172.21.15.132:0/1688363995 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2388040f80 con 0x7f238c0f6510 2024-02-20T00:12:06.482 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:06.478+0000 7f2390a6c700 1 --2- 172.21.15.132:0/1688363995 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f237802ff10 0x7f23780323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:12:06.482 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:06.479+0000 7f2390a6c700 1 --2- 172.21.15.132:0/1688363995 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f237802ff10 0x7f23780323d0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f23800096a0 tx=0x7f2380006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:12:06.485 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:06.483+0000 7f237dffb700 1 -- 172.21.15.132:0/1688363995 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2388014420 con 0x7f238c0f6510 2024-02-20T00:12:06.690 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:06.688+0000 7f23934d1700 1 -- 172.21.15.132:0/1688363995 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f238c108210 con 0x7f238c0f6510 2024-02-20T00:12:06.691 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:06.689+0000 7f237dffb700 1 -- 172.21.15.132:0/1688363995 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2388010230 con 0x7f238c0f6510 2024-02-20T00:12:06.691 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:12:06.691 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:12:06.693 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:06.690+0000 7f23934d1700 1 -- 172.21.15.132:0/1688363995 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f237802ff10 msgr2=0x7f23780323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:12:06.693 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:06.690+0000 7f23934d1700 1 --2- 172.21.15.132:0/1688363995 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f237802ff10 0x7f23780323d0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f23800096a0 tx=0x7f2380006b40).stop 2024-02-20T00:12:06.693 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:06.690+0000 7f23934d1700 1 -- 172.21.15.132:0/1688363995 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f238c0f6510 msgr2=0x7f238c1067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:12:06.693 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:06.690+0000 7f23934d1700 1 --2- 172.21.15.132:0/1688363995 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f238c0f6510 0x7f238c1067e0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f2388005010 tx=0x7f238800b040).stop 2024-02-20T00:12:06.693 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:06.690+0000 7f23934d1700 1 -- 172.21.15.132:0/1688363995 shutdown_connections 2024-02-20T00:12:06.693 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:06.690+0000 7f23934d1700 1 --2- 172.21.15.132:0/1688363995 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f237802ff10 0x7f23780323d0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:12:06.693 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:06.690+0000 7f23934d1700 1 --2- 172.21.15.132:0/1688363995 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f238c0f6510 0x7f238c1067e0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:12:06.693 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:06.690+0000 7f23934d1700 1 -- 172.21.15.132:0/1688363995 >> 172.21.15.132:0/1688363995 conn(0x7f238c0f1c70 msgr2=0x7f238c0f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:12:06.693 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:06.690+0000 7f23934d1700 1 -- 172.21.15.132:0/1688363995 shutdown_connections 2024-02-20T00:12:06.693 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:06.690+0000 7f23934d1700 1 -- 172.21.15.132:0/1688363995 wait complete. 2024-02-20T00:12:06.694 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:12:07.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:12:06 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:12:06.690217+0000 mon.smithi084 (mon.0) 146 : audit [DBG] from='client.? 172.21.15.132:0/1688363995' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:12:08.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:12:07 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:12:06.855476+0000 mgr.smithi084.bmqrnf (mgr.14162) 66 : cluster [DBG] pgmap v59: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:12:08.410 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:12:08.410 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:12:10.192 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:10.189+0000 7fc324925700 1 -- 172.21.15.132:0/1506972910 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc3200f5410 msgr2=0x7fc3200f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:12:10.192 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:10.189+0000 7fc324925700 1 --2- 172.21.15.132:0/1506972910 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc3200f5410 0x7fc3200f5830 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7fc308009fb0 tx=0x7fc308009b30).stop 2024-02-20T00:12:10.193 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:10.189+0000 7fc324925700 1 -- 172.21.15.132:0/1506972910 shutdown_connections 2024-02-20T00:12:10.193 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:10.189+0000 7fc324925700 1 --2- 172.21.15.132:0/1506972910 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc3200f5410 0x7fc3200f5830 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:12:10.193 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:10.189+0000 7fc324925700 1 -- 172.21.15.132:0/1506972910 >> 172.21.15.132:0/1506972910 conn(0x7fc3200f0b70 msgr2=0x7fc3200f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:12:10.193 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:10.189+0000 7fc324925700 1 -- 172.21.15.132:0/1506972910 shutdown_connections 2024-02-20T00:12:10.193 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:10.189+0000 7fc324925700 1 -- 172.21.15.132:0/1506972910 wait complete. 2024-02-20T00:12:10.193 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:10.190+0000 7fc324925700 1 Processor -- start 2024-02-20T00:12:10.193 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:10.190+0000 7fc324925700 1 -- start start 2024-02-20T00:12:10.193 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:10.190+0000 7fc324925700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc3200f5410 0x7fc3200fd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:12:10.193 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:10.190+0000 7fc324925700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3200fb960 con 0x7fc3200f5410 2024-02-20T00:12:10.193 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:10.190+0000 7fc31e59c700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc3200f5410 0x7fc3200fd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:12:10.194 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:10.190+0000 7fc31e59c700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc3200f5410 0x7fc3200fd4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:60228/0 (socket says 172.21.15.132:60228) 2024-02-20T00:12:10.194 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:10.190+0000 7fc31e59c700 1 -- 172.21.15.132:0/107681370 learned_addr learned my addr 172.21.15.132:0/107681370 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:12:10.194 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:10.191+0000 7fc31e59c700 1 -- 172.21.15.132:0/107681370 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc308005040 con 0x7fc3200f5410 2024-02-20T00:12:10.194 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:10.191+0000 7fc31e59c700 1 --2- 172.21.15.132:0/107681370 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc3200f5410 0x7fc3200fd4e0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7fc308005010 tx=0x7fc30800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:12:10.194 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:10.191+0000 7fc316ffd700 1 -- 172.21.15.132:0/107681370 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc308009320 con 0x7fc3200f5410 2024-02-20T00:12:10.194 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:10.191+0000 7fc324925700 1 -- 172.21.15.132:0/107681370 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3200fbb00 con 0x7fc3200f5410 2024-02-20T00:12:10.194 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:10.191+0000 7fc316ffd700 1 -- 172.21.15.132:0/107681370 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc308013380 con 0x7fc3200f5410 2024-02-20T00:12:10.194 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:10.191+0000 7fc316ffd700 1 -- 172.21.15.132:0/107681370 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc308014e50 con 0x7fc3200f5410 2024-02-20T00:12:10.194 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:10.191+0000 7fc324925700 1 -- 172.21.15.132:0/107681370 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3200fbf90 con 0x7fc3200f5410 2024-02-20T00:12:10.195 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:10.192+0000 7fc316ffd700 1 -- 172.21.15.132:0/107681370 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc3080263a0 con 0x7fc3200f5410 2024-02-20T00:12:10.195 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:10.192+0000 7fc316ffd700 1 --2- 172.21.15.132:0/107681370 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fc30c02ff10 0x7fc30c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:12:10.195 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:10.192+0000 7fc324925700 1 -- 172.21.15.132:0/107681370 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc320059100 con 0x7fc3200f5410 2024-02-20T00:12:10.195 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:10.192+0000 7fc316ffd700 1 -- 172.21.15.132:0/107681370 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc308040fc0 con 0x7fc3200f5410 2024-02-20T00:12:10.195 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:10.192+0000 7fc31dd9b700 1 --2- 172.21.15.132:0/107681370 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fc30c02ff10 0x7fc30c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:12:10.195 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:10.192+0000 7fc31dd9b700 1 --2- 172.21.15.132:0/107681370 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fc30c02ff10 0x7fc30c0323d0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7fc3100096a0 tx=0x7fc310006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:12:10.198 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:10.197+0000 7fc316ffd700 1 -- 172.21.15.132:0/107681370 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc308012920 con 0x7fc3200f5410 2024-02-20T00:12:10.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:12:09 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:12:08.855823+0000 mgr.smithi084.bmqrnf (mgr.14162) 67 : cluster [DBG] pgmap v60: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:12:10.402 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:10.401+0000 7fc324925700 1 -- 172.21.15.132:0/107681370 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc3200fafa0 con 0x7fc3200f5410 2024-02-20T00:12:10.402 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:10.401+0000 7fc316ffd700 1 -- 172.21.15.132:0/107681370 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc308012330 con 0x7fc3200f5410 2024-02-20T00:12:10.403 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:12:10.403 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:12:10.404 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:10.403+0000 7fc324925700 1 -- 172.21.15.132:0/107681370 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fc30c02ff10 msgr2=0x7fc30c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:12:10.404 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:10.403+0000 7fc324925700 1 --2- 172.21.15.132:0/107681370 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fc30c02ff10 0x7fc30c0323d0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7fc3100096a0 tx=0x7fc310006b40).stop 2024-02-20T00:12:10.404 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:10.403+0000 7fc324925700 1 -- 172.21.15.132:0/107681370 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc3200f5410 msgr2=0x7fc3200fd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:12:10.404 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:10.403+0000 7fc324925700 1 --2- 172.21.15.132:0/107681370 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc3200f5410 0x7fc3200fd4e0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7fc308005010 tx=0x7fc30800b040).stop 2024-02-20T00:12:10.404 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:10.403+0000 7fc324925700 1 -- 172.21.15.132:0/107681370 shutdown_connections 2024-02-20T00:12:10.405 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:10.403+0000 7fc324925700 1 --2- 172.21.15.132:0/107681370 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fc30c02ff10 0x7fc30c0323d0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:12:10.405 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:10.403+0000 7fc324925700 1 --2- 172.21.15.132:0/107681370 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc3200f5410 0x7fc3200fd4e0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:12:10.405 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:10.403+0000 7fc324925700 1 -- 172.21.15.132:0/107681370 >> 172.21.15.132:0/107681370 conn(0x7fc3200f0b70 msgr2=0x7fc32005fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:12:10.405 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:10.403+0000 7fc324925700 1 -- 172.21.15.132:0/107681370 shutdown_connections 2024-02-20T00:12:10.405 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:10.403+0000 7fc324925700 1 -- 172.21.15.132:0/107681370 wait complete. 2024-02-20T00:12:10.405 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:12:11.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:12:10 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:12:10.402011+0000 mon.smithi084 (mon.0) 147 : audit [DBG] from='client.? 172.21.15.132:0/107681370' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:12:12.035 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:12:12.035 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:12:12.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:12:11 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:12:10.856163+0000 mgr.smithi084.bmqrnf (mgr.14162) 68 : cluster [DBG] pgmap v61: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:12:13.804 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:13.802+0000 7f8b15167700 1 -- 172.21.15.132:0/1213678516 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8b10060240 msgr2=0x7f8b100f9130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:12:13.804 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:13.802+0000 7f8b15167700 1 --2- 172.21.15.132:0/1213678516 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8b10060240 0x7f8b100f9130 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7f8b00009fb0 tx=0x7f8b00009b30).stop 2024-02-20T00:12:13.804 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:13.802+0000 7f8b15167700 1 -- 172.21.15.132:0/1213678516 shutdown_connections 2024-02-20T00:12:13.804 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:13.802+0000 7f8b15167700 1 --2- 172.21.15.132:0/1213678516 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8b10060240 0x7f8b100f9130 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:12:13.804 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:13.802+0000 7f8b15167700 1 -- 172.21.15.132:0/1213678516 >> 172.21.15.132:0/1213678516 conn(0x7f8b100f0b70 msgr2=0x7f8b100f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:12:13.804 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:13.802+0000 7f8b15167700 1 -- 172.21.15.132:0/1213678516 shutdown_connections 2024-02-20T00:12:13.804 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:13.802+0000 7f8b15167700 1 -- 172.21.15.132:0/1213678516 wait complete. 2024-02-20T00:12:13.807 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:13.803+0000 7f8b15167700 1 Processor -- start 2024-02-20T00:12:13.807 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:13.803+0000 7f8b15167700 1 -- start start 2024-02-20T00:12:13.807 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:13.803+0000 7f8b15167700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8b10060240 0x7f8b100f6bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:12:13.807 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:13.803+0000 7f8b15167700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b100f5030 con 0x7f8b10060240 2024-02-20T00:12:13.807 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:13.803+0000 7f8b0ed9d700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8b10060240 0x7f8b100f6bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:12:13.807 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:13.803+0000 7f8b0ed9d700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8b10060240 0x7f8b100f6bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:60258/0 (socket says 172.21.15.132:60258) 2024-02-20T00:12:13.807 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:13.803+0000 7f8b0ed9d700 1 -- 172.21.15.132:0/3462085417 learned_addr learned my addr 172.21.15.132:0/3462085417 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:12:13.807 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:13.804+0000 7f8b0ed9d700 1 -- 172.21.15.132:0/3462085417 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b00005040 con 0x7f8b10060240 2024-02-20T00:12:13.807 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:13.804+0000 7f8b0ed9d700 1 --2- 172.21.15.132:0/3462085417 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8b10060240 0x7f8b100f6bb0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f8b000089f0 tx=0x7f8b0000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:12:13.807 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:13.804+0000 7f8af77fe700 1 -- 172.21.15.132:0/3462085417 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8b00009320 con 0x7f8b10060240 2024-02-20T00:12:13.807 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:13.804+0000 7f8b15167700 1 -- 172.21.15.132:0/3462085417 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b100f51d0 con 0x7f8b10060240 2024-02-20T00:12:13.807 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:13.804+0000 7f8af77fe700 1 -- 172.21.15.132:0/3462085417 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8b00013380 con 0x7f8b10060240 2024-02-20T00:12:13.807 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:13.804+0000 7f8af77fe700 1 -- 172.21.15.132:0/3462085417 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8b00014e20 con 0x7f8b10060240 2024-02-20T00:12:13.808 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:13.804+0000 7f8b15167700 1 -- 172.21.15.132:0/3462085417 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b100f5660 con 0x7f8b10060240 2024-02-20T00:12:13.808 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:13.805+0000 7f8af77fe700 1 -- 172.21.15.132:0/3462085417 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8b000095f0 con 0x7f8b10060240 2024-02-20T00:12:13.808 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:13.805+0000 7f8b15167700 1 -- 172.21.15.132:0/3462085417 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b10059100 con 0x7f8b10060240 2024-02-20T00:12:13.808 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:13.805+0000 7f8af77fe700 1 --2- 172.21.15.132:0/3462085417 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f8af802ff10 0x7f8af80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:12:13.808 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:13.805+0000 7f8af77fe700 1 -- 172.21.15.132:0/3462085417 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8b00040f70 con 0x7f8b10060240 2024-02-20T00:12:13.808 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:13.805+0000 7f8b0e59c700 1 --2- 172.21.15.132:0/3462085417 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f8af802ff10 0x7f8af80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:12:13.808 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:13.806+0000 7f8b0e59c700 1 --2- 172.21.15.132:0/3462085417 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f8af802ff10 0x7f8af80323d0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f8b040096a0 tx=0x7f8b04006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:12:13.811 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:13.810+0000 7f8af77fe700 1 -- 172.21.15.132:0/3462085417 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8b0006d2b0 con 0x7f8b10060240 2024-02-20T00:12:14.017 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:14.015+0000 7f8b15167700 1 -- 172.21.15.132:0/3462085417 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8b10045ad0 con 0x7f8b10060240 2024-02-20T00:12:14.018 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:14.016+0000 7f8af77fe700 1 -- 172.21.15.132:0/3462085417 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8b00014420 con 0x7f8b10060240 2024-02-20T00:12:14.018 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:12:14.018 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:12:14.019 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:14.018+0000 7f8b15167700 1 -- 172.21.15.132:0/3462085417 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f8af802ff10 msgr2=0x7f8af80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:12:14.019 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:14.018+0000 7f8b15167700 1 --2- 172.21.15.132:0/3462085417 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f8af802ff10 0x7f8af80323d0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f8b040096a0 tx=0x7f8b04006b40).stop 2024-02-20T00:12:14.020 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:14.018+0000 7f8b15167700 1 -- 172.21.15.132:0/3462085417 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8b10060240 msgr2=0x7f8b100f6bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:12:14.020 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:14.018+0000 7f8b15167700 1 --2- 172.21.15.132:0/3462085417 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8b10060240 0x7f8b100f6bb0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f8b000089f0 tx=0x7f8b0000b040).stop 2024-02-20T00:12:14.020 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:14.018+0000 7f8b15167700 1 -- 172.21.15.132:0/3462085417 shutdown_connections 2024-02-20T00:12:14.020 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:14.018+0000 7f8b15167700 1 --2- 172.21.15.132:0/3462085417 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f8af802ff10 0x7f8af80323d0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:12:14.020 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:14.018+0000 7f8b15167700 1 --2- 172.21.15.132:0/3462085417 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8b10060240 0x7f8b100f6bb0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:12:14.020 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:14.018+0000 7f8b15167700 1 -- 172.21.15.132:0/3462085417 >> 172.21.15.132:0/3462085417 conn(0x7f8b100f0b70 msgr2=0x7f8b100f15d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:12:14.020 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:14.018+0000 7f8b15167700 1 -- 172.21.15.132:0/3462085417 shutdown_connections 2024-02-20T00:12:14.020 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:14.018+0000 7f8b15167700 1 -- 172.21.15.132:0/3462085417 wait complete. 2024-02-20T00:12:14.020 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:12:14.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:12:13 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:12:12.856419+0000 mgr.smithi084.bmqrnf (mgr.14162) 69 : cluster [DBG] pgmap v62: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:12:15.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:12:14 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:12:14.016794+0000 mon.smithi084 (mon.0) 148 : audit [DBG] from='client.? 172.21.15.132:0/3462085417' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:12:15.638 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:12:15.638 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:12:16.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:12:15 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:12:14.856737+0000 mgr.smithi084.bmqrnf (mgr.14162) 70 : cluster [DBG] pgmap v63: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:12:17.434 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:17.431+0000 7f7436568700 1 -- 172.21.15.132:0/822890601 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f74300f51a0 msgr2=0x7f74300f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:12:17.434 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:17.431+0000 7f7436568700 1 --2- 172.21.15.132:0/822890601 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f74300f51a0 0x7f74300f55c0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f7420009fb0 tx=0x7f7420009b30).stop 2024-02-20T00:12:17.434 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:17.431+0000 7f7436568700 1 -- 172.21.15.132:0/822890601 shutdown_connections 2024-02-20T00:12:17.434 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:17.431+0000 7f7436568700 1 --2- 172.21.15.132:0/822890601 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f74300f51a0 0x7f74300f55c0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:12:17.434 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:17.431+0000 7f7436568700 1 -- 172.21.15.132:0/822890601 >> 172.21.15.132:0/822890601 conn(0x7f74300f09a0 msgr2=0x7f74300f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:12:17.434 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:17.431+0000 7f7436568700 1 -- 172.21.15.132:0/822890601 shutdown_connections 2024-02-20T00:12:17.434 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:17.431+0000 7f7436568700 1 -- 172.21.15.132:0/822890601 wait complete. 2024-02-20T00:12:17.435 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:17.431+0000 7f7436568700 1 Processor -- start 2024-02-20T00:12:17.435 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:17.431+0000 7f7436568700 1 -- start start 2024-02-20T00:12:17.435 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:17.432+0000 7f7436568700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f74300f51a0 0x7f74301054d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:12:17.435 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:17.432+0000 7f7436568700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74301059c0 con 0x7f74300f51a0 2024-02-20T00:12:17.435 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:17.432+0000 7f7435566700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f74300f51a0 0x7f74301054d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:12:17.435 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:17.432+0000 7f7435566700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f74300f51a0 0x7f74301054d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:57368/0 (socket says 172.21.15.132:57368) 2024-02-20T00:12:17.435 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:17.432+0000 7f7435566700 1 -- 172.21.15.132:0/4078976901 learned_addr learned my addr 172.21.15.132:0/4078976901 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:12:17.435 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:17.432+0000 7f7435566700 1 -- 172.21.15.132:0/4078976901 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7420005040 con 0x7f74300f51a0 2024-02-20T00:12:17.435 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:17.432+0000 7f7435566700 1 --2- 172.21.15.132:0/4078976901 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f74300f51a0 0x7f74301054d0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f74200089f0 tx=0x7f742000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:12:17.435 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:17.432+0000 7f7425ffb700 1 -- 172.21.15.132:0/4078976901 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7420009320 con 0x7f74300f51a0 2024-02-20T00:12:17.435 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:17.432+0000 7f7436568700 1 -- 172.21.15.132:0/4078976901 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7430105bc0 con 0x7f74300f51a0 2024-02-20T00:12:17.435 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:17.432+0000 7f7425ffb700 1 -- 172.21.15.132:0/4078976901 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7420013380 con 0x7f74300f51a0 2024-02-20T00:12:17.435 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:17.432+0000 7f7425ffb700 1 -- 172.21.15.132:0/4078976901 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7420014e20 con 0x7f74300f51a0 2024-02-20T00:12:17.436 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:17.433+0000 7f7436568700 1 -- 172.21.15.132:0/4078976901 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7430106050 con 0x7f74300f51a0 2024-02-20T00:12:17.436 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:17.433+0000 7f7425ffb700 1 -- 172.21.15.132:0/4078976901 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7420014420 con 0x7f74300f51a0 2024-02-20T00:12:17.436 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:17.433+0000 7f7436568700 1 -- 172.21.15.132:0/4078976901 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7430045ad0 con 0x7f74300f51a0 2024-02-20T00:12:17.437 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:17.434+0000 7f7425ffb700 1 --2- 172.21.15.132:0/4078976901 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f741c02ff10 0x7f741c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:12:17.437 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:17.434+0000 7f7425ffb700 1 -- 172.21.15.132:0/4078976901 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7420042da0 con 0x7f74300f51a0 2024-02-20T00:12:17.437 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:17.434+0000 7f7434d65700 1 --2- 172.21.15.132:0/4078976901 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f741c02ff10 0x7f741c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:12:17.437 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:17.434+0000 7f7434d65700 1 --2- 172.21.15.132:0/4078976901 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f741c02ff10 0x7f741c0323d0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f742c0096a0 tx=0x7f742c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:12:17.439 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:17.438+0000 7f7425ffb700 1 -- 172.21.15.132:0/4078976901 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7420014790 con 0x7f74300f51a0 2024-02-20T00:12:17.644 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:17.643+0000 7f7436568700 1 -- 172.21.15.132:0/4078976901 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7430106ea0 con 0x7f74300f51a0 2024-02-20T00:12:17.644 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:17.643+0000 7f7425ffb700 1 -- 172.21.15.132:0/4078976901 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f742001f7f0 con 0x7f74300f51a0 2024-02-20T00:12:17.645 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:12:17.645 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:12:17.646 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:17.645+0000 7f7436568700 1 -- 172.21.15.132:0/4078976901 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f741c02ff10 msgr2=0x7f741c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:12:17.646 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:17.645+0000 7f7436568700 1 --2- 172.21.15.132:0/4078976901 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f741c02ff10 0x7f741c0323d0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f742c0096a0 tx=0x7f742c006b40).stop 2024-02-20T00:12:17.646 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:17.645+0000 7f7436568700 1 -- 172.21.15.132:0/4078976901 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f74300f51a0 msgr2=0x7f74301054d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:12:17.646 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:17.645+0000 7f7436568700 1 --2- 172.21.15.132:0/4078976901 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f74300f51a0 0x7f74301054d0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f74200089f0 tx=0x7f742000b040).stop 2024-02-20T00:12:17.647 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:17.645+0000 7f7436568700 1 -- 172.21.15.132:0/4078976901 shutdown_connections 2024-02-20T00:12:17.647 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:17.645+0000 7f7436568700 1 --2- 172.21.15.132:0/4078976901 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f741c02ff10 0x7f741c0323d0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:12:17.647 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:17.645+0000 7f7436568700 1 --2- 172.21.15.132:0/4078976901 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f74300f51a0 0x7f74301054d0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:12:17.647 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:17.645+0000 7f7436568700 1 -- 172.21.15.132:0/4078976901 >> 172.21.15.132:0/4078976901 conn(0x7f74300f09a0 msgr2=0x7f74300f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:12:17.647 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:17.645+0000 7f7436568700 1 -- 172.21.15.132:0/4078976901 shutdown_connections 2024-02-20T00:12:17.647 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:17.645+0000 7f7436568700 1 -- 172.21.15.132:0/4078976901 wait complete. 2024-02-20T00:12:17.647 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:12:18.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:12:17 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:12:16.857023+0000 mgr.smithi084.bmqrnf (mgr.14162) 71 : cluster [DBG] pgmap v64: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:12:18.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:12:17 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:12:17.643754+0000 mon.smithi084 (mon.0) 149 : audit [DBG] from='client.? 172.21.15.132:0/4078976901' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:12:19.362 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:12:19.362 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:12:20.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:12:19 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:12:18.857335+0000 mgr.smithi084.bmqrnf (mgr.14162) 72 : cluster [DBG] pgmap v65: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:12:21.229 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:21.227+0000 7f9269e4c700 1 -- 172.21.15.132:0/937172419 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f92640f5410 msgr2=0x7f92640f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:12:21.230 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:21.227+0000 7f9269e4c700 1 --2- 172.21.15.132:0/937172419 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f92640f5410 0x7f92640f5830 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f9254009fb0 tx=0x7f9254009b30).stop 2024-02-20T00:12:21.230 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:21.227+0000 7f9269e4c700 1 -- 172.21.15.132:0/937172419 shutdown_connections 2024-02-20T00:12:21.230 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:21.227+0000 7f9269e4c700 1 --2- 172.21.15.132:0/937172419 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f92640f5410 0x7f92640f5830 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:12:21.230 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:21.227+0000 7f9269e4c700 1 -- 172.21.15.132:0/937172419 >> 172.21.15.132:0/937172419 conn(0x7f92640f0b70 msgr2=0x7f92640f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:12:21.230 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:21.227+0000 7f9269e4c700 1 -- 172.21.15.132:0/937172419 shutdown_connections 2024-02-20T00:12:21.230 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:21.227+0000 7f9269e4c700 1 -- 172.21.15.132:0/937172419 wait complete. 2024-02-20T00:12:21.230 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:21.228+0000 7f9269e4c700 1 Processor -- start 2024-02-20T00:12:21.230 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:21.228+0000 7f9269e4c700 1 -- start start 2024-02-20T00:12:21.230 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:21.228+0000 7f9269e4c700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f92640f5410 0x7f9264109530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:12:21.230 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:21.228+0000 7f9269e4c700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9264060220 con 0x7f92640f5410 2024-02-20T00:12:21.230 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:21.228+0000 7f92637fe700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f92640f5410 0x7f9264109530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:12:21.231 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:21.228+0000 7f92637fe700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f92640f5410 0x7f9264109530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:57390/0 (socket says 172.21.15.132:57390) 2024-02-20T00:12:21.231 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:21.228+0000 7f92637fe700 1 -- 172.21.15.132:0/2838020824 learned_addr learned my addr 172.21.15.132:0/2838020824 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:12:21.233 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:21.229+0000 7f92637fe700 1 -- 172.21.15.132:0/2838020824 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9254005040 con 0x7f92640f5410 2024-02-20T00:12:21.233 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:21.229+0000 7f92637fe700 1 --2- 172.21.15.132:0/2838020824 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f92640f5410 0x7f9264109530 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f9254005010 tx=0x7f925400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:12:21.233 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:21.229+0000 7f924bfff700 1 -- 172.21.15.132:0/2838020824 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9254009320 con 0x7f92640f5410 2024-02-20T00:12:21.233 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:21.229+0000 7f9269e4c700 1 -- 172.21.15.132:0/2838020824 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9264060420 con 0x7f92640f5410 2024-02-20T00:12:21.233 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:21.229+0000 7f924bfff700 1 -- 172.21.15.132:0/2838020824 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9254013380 con 0x7f92640f5410 2024-02-20T00:12:21.233 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:21.229+0000 7f924bfff700 1 -- 172.21.15.132:0/2838020824 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9254014e50 con 0x7f92640f5410 2024-02-20T00:12:21.233 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:21.229+0000 7f9269e4c700 1 -- 172.21.15.132:0/2838020824 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9264109be0 con 0x7f92640f5410 2024-02-20T00:12:21.233 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:21.230+0000 7f924bfff700 1 -- 172.21.15.132:0/2838020824 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9254014420 con 0x7f92640f5410 2024-02-20T00:12:21.233 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:21.230+0000 7f9269e4c700 1 -- 172.21.15.132:0/2838020824 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9264059100 con 0x7f92640f5410 2024-02-20T00:12:21.233 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:21.230+0000 7f924bfff700 1 --2- 172.21.15.132:0/2838020824 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f924c02ff10 0x7f924c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:12:21.233 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:21.230+0000 7f924bfff700 1 -- 172.21.15.132:0/2838020824 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9254040f40 con 0x7f92640f5410 2024-02-20T00:12:21.233 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:21.230+0000 7f9262ffd700 1 --2- 172.21.15.132:0/2838020824 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f924c02ff10 0x7f924c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:12:21.234 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:21.231+0000 7f9262ffd700 1 --2- 172.21.15.132:0/2838020824 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f924c02ff10 0x7f924c0323d0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f92580096a0 tx=0x7f9258006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:12:21.236 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:21.235+0000 7f924bfff700 1 -- 172.21.15.132:0/2838020824 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f92540095f0 con 0x7f92640f5410 2024-02-20T00:12:21.443 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:21.442+0000 7f9269e4c700 1 -- 172.21.15.132:0/2838020824 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9264045ad0 con 0x7f92640f5410 2024-02-20T00:12:21.444 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:21.442+0000 7f924bfff700 1 -- 172.21.15.132:0/2838020824 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f925401db70 con 0x7f92640f5410 2024-02-20T00:12:21.444 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:12:21.444 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:12:21.445 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:21.444+0000 7f9269e4c700 1 -- 172.21.15.132:0/2838020824 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f924c02ff10 msgr2=0x7f924c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:12:21.446 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:21.444+0000 7f9269e4c700 1 --2- 172.21.15.132:0/2838020824 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f924c02ff10 0x7f924c0323d0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f92580096a0 tx=0x7f9258006b40).stop 2024-02-20T00:12:21.446 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:21.444+0000 7f9269e4c700 1 -- 172.21.15.132:0/2838020824 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f92640f5410 msgr2=0x7f9264109530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:12:21.446 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:21.444+0000 7f9269e4c700 1 --2- 172.21.15.132:0/2838020824 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f92640f5410 0x7f9264109530 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f9254005010 tx=0x7f925400b040).stop 2024-02-20T00:12:21.446 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:21.444+0000 7f9269e4c700 1 -- 172.21.15.132:0/2838020824 shutdown_connections 2024-02-20T00:12:21.446 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:21.444+0000 7f9269e4c700 1 --2- 172.21.15.132:0/2838020824 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f924c02ff10 0x7f924c0323d0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:12:21.446 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:21.444+0000 7f9269e4c700 1 --2- 172.21.15.132:0/2838020824 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f92640f5410 0x7f9264109530 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:12:21.446 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:21.444+0000 7f9269e4c700 1 -- 172.21.15.132:0/2838020824 >> 172.21.15.132:0/2838020824 conn(0x7f92640f0b70 msgr2=0x7f92640f15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:12:21.446 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:21.444+0000 7f9269e4c700 1 -- 172.21.15.132:0/2838020824 shutdown_connections 2024-02-20T00:12:21.446 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:21.444+0000 7f9269e4c700 1 -- 172.21.15.132:0/2838020824 wait complete. 2024-02-20T00:12:21.446 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:12:22.107 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:12:21 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:12:20.857717+0000 mgr.smithi084.bmqrnf (mgr.14162) 73 : cluster [DBG] pgmap v66: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:12:22.108 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:12:21 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:12:21.443024+0000 mon.smithi084 (mon.0) 150 : audit [DBG] from='client.? 172.21.15.132:0/2838020824' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:12:23.027 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:12:23.027 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:12:24.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:12:23 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:12:22.857995+0000 mgr.smithi084.bmqrnf (mgr.14162) 74 : cluster [DBG] pgmap v67: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:12:24.785 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:24.781+0000 7efe8810b700 1 -- 172.21.15.132:0/3411547599 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7efe80060220 msgr2=0x7efe800f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:12:24.785 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:24.781+0000 7efe8810b700 1 --2- 172.21.15.132:0/3411547599 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7efe80060220 0x7efe800f9150 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7efe70009fb0 tx=0x7efe70009b30).stop 2024-02-20T00:12:24.785 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:24.781+0000 7efe8810b700 1 -- 172.21.15.132:0/3411547599 shutdown_connections 2024-02-20T00:12:24.786 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:24.781+0000 7efe8810b700 1 --2- 172.21.15.132:0/3411547599 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7efe80060220 0x7efe800f9150 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:12:24.786 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:24.781+0000 7efe8810b700 1 -- 172.21.15.132:0/3411547599 >> 172.21.15.132:0/3411547599 conn(0x7efe800f0b70 msgr2=0x7efe800f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:12:24.786 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:24.781+0000 7efe8810b700 1 -- 172.21.15.132:0/3411547599 shutdown_connections 2024-02-20T00:12:24.786 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:24.781+0000 7efe8810b700 1 -- 172.21.15.132:0/3411547599 wait complete. 2024-02-20T00:12:24.786 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:24.782+0000 7efe8810b700 1 Processor -- start 2024-02-20T00:12:24.786 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:24.782+0000 7efe8810b700 1 -- start start 2024-02-20T00:12:24.786 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:24.782+0000 7efe8810b700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7efe80060220 0x7efe801067a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:12:24.786 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:24.782+0000 7efe8810b700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe80106c90 con 0x7efe80060220 2024-02-20T00:12:24.786 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:24.782+0000 7efe85ea7700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7efe80060220 0x7efe801067a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:12:24.786 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:24.782+0000 7efe85ea7700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7efe80060220 0x7efe801067a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:57404/0 (socket says 172.21.15.132:57404) 2024-02-20T00:12:24.786 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:24.782+0000 7efe85ea7700 1 -- 172.21.15.132:0/986277079 learned_addr learned my addr 172.21.15.132:0/986277079 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:12:24.787 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:24.782+0000 7efe85ea7700 1 -- 172.21.15.132:0/986277079 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe70005040 con 0x7efe80060220 2024-02-20T00:12:24.787 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:24.782+0000 7efe85ea7700 1 --2- 172.21.15.132:0/986277079 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7efe80060220 0x7efe801067a0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7efe700089f0 tx=0x7efe7000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:12:24.787 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:24.783+0000 7efe767fc700 1 -- 172.21.15.132:0/986277079 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efe70009320 con 0x7efe80060220 2024-02-20T00:12:24.787 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:24.783+0000 7efe8810b700 1 -- 172.21.15.132:0/986277079 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe80106e90 con 0x7efe80060220 2024-02-20T00:12:24.787 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:24.783+0000 7efe767fc700 1 -- 172.21.15.132:0/986277079 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efe70013380 con 0x7efe80060220 2024-02-20T00:12:24.787 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:24.783+0000 7efe767fc700 1 -- 172.21.15.132:0/986277079 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efe70014e20 con 0x7efe80060220 2024-02-20T00:12:24.787 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:24.783+0000 7efe8810b700 1 -- 172.21.15.132:0/986277079 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe80107320 con 0x7efe80060220 2024-02-20T00:12:24.788 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:24.783+0000 7efe767fc700 1 -- 172.21.15.132:0/986277079 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7efe700095f0 con 0x7efe80060220 2024-02-20T00:12:24.788 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:24.783+0000 7efe8810b700 1 -- 172.21.15.132:0/986277079 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe80045ad0 con 0x7efe80060220 2024-02-20T00:12:24.788 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:24.784+0000 7efe767fc700 1 --2- 172.21.15.132:0/986277079 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7efe6c02ff10 0x7efe6c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:12:24.788 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:24.784+0000 7efe767fc700 1 -- 172.21.15.132:0/986277079 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7efe70040fb0 con 0x7efe80060220 2024-02-20T00:12:24.788 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:24.784+0000 7efe856a6700 1 --2- 172.21.15.132:0/986277079 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7efe6c02ff10 0x7efe6c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:12:24.788 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:24.784+0000 7efe856a6700 1 --2- 172.21.15.132:0/986277079 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7efe6c02ff10 0x7efe6c0323d0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7efe7c0096a0 tx=0x7efe7c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:12:24.790 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:24.788+0000 7efe767fc700 1 -- 172.21.15.132:0/986277079 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7efe7006d2b0 con 0x7efe80060220 2024-02-20T00:12:24.996 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:24.993+0000 7efe8810b700 1 -- 172.21.15.132:0/986277079 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7efe80108170 con 0x7efe80060220 2024-02-20T00:12:24.997 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:24.994+0000 7efe767fc700 1 -- 172.21.15.132:0/986277079 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7efe70014420 con 0x7efe80060220 2024-02-20T00:12:24.997 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:12:24.997 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:12:24.999 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:24.996+0000 7efe8810b700 1 -- 172.21.15.132:0/986277079 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7efe6c02ff10 msgr2=0x7efe6c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:12:24.999 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:24.996+0000 7efe8810b700 1 --2- 172.21.15.132:0/986277079 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7efe6c02ff10 0x7efe6c0323d0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7efe7c0096a0 tx=0x7efe7c006b40).stop 2024-02-20T00:12:24.999 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:24.996+0000 7efe8810b700 1 -- 172.21.15.132:0/986277079 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7efe80060220 msgr2=0x7efe801067a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:12:24.999 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:24.996+0000 7efe8810b700 1 --2- 172.21.15.132:0/986277079 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7efe80060220 0x7efe801067a0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7efe700089f0 tx=0x7efe7000b040).stop 2024-02-20T00:12:24.999 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:24.996+0000 7efe8810b700 1 -- 172.21.15.132:0/986277079 shutdown_connections 2024-02-20T00:12:24.999 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:24.996+0000 7efe8810b700 1 --2- 172.21.15.132:0/986277079 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7efe6c02ff10 0x7efe6c0323d0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:12:24.999 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:24.996+0000 7efe8810b700 1 --2- 172.21.15.132:0/986277079 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7efe80060220 0x7efe801067a0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:12:24.999 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:24.996+0000 7efe8810b700 1 -- 172.21.15.132:0/986277079 >> 172.21.15.132:0/986277079 conn(0x7efe800f0b70 msgr2=0x7efe800f15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:12:24.999 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:24.996+0000 7efe8810b700 1 -- 172.21.15.132:0/986277079 shutdown_connections 2024-02-20T00:12:24.999 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:24.996+0000 7efe8810b700 1 -- 172.21.15.132:0/986277079 wait complete. 2024-02-20T00:12:25.000 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:12:26.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:12:25 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:12:24.858399+0000 mgr.smithi084.bmqrnf (mgr.14162) 75 : cluster [DBG] pgmap v68: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:12:26.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:12:25 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:12:24.995789+0000 mon.smithi084 (mon.0) 151 : audit [DBG] from='client.? 172.21.15.132:0/986277079' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:12:26.627 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:12:26.627 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:12:28.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:12:27 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:12:26.858704+0000 mgr.smithi084.bmqrnf (mgr.14162) 76 : cluster [DBG] pgmap v69: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:12:28.416 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:28.412+0000 7f9ef7157700 1 -- 172.21.15.132:0/684975910 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9ef00f5410 msgr2=0x7f9ef00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:12:28.417 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:28.412+0000 7f9ef7157700 1 --2- 172.21.15.132:0/684975910 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9ef00f5410 0x7f9ef00f5830 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7f9ee4009fb0 tx=0x7f9ee4009b30).stop 2024-02-20T00:12:28.417 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:28.412+0000 7f9ef7157700 1 -- 172.21.15.132:0/684975910 shutdown_connections 2024-02-20T00:12:28.417 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:28.412+0000 7f9ef7157700 1 --2- 172.21.15.132:0/684975910 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9ef00f5410 0x7f9ef00f5830 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:12:28.417 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:28.412+0000 7f9ef7157700 1 -- 172.21.15.132:0/684975910 >> 172.21.15.132:0/684975910 conn(0x7f9ef00f0b70 msgr2=0x7f9ef00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:12:28.417 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:28.412+0000 7f9ef7157700 1 -- 172.21.15.132:0/684975910 shutdown_connections 2024-02-20T00:12:28.417 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:28.412+0000 7f9ef7157700 1 -- 172.21.15.132:0/684975910 wait complete. 2024-02-20T00:12:28.417 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:28.413+0000 7f9ef7157700 1 Processor -- start 2024-02-20T00:12:28.417 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:28.413+0000 7f9ef7157700 1 -- start start 2024-02-20T00:12:28.417 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:28.413+0000 7f9ef7157700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9ef00f5410 0x7f9ef01094e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:12:28.417 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:28.413+0000 7f9ef7157700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ef0060280 con 0x7f9ef00f5410 2024-02-20T00:12:28.417 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:28.413+0000 7f9ef4ef3700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9ef00f5410 0x7f9ef01094e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:12:28.418 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:28.413+0000 7f9ef4ef3700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9ef00f5410 0x7f9ef01094e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:39426/0 (socket says 172.21.15.132:39426) 2024-02-20T00:12:28.418 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:28.413+0000 7f9ef4ef3700 1 -- 172.21.15.132:0/1417715963 learned_addr learned my addr 172.21.15.132:0/1417715963 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:12:28.418 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:28.414+0000 7f9ef4ef3700 1 -- 172.21.15.132:0/1417715963 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ee4005040 con 0x7f9ef00f5410 2024-02-20T00:12:28.418 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:28.414+0000 7f9ef4ef3700 1 --2- 172.21.15.132:0/1417715963 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9ef00f5410 0x7f9ef01094e0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f9ee4005010 tx=0x7f9ee400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:12:28.418 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:28.414+0000 7f9eed7fa700 1 -- 172.21.15.132:0/1417715963 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9ee4009320 con 0x7f9ef00f5410 2024-02-20T00:12:28.418 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:28.414+0000 7f9ef7157700 1 -- 172.21.15.132:0/1417715963 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ef0060480 con 0x7f9ef00f5410 2024-02-20T00:12:28.418 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:28.414+0000 7f9eed7fa700 1 -- 172.21.15.132:0/1417715963 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9ee4013380 con 0x7f9ef00f5410 2024-02-20T00:12:28.418 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:28.414+0000 7f9eed7fa700 1 -- 172.21.15.132:0/1417715963 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9ee4014e50 con 0x7f9ef00f5410 2024-02-20T00:12:28.418 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:28.414+0000 7f9ef7157700 1 -- 172.21.15.132:0/1417715963 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ef0109c10 con 0x7f9ef00f5410 2024-02-20T00:12:28.419 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:28.415+0000 7f9eed7fa700 1 -- 172.21.15.132:0/1417715963 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9ee40263a0 con 0x7f9ef00f5410 2024-02-20T00:12:28.419 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:28.415+0000 7f9eed7fa700 1 --2- 172.21.15.132:0/1417715963 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f9ee002ff10 0x7f9ee00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:12:28.419 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:28.415+0000 7f9eed7fa700 1 -- 172.21.15.132:0/1417715963 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9ee4040ef0 con 0x7f9ef00f5410 2024-02-20T00:12:28.419 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:28.415+0000 7f9ef7157700 1 -- 172.21.15.132:0/1417715963 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ef0045ad0 con 0x7f9ef00f5410 2024-02-20T00:12:28.419 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:28.415+0000 7f9eeffff700 1 --2- 172.21.15.132:0/1417715963 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f9ee002ff10 0x7f9ee00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:12:28.419 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:28.416+0000 7f9eeffff700 1 --2- 172.21.15.132:0/1417715963 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f9ee002ff10 0x7f9ee00323d0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f9edc0096a0 tx=0x7f9edc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:12:28.422 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:28.420+0000 7f9eed7fa700 1 -- 172.21.15.132:0/1417715963 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9ee4012920 con 0x7f9ef00f5410 2024-02-20T00:12:28.626 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:28.623+0000 7f9ef7157700 1 -- 172.21.15.132:0/1417715963 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9ef00fb090 con 0x7f9ef00f5410 2024-02-20T00:12:28.626 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:28.624+0000 7f9eed7fa700 1 -- 172.21.15.132:0/1417715963 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9ee402b020 con 0x7f9ef00f5410 2024-02-20T00:12:28.627 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:12:28.627 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:12:28.628 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:28.626+0000 7f9ef7157700 1 -- 172.21.15.132:0/1417715963 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f9ee002ff10 msgr2=0x7f9ee00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:12:28.628 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:28.626+0000 7f9ef7157700 1 --2- 172.21.15.132:0/1417715963 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f9ee002ff10 0x7f9ee00323d0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f9edc0096a0 tx=0x7f9edc006b40).stop 2024-02-20T00:12:28.629 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:28.626+0000 7f9ef7157700 1 -- 172.21.15.132:0/1417715963 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9ef00f5410 msgr2=0x7f9ef01094e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:12:28.629 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:28.626+0000 7f9ef7157700 1 --2- 172.21.15.132:0/1417715963 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9ef00f5410 0x7f9ef01094e0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f9ee4005010 tx=0x7f9ee400b040).stop 2024-02-20T00:12:28.629 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:28.626+0000 7f9ef7157700 1 -- 172.21.15.132:0/1417715963 shutdown_connections 2024-02-20T00:12:28.629 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:28.626+0000 7f9ef7157700 1 --2- 172.21.15.132:0/1417715963 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f9ee002ff10 0x7f9ee00323d0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:12:28.629 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:28.626+0000 7f9ef7157700 1 --2- 172.21.15.132:0/1417715963 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9ef00f5410 0x7f9ef01094e0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:12:28.629 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:28.626+0000 7f9ef7157700 1 -- 172.21.15.132:0/1417715963 >> 172.21.15.132:0/1417715963 conn(0x7f9ef00f0b70 msgr2=0x7f9ef00f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:12:28.629 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:28.626+0000 7f9ef7157700 1 -- 172.21.15.132:0/1417715963 shutdown_connections 2024-02-20T00:12:28.629 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:28.626+0000 7f9ef7157700 1 -- 172.21.15.132:0/1417715963 wait complete. 2024-02-20T00:12:28.630 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:12:29.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:12:28 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:12:28.625804+0000 mon.smithi084 (mon.0) 152 : audit [DBG] from='client.? 172.21.15.132:0/1417715963' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:12:30.250 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:12:30.251 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:12:30.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:12:29 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:12:28.858959+0000 mgr.smithi084.bmqrnf (mgr.14162) 77 : cluster [DBG] pgmap v70: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:12:31.968 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:31.965+0000 7f4bec952700 1 -- 172.21.15.132:0/593700973 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4be80f62a0 msgr2=0x7f4be80f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:12:31.968 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:31.965+0000 7f4bec952700 1 --2- 172.21.15.132:0/593700973 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4be80f62a0 0x7f4be80f66c0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f4bdc009fb0 tx=0x7f4bdc009b30).stop 2024-02-20T00:12:31.968 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:31.965+0000 7f4bec952700 1 -- 172.21.15.132:0/593700973 shutdown_connections 2024-02-20T00:12:31.968 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:31.965+0000 7f4bec952700 1 --2- 172.21.15.132:0/593700973 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4be80f62a0 0x7f4be80f66c0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:12:31.968 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:31.965+0000 7f4bec952700 1 -- 172.21.15.132:0/593700973 >> 172.21.15.132:0/593700973 conn(0x7f4be80f1aa0 msgr2=0x7f4be80f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:12:31.968 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:31.965+0000 7f4bec952700 1 -- 172.21.15.132:0/593700973 shutdown_connections 2024-02-20T00:12:31.968 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:31.965+0000 7f4bec952700 1 -- 172.21.15.132:0/593700973 wait complete. 2024-02-20T00:12:31.968 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:31.965+0000 7f4bec952700 1 Processor -- start 2024-02-20T00:12:31.968 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:31.966+0000 7f4bec952700 1 -- start start 2024-02-20T00:12:31.968 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:31.966+0000 7f4bec952700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4be80f62a0 0x7f4be81065d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:12:31.969 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:31.966+0000 7f4bec952700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4be8106ac0 con 0x7f4be80f62a0 2024-02-20T00:12:31.970 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:31.966+0000 7f4be77fe700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4be80f62a0 0x7f4be81065d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:12:31.970 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:31.966+0000 7f4be77fe700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4be80f62a0 0x7f4be81065d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:39438/0 (socket says 172.21.15.132:39438) 2024-02-20T00:12:31.970 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:31.966+0000 7f4be77fe700 1 -- 172.21.15.132:0/1507388498 learned_addr learned my addr 172.21.15.132:0/1507388498 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:12:31.970 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:31.966+0000 7f4be77fe700 1 -- 172.21.15.132:0/1507388498 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bdc005040 con 0x7f4be80f62a0 2024-02-20T00:12:31.970 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:31.967+0000 7f4be77fe700 1 --2- 172.21.15.132:0/1507388498 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4be80f62a0 0x7f4be81065d0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f4bdc0089f0 tx=0x7f4bdc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:12:31.970 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:31.967+0000 7f4bcffff700 1 -- 172.21.15.132:0/1507388498 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4bdc009320 con 0x7f4be80f62a0 2024-02-20T00:12:31.970 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:31.967+0000 7f4bec952700 1 -- 172.21.15.132:0/1507388498 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4be8106cc0 con 0x7f4be80f62a0 2024-02-20T00:12:31.970 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:31.967+0000 7f4bcffff700 1 -- 172.21.15.132:0/1507388498 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4bdc013380 con 0x7f4be80f62a0 2024-02-20T00:12:31.971 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:31.967+0000 7f4bcffff700 1 -- 172.21.15.132:0/1507388498 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4bdc014e20 con 0x7f4be80f62a0 2024-02-20T00:12:31.971 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:31.967+0000 7f4bec952700 1 -- 172.21.15.132:0/1507388498 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4be8107150 con 0x7f4be80f62a0 2024-02-20T00:12:31.971 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:31.968+0000 7f4bcffff700 1 -- 172.21.15.132:0/1507388498 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4bdc0095f0 con 0x7f4be80f62a0 2024-02-20T00:12:31.971 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:31.968+0000 7f4bec952700 1 -- 172.21.15.132:0/1507388498 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4be8045ad0 con 0x7f4be80f62a0 2024-02-20T00:12:31.971 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:31.968+0000 7f4bcffff700 1 --2- 172.21.15.132:0/1507388498 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f4bd002ff10 0x7f4bd00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:12:31.971 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:31.968+0000 7f4bcffff700 1 -- 172.21.15.132:0/1507388498 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4bdc040e90 con 0x7f4be80f62a0 2024-02-20T00:12:31.971 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:31.968+0000 7f4be6ffd700 1 --2- 172.21.15.132:0/1507388498 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f4bd002ff10 0x7f4bd00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:12:31.971 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:31.968+0000 7f4be6ffd700 1 --2- 172.21.15.132:0/1507388498 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f4bd002ff10 0x7f4bd00323d0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f4bd80096a0 tx=0x7f4bd8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:12:31.975 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:31.973+0000 7f4bcffff700 1 -- 172.21.15.132:0/1507388498 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4bdc014420 con 0x7f4be80f62a0 2024-02-20T00:12:32.181 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:32.178+0000 7f4bec952700 1 -- 172.21.15.132:0/1507388498 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4be8108000 con 0x7f4be80f62a0 2024-02-20T00:12:32.181 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:32.179+0000 7f4bcffff700 1 -- 172.21.15.132:0/1507388498 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4bdc01d980 con 0x7f4be80f62a0 2024-02-20T00:12:32.181 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:12:32.182 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:12:32.183 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:32.181+0000 7f4bec952700 1 -- 172.21.15.132:0/1507388498 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f4bd002ff10 msgr2=0x7f4bd00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:12:32.183 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:32.181+0000 7f4bec952700 1 --2- 172.21.15.132:0/1507388498 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f4bd002ff10 0x7f4bd00323d0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f4bd80096a0 tx=0x7f4bd8006b40).stop 2024-02-20T00:12:32.183 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:32.181+0000 7f4bec952700 1 -- 172.21.15.132:0/1507388498 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4be80f62a0 msgr2=0x7f4be81065d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:12:32.184 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:32.181+0000 7f4bec952700 1 --2- 172.21.15.132:0/1507388498 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4be80f62a0 0x7f4be81065d0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f4bdc0089f0 tx=0x7f4bdc00b040).stop 2024-02-20T00:12:32.184 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:32.181+0000 7f4bec952700 1 -- 172.21.15.132:0/1507388498 shutdown_connections 2024-02-20T00:12:32.184 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:32.181+0000 7f4bec952700 1 --2- 172.21.15.132:0/1507388498 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f4bd002ff10 0x7f4bd00323d0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:12:32.184 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:32.181+0000 7f4bec952700 1 --2- 172.21.15.132:0/1507388498 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4be80f62a0 0x7f4be81065d0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:12:32.184 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:32.181+0000 7f4bec952700 1 -- 172.21.15.132:0/1507388498 >> 172.21.15.132:0/1507388498 conn(0x7f4be80f1aa0 msgr2=0x7f4be80f2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:12:32.184 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:32.181+0000 7f4bec952700 1 -- 172.21.15.132:0/1507388498 shutdown_connections 2024-02-20T00:12:32.184 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:32.181+0000 7f4bec952700 1 -- 172.21.15.132:0/1507388498 wait complete. 2024-02-20T00:12:32.184 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:12:32.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:12:31 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:12:30.859264+0000 mgr.smithi084.bmqrnf (mgr.14162) 78 : cluster [DBG] pgmap v71: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:12:33.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:12:32 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:12:32.180746+0000 mon.smithi084 (mon.0) 153 : audit [DBG] from='client.? 172.21.15.132:0/1507388498' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:12:33.712 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:12:33.713 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:12:34.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:12:33 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:12:32.859484+0000 mgr.smithi084.bmqrnf (mgr.14162) 79 : cluster [DBG] pgmap v72: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:12:35.591 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:35.588+0000 7fc535f5c700 1 -- 172.21.15.132:0/453531958 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc530060220 msgr2=0x7fc5300f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:12:35.591 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:35.588+0000 7fc535f5c700 1 --2- 172.21.15.132:0/453531958 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc530060220 0x7fc5300f9150 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7fc518009fb0 tx=0x7fc518009b30).stop 2024-02-20T00:12:35.591 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:35.588+0000 7fc535f5c700 1 -- 172.21.15.132:0/453531958 shutdown_connections 2024-02-20T00:12:35.591 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:35.588+0000 7fc535f5c700 1 --2- 172.21.15.132:0/453531958 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc530060220 0x7fc5300f9150 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:12:35.591 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:35.588+0000 7fc535f5c700 1 -- 172.21.15.132:0/453531958 >> 172.21.15.132:0/453531958 conn(0x7fc5300f0b70 msgr2=0x7fc5300f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:12:35.591 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:35.588+0000 7fc535f5c700 1 -- 172.21.15.132:0/453531958 shutdown_connections 2024-02-20T00:12:35.592 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:35.588+0000 7fc535f5c700 1 -- 172.21.15.132:0/453531958 wait complete. 2024-02-20T00:12:35.592 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:35.589+0000 7fc535f5c700 1 Processor -- start 2024-02-20T00:12:35.592 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:35.589+0000 7fc535f5c700 1 -- start start 2024-02-20T00:12:35.593 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:35.589+0000 7fc535f5c700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc530060220 0x7fc5301067a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:12:35.594 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:35.589+0000 7fc535f5c700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc530106c90 con 0x7fc530060220 2024-02-20T00:12:35.594 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:35.590+0000 7fc52f7fe700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc530060220 0x7fc5301067a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:12:35.594 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:35.590+0000 7fc52f7fe700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc530060220 0x7fc5301067a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:39456/0 (socket says 172.21.15.132:39456) 2024-02-20T00:12:35.594 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:35.590+0000 7fc52f7fe700 1 -- 172.21.15.132:0/1442807873 learned_addr learned my addr 172.21.15.132:0/1442807873 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:12:35.594 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:35.590+0000 7fc52f7fe700 1 -- 172.21.15.132:0/1442807873 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc518005040 con 0x7fc530060220 2024-02-20T00:12:35.594 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:35.590+0000 7fc52f7fe700 1 --2- 172.21.15.132:0/1442807873 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc530060220 0x7fc5301067a0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7fc5180089f0 tx=0x7fc51800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:12:35.594 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:35.590+0000 7fc527fff700 1 -- 172.21.15.132:0/1442807873 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc518009320 con 0x7fc530060220 2024-02-20T00:12:35.594 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:35.590+0000 7fc535f5c700 1 -- 172.21.15.132:0/1442807873 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc530106e90 con 0x7fc530060220 2024-02-20T00:12:35.594 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:35.590+0000 7fc535f5c700 1 -- 172.21.15.132:0/1442807873 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc530107320 con 0x7fc530060220 2024-02-20T00:12:35.594 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:35.590+0000 7fc527fff700 1 -- 172.21.15.132:0/1442807873 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc518013380 con 0x7fc530060220 2024-02-20T00:12:35.594 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:35.590+0000 7fc527fff700 1 -- 172.21.15.132:0/1442807873 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc518014e40 con 0x7fc530060220 2024-02-20T00:12:35.594 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:35.591+0000 7fc527fff700 1 -- 172.21.15.132:0/1442807873 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc5180263a0 con 0x7fc530060220 2024-02-20T00:12:35.595 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:35.591+0000 7fc527fff700 1 --2- 172.21.15.132:0/1442807873 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fc51c02ff10 0x7fc51c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:12:35.595 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:35.591+0000 7fc527fff700 1 -- 172.21.15.132:0/1442807873 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc518040ec0 con 0x7fc530060220 2024-02-20T00:12:35.595 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:35.591+0000 7fc535f5c700 1 -- 172.21.15.132:0/1442807873 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc530045ad0 con 0x7fc530060220 2024-02-20T00:12:35.595 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:35.591+0000 7fc52effd700 1 --2- 172.21.15.132:0/1442807873 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fc51c02ff10 0x7fc51c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:12:35.595 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:35.592+0000 7fc52effd700 1 --2- 172.21.15.132:0/1442807873 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fc51c02ff10 0x7fc51c0323d0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7fc5200096a0 tx=0x7fc520006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:12:35.598 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:35.596+0000 7fc527fff700 1 -- 172.21.15.132:0/1442807873 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc518012750 con 0x7fc530060220 2024-02-20T00:12:35.801 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:35.799+0000 7fc535f5c700 1 -- 172.21.15.132:0/1442807873 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc530108040 con 0x7fc530060220 2024-02-20T00:12:35.802 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:35.799+0000 7fc527fff700 1 -- 172.21.15.132:0/1442807873 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc51802b020 con 0x7fc530060220 2024-02-20T00:12:35.802 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:12:35.802 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:12:35.803 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:35.801+0000 7fc535f5c700 1 -- 172.21.15.132:0/1442807873 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fc51c02ff10 msgr2=0x7fc51c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:12:35.804 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:35.801+0000 7fc535f5c700 1 --2- 172.21.15.132:0/1442807873 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fc51c02ff10 0x7fc51c0323d0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7fc5200096a0 tx=0x7fc520006b40).stop 2024-02-20T00:12:35.804 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:35.801+0000 7fc535f5c700 1 -- 172.21.15.132:0/1442807873 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc530060220 msgr2=0x7fc5301067a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:12:35.804 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:35.801+0000 7fc535f5c700 1 --2- 172.21.15.132:0/1442807873 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc530060220 0x7fc5301067a0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7fc5180089f0 tx=0x7fc51800b040).stop 2024-02-20T00:12:35.804 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:35.801+0000 7fc535f5c700 1 -- 172.21.15.132:0/1442807873 shutdown_connections 2024-02-20T00:12:35.804 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:35.801+0000 7fc535f5c700 1 --2- 172.21.15.132:0/1442807873 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fc51c02ff10 0x7fc51c0323d0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:12:35.804 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:35.801+0000 7fc535f5c700 1 --2- 172.21.15.132:0/1442807873 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc530060220 0x7fc5301067a0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:12:35.804 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:35.801+0000 7fc535f5c700 1 -- 172.21.15.132:0/1442807873 >> 172.21.15.132:0/1442807873 conn(0x7fc5300f0b70 msgr2=0x7fc5300f1600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:12:35.804 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:35.801+0000 7fc535f5c700 1 -- 172.21.15.132:0/1442807873 shutdown_connections 2024-02-20T00:12:35.804 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:35.801+0000 7fc535f5c700 1 -- 172.21.15.132:0/1442807873 wait complete. 2024-02-20T00:12:35.804 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:12:36.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:12:35 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:12:34.859747+0000 mgr.smithi084.bmqrnf (mgr.14162) 80 : cluster [DBG] pgmap v73: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:12:36.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:12:35 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:12:35.801129+0000 mon.smithi084 (mon.0) 154 : audit [DBG] from='client.? 172.21.15.132:0/1442807873' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:12:37.563 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:12:37.563 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:12:38.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:12:37 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:12:36.860103+0000 mgr.smithi084.bmqrnf (mgr.14162) 81 : cluster [DBG] pgmap v74: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:12:39.311 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:39.307+0000 7fdd2c3ae700 1 -- 172.21.15.132:0/1745291800 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdd240f5ce0 msgr2=0x7fdd240f80d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:12:39.311 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:39.307+0000 7fdd2c3ae700 1 --2- 172.21.15.132:0/1745291800 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdd240f5ce0 0x7fdd240f80d0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7fdd14009fb0 tx=0x7fdd14009b30).stop 2024-02-20T00:12:39.311 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:39.307+0000 7fdd2c3ae700 1 -- 172.21.15.132:0/1745291800 shutdown_connections 2024-02-20T00:12:39.312 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:39.307+0000 7fdd2c3ae700 1 --2- 172.21.15.132:0/1745291800 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdd240f5ce0 0x7fdd240f80d0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:12:39.312 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:39.307+0000 7fdd2c3ae700 1 -- 172.21.15.132:0/1745291800 >> 172.21.15.132:0/1745291800 conn(0x7fdd240efe40 msgr2=0x7fdd240f2260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:12:39.312 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:39.307+0000 7fdd2c3ae700 1 -- 172.21.15.132:0/1745291800 shutdown_connections 2024-02-20T00:12:39.312 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:39.307+0000 7fdd2c3ae700 1 -- 172.21.15.132:0/1745291800 wait complete. 2024-02-20T00:12:39.312 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:39.308+0000 7fdd2c3ae700 1 Processor -- start 2024-02-20T00:12:39.312 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:39.308+0000 7fdd2c3ae700 1 -- start start 2024-02-20T00:12:39.312 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:39.308+0000 7fdd2c3ae700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdd240f5ce0 0x7fdd24105690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:12:39.312 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:39.308+0000 7fdd2c3ae700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd24105b80 con 0x7fdd240f5ce0 2024-02-20T00:12:39.312 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:39.308+0000 7fdd2a14a700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdd240f5ce0 0x7fdd24105690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:12:39.312 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:39.308+0000 7fdd2a14a700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdd240f5ce0 0x7fdd24105690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:45302/0 (socket says 172.21.15.132:45302) 2024-02-20T00:12:39.312 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:39.308+0000 7fdd2a14a700 1 -- 172.21.15.132:0/2903696071 learned_addr learned my addr 172.21.15.132:0/2903696071 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:12:39.312 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:39.309+0000 7fdd2a14a700 1 -- 172.21.15.132:0/2903696071 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd14005040 con 0x7fdd240f5ce0 2024-02-20T00:12:39.312 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:39.309+0000 7fdd2a14a700 1 --2- 172.21.15.132:0/2903696071 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdd240f5ce0 0x7fdd24105690 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7fdd14005010 tx=0x7fdd1400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:12:39.313 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:39.309+0000 7fdd1affd700 1 -- 172.21.15.132:0/2903696071 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdd14009320 con 0x7fdd240f5ce0 2024-02-20T00:12:39.314 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:39.309+0000 7fdd2c3ae700 1 -- 172.21.15.132:0/2903696071 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd24105d80 con 0x7fdd240f5ce0 2024-02-20T00:12:39.314 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:39.309+0000 7fdd1affd700 1 -- 172.21.15.132:0/2903696071 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdd14013380 con 0x7fdd240f5ce0 2024-02-20T00:12:39.314 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:39.309+0000 7fdd1affd700 1 -- 172.21.15.132:0/2903696071 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdd14014e50 con 0x7fdd240f5ce0 2024-02-20T00:12:39.314 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:39.309+0000 7fdd2c3ae700 1 -- 172.21.15.132:0/2903696071 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd24106210 con 0x7fdd240f5ce0 2024-02-20T00:12:39.314 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:39.310+0000 7fdd1affd700 1 -- 172.21.15.132:0/2903696071 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdd14014420 con 0x7fdd240f5ce0 2024-02-20T00:12:39.314 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:39.310+0000 7fdd2c3ae700 1 -- 172.21.15.132:0/2903696071 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd24045ad0 con 0x7fdd240f5ce0 2024-02-20T00:12:39.314 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:39.310+0000 7fdd1affd700 1 --2- 172.21.15.132:0/2903696071 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fdd1002ff10 0x7fdd100323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:12:39.314 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:39.310+0000 7fdd1affd700 1 -- 172.21.15.132:0/2903696071 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdd14040e30 con 0x7fdd240f5ce0 2024-02-20T00:12:39.314 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:39.310+0000 7fdd29949700 1 --2- 172.21.15.132:0/2903696071 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fdd1002ff10 0x7fdd100323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:12:39.314 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:39.311+0000 7fdd29949700 1 --2- 172.21.15.132:0/2903696071 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fdd1002ff10 0x7fdd100323d0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7fdd200096a0 tx=0x7fdd20006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:12:39.317 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:39.315+0000 7fdd1affd700 1 -- 172.21.15.132:0/2903696071 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdd140095f0 con 0x7fdd240f5ce0 2024-02-20T00:12:39.520 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:39.518+0000 7fdd2c3ae700 1 -- 172.21.15.132:0/2903696071 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdd241070c0 con 0x7fdd240f5ce0 2024-02-20T00:12:39.520 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:39.518+0000 7fdd1affd700 1 -- 172.21.15.132:0/2903696071 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fdd1401db70 con 0x7fdd240f5ce0 2024-02-20T00:12:39.521 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:12:39.521 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:12:39.522 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:39.520+0000 7fdd2c3ae700 1 -- 172.21.15.132:0/2903696071 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fdd1002ff10 msgr2=0x7fdd100323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:12:39.522 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:39.520+0000 7fdd2c3ae700 1 --2- 172.21.15.132:0/2903696071 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fdd1002ff10 0x7fdd100323d0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7fdd200096a0 tx=0x7fdd20006b40).stop 2024-02-20T00:12:39.522 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:39.520+0000 7fdd2c3ae700 1 -- 172.21.15.132:0/2903696071 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdd240f5ce0 msgr2=0x7fdd24105690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:12:39.522 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:39.520+0000 7fdd2c3ae700 1 --2- 172.21.15.132:0/2903696071 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdd240f5ce0 0x7fdd24105690 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7fdd14005010 tx=0x7fdd1400b040).stop 2024-02-20T00:12:39.522 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:39.520+0000 7fdd2c3ae700 1 -- 172.21.15.132:0/2903696071 shutdown_connections 2024-02-20T00:12:39.522 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:39.520+0000 7fdd2c3ae700 1 --2- 172.21.15.132:0/2903696071 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fdd1002ff10 0x7fdd100323d0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:12:39.523 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:39.520+0000 7fdd2c3ae700 1 --2- 172.21.15.132:0/2903696071 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdd240f5ce0 0x7fdd24105690 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:12:39.523 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:39.520+0000 7fdd2c3ae700 1 -- 172.21.15.132:0/2903696071 >> 172.21.15.132:0/2903696071 conn(0x7fdd240efe40 msgr2=0x7fdd240f08a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:12:39.523 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:39.520+0000 7fdd2c3ae700 1 -- 172.21.15.132:0/2903696071 shutdown_connections 2024-02-20T00:12:39.523 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:39.520+0000 7fdd2c3ae700 1 -- 172.21.15.132:0/2903696071 wait complete. 2024-02-20T00:12:39.523 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:12:40.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:12:39 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:12:38.860467+0000 mgr.smithi084.bmqrnf (mgr.14162) 82 : cluster [DBG] pgmap v75: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:12:40.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:12:39 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:12:39.519906+0000 mon.smithi084 (mon.0) 155 : audit [DBG] from='client.? 172.21.15.132:0/2903696071' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:12:41.148 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:12:41.148 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:12:42.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:12:41 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:12:40.860901+0000 mgr.smithi084.bmqrnf (mgr.14162) 83 : cluster [DBG] pgmap v76: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:12:42.834 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:42.831+0000 7fcccd82d700 1 -- 172.21.15.132:0/858060420 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fccc80f51a0 msgr2=0x7fccc80f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:12:42.835 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:42.831+0000 7fcccd82d700 1 --2- 172.21.15.132:0/858060420 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fccc80f51a0 0x7fccc80f55c0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7fccbc009fb0 tx=0x7fccbc009b30).stop 2024-02-20T00:12:42.835 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:42.832+0000 7fcccd82d700 1 -- 172.21.15.132:0/858060420 shutdown_connections 2024-02-20T00:12:42.835 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:42.832+0000 7fcccd82d700 1 --2- 172.21.15.132:0/858060420 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fccc80f51a0 0x7fccc80f55c0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:12:42.835 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:42.832+0000 7fcccd82d700 1 -- 172.21.15.132:0/858060420 >> 172.21.15.132:0/858060420 conn(0x7fccc80f09a0 msgr2=0x7fccc80f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:12:42.835 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:42.832+0000 7fcccd82d700 1 -- 172.21.15.132:0/858060420 shutdown_connections 2024-02-20T00:12:42.835 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:42.832+0000 7fcccd82d700 1 -- 172.21.15.132:0/858060420 wait complete. 2024-02-20T00:12:42.835 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:42.832+0000 7fcccd82d700 1 Processor -- start 2024-02-20T00:12:42.835 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:42.832+0000 7fcccd82d700 1 -- start start 2024-02-20T00:12:42.837 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:42.833+0000 7fcccd82d700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fccc80f51a0 0x7fccc81054d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:12:42.837 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:42.833+0000 7fcccd82d700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccc81059c0 con 0x7fccc80f51a0 2024-02-20T00:12:42.837 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:42.833+0000 7fcccc82b700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fccc80f51a0 0x7fccc81054d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:12:42.837 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:42.833+0000 7fcccc82b700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fccc80f51a0 0x7fccc81054d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:45314/0 (socket says 172.21.15.132:45314) 2024-02-20T00:12:42.837 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:42.833+0000 7fcccc82b700 1 -- 172.21.15.132:0/3371874787 learned_addr learned my addr 172.21.15.132:0/3371874787 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:12:42.838 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:42.833+0000 7fcccc82b700 1 -- 172.21.15.132:0/3371874787 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccbc005040 con 0x7fccc80f51a0 2024-02-20T00:12:42.838 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:42.833+0000 7fcccc82b700 1 --2- 172.21.15.132:0/3371874787 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fccc80f51a0 0x7fccc81054d0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7fccbc0089f0 tx=0x7fccbc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:12:42.838 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:42.834+0000 7fccc57fa700 1 -- 172.21.15.132:0/3371874787 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fccbc009320 con 0x7fccc80f51a0 2024-02-20T00:12:42.838 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:42.834+0000 7fcccd82d700 1 -- 172.21.15.132:0/3371874787 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccc8105bc0 con 0x7fccc80f51a0 2024-02-20T00:12:42.838 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:42.834+0000 7fccc57fa700 1 -- 172.21.15.132:0/3371874787 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fccbc013380 con 0x7fccc80f51a0 2024-02-20T00:12:42.838 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:42.834+0000 7fccc57fa700 1 -- 172.21.15.132:0/3371874787 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fccbc014e20 con 0x7fccc80f51a0 2024-02-20T00:12:42.838 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:42.834+0000 7fcccd82d700 1 -- 172.21.15.132:0/3371874787 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccc8106050 con 0x7fccc80f51a0 2024-02-20T00:12:42.838 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:42.834+0000 7fccc57fa700 1 -- 172.21.15.132:0/3371874787 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fccbc0095f0 con 0x7fccc80f51a0 2024-02-20T00:12:42.838 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:42.834+0000 7fcccd82d700 1 -- 172.21.15.132:0/3371874787 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccc8045ad0 con 0x7fccc80f51a0 2024-02-20T00:12:42.838 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:42.835+0000 7fccc57fa700 1 --2- 172.21.15.132:0/3371874787 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fccb802ff10 0x7fccb80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:12:42.838 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:42.835+0000 7fccc57fa700 1 -- 172.21.15.132:0/3371874787 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fccbc040f50 con 0x7fccc80f51a0 2024-02-20T00:12:42.838 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:42.835+0000 7fccc7fff700 1 --2- 172.21.15.132:0/3371874787 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fccb802ff10 0x7fccb80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:12:42.838 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:42.835+0000 7fccc7fff700 1 --2- 172.21.15.132:0/3371874787 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fccb802ff10 0x7fccb80323d0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7fccb40096a0 tx=0x7fccb4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:12:42.841 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:42.839+0000 7fccc57fa700 1 -- 172.21.15.132:0/3371874787 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fccbc014420 con 0x7fccc80f51a0 2024-02-20T00:12:43.044 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:43.042+0000 7fcccd82d700 1 -- 172.21.15.132:0/3371874787 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fccc8106f00 con 0x7fccc80f51a0 2024-02-20T00:12:43.045 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:43.044+0000 7fccc57fa700 1 -- 172.21.15.132:0/3371874787 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fccbc01d980 con 0x7fccc80f51a0 2024-02-20T00:12:43.045 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:12:43.045 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:12:43.047 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:43.044+0000 7fcccd82d700 1 -- 172.21.15.132:0/3371874787 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fccb802ff10 msgr2=0x7fccb80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:12:43.047 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:43.044+0000 7fcccd82d700 1 --2- 172.21.15.132:0/3371874787 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fccb802ff10 0x7fccb80323d0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7fccb40096a0 tx=0x7fccb4006b40).stop 2024-02-20T00:12:43.047 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:43.044+0000 7fcccd82d700 1 -- 172.21.15.132:0/3371874787 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fccc80f51a0 msgr2=0x7fccc81054d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:12:43.047 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:43.044+0000 7fcccd82d700 1 --2- 172.21.15.132:0/3371874787 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fccc80f51a0 0x7fccc81054d0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7fccbc0089f0 tx=0x7fccbc00b040).stop 2024-02-20T00:12:43.047 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:43.044+0000 7fcccd82d700 1 -- 172.21.15.132:0/3371874787 shutdown_connections 2024-02-20T00:12:43.047 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:43.044+0000 7fcccd82d700 1 --2- 172.21.15.132:0/3371874787 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fccb802ff10 0x7fccb80323d0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:12:43.047 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:43.044+0000 7fcccd82d700 1 --2- 172.21.15.132:0/3371874787 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fccc80f51a0 0x7fccc81054d0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:12:43.047 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:43.044+0000 7fcccd82d700 1 -- 172.21.15.132:0/3371874787 >> 172.21.15.132:0/3371874787 conn(0x7fccc80f09a0 msgr2=0x7fccc80f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:12:43.047 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:43.045+0000 7fcccd82d700 1 -- 172.21.15.132:0/3371874787 shutdown_connections 2024-02-20T00:12:43.047 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:43.045+0000 7fcccd82d700 1 -- 172.21.15.132:0/3371874787 wait complete. 2024-02-20T00:12:43.048 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:12:44.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:12:43 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:12:42.861208+0000 mgr.smithi084.bmqrnf (mgr.14162) 84 : cluster [DBG] pgmap v77: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:12:44.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:12:43 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:12:43.044408+0000 mon.smithi084 (mon.0) 156 : audit [DBG] from='client.? 172.21.15.132:0/3371874787' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:12:44.687 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:12:44.688 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:12:46.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:12:45 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:12:44.861472+0000 mgr.smithi084.bmqrnf (mgr.14162) 85 : cluster [DBG] pgmap v78: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:12:46.529 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:46.527+0000 7fa701baa700 1 -- 172.21.15.132:0/1021995931 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa6fc060280 msgr2=0x7fa6fc0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:12:46.529 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:46.528+0000 7fa701baa700 1 --2- 172.21.15.132:0/1021995931 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa6fc060280 0x7fa6fc0f90f0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7fa6ec009fb0 tx=0x7fa6ec009b30).stop 2024-02-20T00:12:46.530 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:46.528+0000 7fa701baa700 1 -- 172.21.15.132:0/1021995931 shutdown_connections 2024-02-20T00:12:46.530 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:46.528+0000 7fa701baa700 1 --2- 172.21.15.132:0/1021995931 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa6fc060280 0x7fa6fc0f90f0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:12:46.530 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:46.528+0000 7fa701baa700 1 -- 172.21.15.132:0/1021995931 >> 172.21.15.132:0/1021995931 conn(0x7fa6fc0f0b70 msgr2=0x7fa6fc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:12:46.530 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:46.528+0000 7fa701baa700 1 -- 172.21.15.132:0/1021995931 shutdown_connections 2024-02-20T00:12:46.530 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:46.528+0000 7fa701baa700 1 -- 172.21.15.132:0/1021995931 wait complete. 2024-02-20T00:12:46.530 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:46.528+0000 7fa701baa700 1 Processor -- start 2024-02-20T00:12:46.532 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:46.528+0000 7fa701baa700 1 -- start start 2024-02-20T00:12:46.532 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:46.529+0000 7fa701baa700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa6fc060280 0x7fa6fc106780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:12:46.532 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:46.529+0000 7fa701baa700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6fc106c70 con 0x7fa6fc060280 2024-02-20T00:12:46.532 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:46.529+0000 7fa6fb7fe700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa6fc060280 0x7fa6fc106780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:12:46.532 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:46.529+0000 7fa6fb7fe700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa6fc060280 0x7fa6fc106780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:45326/0 (socket says 172.21.15.132:45326) 2024-02-20T00:12:46.532 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:46.529+0000 7fa6fb7fe700 1 -- 172.21.15.132:0/2891815645 learned_addr learned my addr 172.21.15.132:0/2891815645 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:12:46.532 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:46.529+0000 7fa6fb7fe700 1 -- 172.21.15.132:0/2891815645 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6ec005040 con 0x7fa6fc060280 2024-02-20T00:12:46.532 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:46.529+0000 7fa6fb7fe700 1 --2- 172.21.15.132:0/2891815645 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa6fc060280 0x7fa6fc106780 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7fa6ec009830 tx=0x7fa6ec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:12:46.532 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:46.530+0000 7fa6e3fff700 1 -- 172.21.15.132:0/2891815645 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa6ec009320 con 0x7fa6fc060280 2024-02-20T00:12:46.533 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:46.530+0000 7fa6e3fff700 1 -- 172.21.15.132:0/2891815645 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa6ec013380 con 0x7fa6fc060280 2024-02-20T00:12:46.533 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:46.530+0000 7fa701baa700 1 -- 172.21.15.132:0/2891815645 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6fc106e70 con 0x7fa6fc060280 2024-02-20T00:12:46.533 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:46.530+0000 7fa6e3fff700 1 -- 172.21.15.132:0/2891815645 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa6ec014420 con 0x7fa6fc060280 2024-02-20T00:12:46.533 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:46.530+0000 7fa701baa700 1 -- 172.21.15.132:0/2891815645 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6fc107300 con 0x7fa6fc060280 2024-02-20T00:12:46.533 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:46.530+0000 7fa6e3fff700 1 -- 172.21.15.132:0/2891815645 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa6ec01e420 con 0x7fa6fc060280 2024-02-20T00:12:46.533 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:46.530+0000 7fa701baa700 1 -- 172.21.15.132:0/2891815645 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6fc045ad0 con 0x7fa6fc060280 2024-02-20T00:12:46.533 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:46.531+0000 7fa6e3fff700 1 --2- 172.21.15.132:0/2891815645 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fa6e402ffb0 0x7fa6e4032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:12:46.533 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:46.531+0000 7fa6e3fff700 1 -- 172.21.15.132:0/2891815645 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa6ec040eb0 con 0x7fa6fc060280 2024-02-20T00:12:46.533 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:46.531+0000 7fa6faffd700 1 --2- 172.21.15.132:0/2891815645 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fa6e402ffb0 0x7fa6e4032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:12:46.534 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:46.531+0000 7fa6faffd700 1 --2- 172.21.15.132:0/2891815645 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fa6e402ffb0 0x7fa6e4032470 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7fa6f00096a0 tx=0x7fa6f0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:12:46.536 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:46.535+0000 7fa6e3fff700 1 -- 172.21.15.132:0/2891815645 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa6ec01cb10 con 0x7fa6fc060280 2024-02-20T00:12:46.740 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:46.739+0000 7fa701baa700 1 -- 172.21.15.132:0/2891815645 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa6fc0fc090 con 0x7fa6fc060280 2024-02-20T00:12:46.741 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:46.740+0000 7fa6e3fff700 1 -- 172.21.15.132:0/2891815645 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa6ec02c2d0 con 0x7fa6fc060280 2024-02-20T00:12:46.741 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:12:46.741 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:12:46.743 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:46.741+0000 7fa701baa700 1 -- 172.21.15.132:0/2891815645 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fa6e402ffb0 msgr2=0x7fa6e4032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:12:46.743 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:46.741+0000 7fa701baa700 1 --2- 172.21.15.132:0/2891815645 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fa6e402ffb0 0x7fa6e4032470 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7fa6f00096a0 tx=0x7fa6f0006b40).stop 2024-02-20T00:12:46.743 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:46.741+0000 7fa701baa700 1 -- 172.21.15.132:0/2891815645 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa6fc060280 msgr2=0x7fa6fc106780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:12:46.743 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:46.741+0000 7fa701baa700 1 --2- 172.21.15.132:0/2891815645 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa6fc060280 0x7fa6fc106780 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7fa6ec009830 tx=0x7fa6ec00b040).stop 2024-02-20T00:12:46.743 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:46.741+0000 7fa701baa700 1 -- 172.21.15.132:0/2891815645 shutdown_connections 2024-02-20T00:12:46.743 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:46.742+0000 7fa701baa700 1 --2- 172.21.15.132:0/2891815645 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fa6e402ffb0 0x7fa6e4032470 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:12:46.743 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:46.742+0000 7fa701baa700 1 --2- 172.21.15.132:0/2891815645 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa6fc060280 0x7fa6fc106780 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:12:46.743 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:46.742+0000 7fa701baa700 1 -- 172.21.15.132:0/2891815645 >> 172.21.15.132:0/2891815645 conn(0x7fa6fc0f0b70 msgr2=0x7fa6fc0f15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:12:46.743 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:46.742+0000 7fa701baa700 1 -- 172.21.15.132:0/2891815645 shutdown_connections 2024-02-20T00:12:46.744 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:46.742+0000 7fa701baa700 1 -- 172.21.15.132:0/2891815645 wait complete. 2024-02-20T00:12:46.744 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:12:47.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:12:46 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:12:46.740451+0000 mon.smithi084 (mon.0) 157 : audit [DBG] from='client.? 172.21.15.132:0/2891815645' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:12:48.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:12:47 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:12:46.861716+0000 mgr.smithi084.bmqrnf (mgr.14162) 86 : cluster [DBG] pgmap v79: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:12:48.403 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:12:48.403 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:12:50.148 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:50.145+0000 7f32c54a6700 1 -- 172.21.15.132:0/1052543038 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f32c00f6510 msgr2=0x7f32c00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:12:50.151 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:50.145+0000 7f32c54a6700 1 --2- 172.21.15.132:0/1052543038 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f32c00f6510 0x7f32c00f6930 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f32a8009fb0 tx=0x7f32a8009b30).stop 2024-02-20T00:12:50.151 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:50.145+0000 7f32c54a6700 1 -- 172.21.15.132:0/1052543038 shutdown_connections 2024-02-20T00:12:50.151 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:50.145+0000 7f32c54a6700 1 --2- 172.21.15.132:0/1052543038 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f32c00f6510 0x7f32c00f6930 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:12:50.151 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:50.145+0000 7f32c54a6700 1 -- 172.21.15.132:0/1052543038 >> 172.21.15.132:0/1052543038 conn(0x7f32c00f1c70 msgr2=0x7f32c00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:12:50.151 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:50.145+0000 7f32c54a6700 1 -- 172.21.15.132:0/1052543038 shutdown_connections 2024-02-20T00:12:50.151 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:50.145+0000 7f32c54a6700 1 -- 172.21.15.132:0/1052543038 wait complete. 2024-02-20T00:12:50.151 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:50.146+0000 7f32c54a6700 1 Processor -- start 2024-02-20T00:12:50.152 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:50.146+0000 7f32c54a6700 1 -- start start 2024-02-20T00:12:50.152 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:50.146+0000 7f32c54a6700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f32c00f6510 0x7f32c01089c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:12:50.152 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:50.146+0000 7f32c54a6700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32c0108eb0 con 0x7f32c00f6510 2024-02-20T00:12:50.152 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:50.146+0000 7f32beffd700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f32c00f6510 0x7f32c01089c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:12:50.152 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:50.146+0000 7f32beffd700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f32c00f6510 0x7f32c01089c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:48280/0 (socket says 172.21.15.132:48280) 2024-02-20T00:12:50.152 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:50.146+0000 7f32beffd700 1 -- 172.21.15.132:0/2178343284 learned_addr learned my addr 172.21.15.132:0/2178343284 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:12:50.152 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:50.147+0000 7f32beffd700 1 -- 172.21.15.132:0/2178343284 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32a8005040 con 0x7f32c00f6510 2024-02-20T00:12:50.152 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:50.147+0000 7f32beffd700 1 --2- 172.21.15.132:0/2178343284 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f32c00f6510 0x7f32c01089c0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f32a80089f0 tx=0x7f32a800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:12:50.152 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:50.147+0000 7f32b77fe700 1 -- 172.21.15.132:0/2178343284 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f32a8009320 con 0x7f32c00f6510 2024-02-20T00:12:50.152 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:50.147+0000 7f32c54a6700 1 -- 172.21.15.132:0/2178343284 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32c01090b0 con 0x7f32c00f6510 2024-02-20T00:12:50.152 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:50.147+0000 7f32b77fe700 1 -- 172.21.15.132:0/2178343284 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f32a8013380 con 0x7f32c00f6510 2024-02-20T00:12:50.152 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:50.147+0000 7f32b77fe700 1 -- 172.21.15.132:0/2178343284 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f32a801ed80 con 0x7f32c00f6510 2024-02-20T00:12:50.152 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:50.147+0000 7f32c54a6700 1 -- 172.21.15.132:0/2178343284 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32c0109540 con 0x7f32c00f6510 2024-02-20T00:12:50.154 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:50.148+0000 7f32b77fe700 1 -- 172.21.15.132:0/2178343284 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f32a8014420 con 0x7f32c00f6510 2024-02-20T00:12:50.154 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:50.148+0000 7f32c54a6700 1 -- 172.21.15.132:0/2178343284 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32c0045ad0 con 0x7f32c00f6510 2024-02-20T00:12:50.154 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:50.148+0000 7f32b77fe700 1 --2- 172.21.15.132:0/2178343284 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f32ac02ff60 0x7f32ac032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:12:50.154 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:50.148+0000 7f32b77fe700 1 -- 172.21.15.132:0/2178343284 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f32a80411f0 con 0x7f32c00f6510 2024-02-20T00:12:50.154 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:50.148+0000 7f32be7fc700 1 --2- 172.21.15.132:0/2178343284 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f32ac02ff60 0x7f32ac032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:12:50.154 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:50.149+0000 7f32be7fc700 1 --2- 172.21.15.132:0/2178343284 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f32ac02ff60 0x7f32ac032420 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f32b00096a0 tx=0x7f32b0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:12:50.154 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:50.153+0000 7f32b77fe700 1 -- 172.21.15.132:0/2178343284 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f32a801c6e0 con 0x7f32c00f6510 2024-02-20T00:12:50.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:12:49 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:12:48.862033+0000 mgr.smithi084.bmqrnf (mgr.14162) 87 : cluster [DBG] pgmap v80: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:12:50.358 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:50.356+0000 7f32c54a6700 1 -- 172.21.15.132:0/2178343284 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f32c010a3f0 con 0x7f32c00f6510 2024-02-20T00:12:50.358 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:50.357+0000 7f32b77fe700 1 -- 172.21.15.132:0/2178343284 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f32a802c2d0 con 0x7f32c00f6510 2024-02-20T00:12:50.358 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:12:50.359 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:12:50.360 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:50.358+0000 7f32c54a6700 1 -- 172.21.15.132:0/2178343284 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f32ac02ff60 msgr2=0x7f32ac032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:12:50.360 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:50.358+0000 7f32c54a6700 1 --2- 172.21.15.132:0/2178343284 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f32ac02ff60 0x7f32ac032420 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f32b00096a0 tx=0x7f32b0006b40).stop 2024-02-20T00:12:50.360 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:50.358+0000 7f32c54a6700 1 -- 172.21.15.132:0/2178343284 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f32c00f6510 msgr2=0x7f32c01089c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:12:50.360 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:50.358+0000 7f32c54a6700 1 --2- 172.21.15.132:0/2178343284 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f32c00f6510 0x7f32c01089c0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f32a80089f0 tx=0x7f32a800b040).stop 2024-02-20T00:12:50.361 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:50.358+0000 7f32c54a6700 1 -- 172.21.15.132:0/2178343284 shutdown_connections 2024-02-20T00:12:50.361 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:50.358+0000 7f32c54a6700 1 --2- 172.21.15.132:0/2178343284 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f32ac02ff60 0x7f32ac032420 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:12:50.361 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:50.358+0000 7f32c54a6700 1 --2- 172.21.15.132:0/2178343284 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f32c00f6510 0x7f32c01089c0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:12:50.361 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:50.358+0000 7f32c54a6700 1 -- 172.21.15.132:0/2178343284 >> 172.21.15.132:0/2178343284 conn(0x7f32c00f1c70 msgr2=0x7f32c00f2640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:12:50.361 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:50.359+0000 7f32c54a6700 1 -- 172.21.15.132:0/2178343284 shutdown_connections 2024-02-20T00:12:50.361 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:50.359+0000 7f32c54a6700 1 -- 172.21.15.132:0/2178343284 wait complete. 2024-02-20T00:12:50.361 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:12:51.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:12:50 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:12:50.357383+0000 mon.smithi084 (mon.0) 158 : audit [DBG] from='client.? 172.21.15.132:0/2178343284' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:12:51.930 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:12:51.931 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:12:52.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:12:51 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:12:50.862240+0000 mgr.smithi084.bmqrnf (mgr.14162) 88 : cluster [DBG] pgmap v81: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:12:53.922 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:53.870+0000 7fe824ca3700 1 -- 172.21.15.132:0/3884624454 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe8200f5410 msgr2=0x7fe8200f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:12:53.922 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:53.870+0000 7fe824ca3700 1 --2- 172.21.15.132:0/3884624454 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe8200f5410 0x7fe8200f5830 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7fe810009fb0 tx=0x7fe810009b30).stop 2024-02-20T00:12:53.922 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:53.870+0000 7fe824ca3700 1 -- 172.21.15.132:0/3884624454 shutdown_connections 2024-02-20T00:12:53.923 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:53.870+0000 7fe824ca3700 1 --2- 172.21.15.132:0/3884624454 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe8200f5410 0x7fe8200f5830 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:12:53.923 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:53.870+0000 7fe824ca3700 1 -- 172.21.15.132:0/3884624454 >> 172.21.15.132:0/3884624454 conn(0x7fe8200f0b70 msgr2=0x7fe8200f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:12:53.923 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:53.870+0000 7fe824ca3700 1 -- 172.21.15.132:0/3884624454 shutdown_connections 2024-02-20T00:12:53.923 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:53.870+0000 7fe824ca3700 1 -- 172.21.15.132:0/3884624454 wait complete. 2024-02-20T00:12:53.923 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:53.870+0000 7fe824ca3700 1 Processor -- start 2024-02-20T00:12:53.923 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:53.870+0000 7fe824ca3700 1 -- start start 2024-02-20T00:12:53.923 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:53.871+0000 7fe824ca3700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe8200f5410 0x7fe820105710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:12:53.923 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:53.871+0000 7fe824ca3700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe820105c00 con 0x7fe8200f5410 2024-02-20T00:12:53.923 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:53.871+0000 7fe81e59c700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe8200f5410 0x7fe820105710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:12:53.923 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:53.871+0000 7fe81e59c700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe8200f5410 0x7fe820105710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:48288/0 (socket says 172.21.15.132:48288) 2024-02-20T00:12:53.923 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:53.871+0000 7fe81e59c700 1 -- 172.21.15.132:0/2322911332 learned_addr learned my addr 172.21.15.132:0/2322911332 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:12:53.923 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:53.871+0000 7fe81e59c700 1 -- 172.21.15.132:0/2322911332 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe810005040 con 0x7fe8200f5410 2024-02-20T00:12:53.923 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:53.871+0000 7fe81e59c700 1 --2- 172.21.15.132:0/2322911332 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe8200f5410 0x7fe820105710 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7fe810005010 tx=0x7fe81000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:12:53.923 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:53.871+0000 7fe80effd700 1 -- 172.21.15.132:0/2322911332 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe810009320 con 0x7fe8200f5410 2024-02-20T00:12:53.924 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:53.871+0000 7fe824ca3700 1 -- 172.21.15.132:0/2322911332 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe820105e00 con 0x7fe8200f5410 2024-02-20T00:12:53.924 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:53.871+0000 7fe80effd700 1 -- 172.21.15.132:0/2322911332 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe810013380 con 0x7fe8200f5410 2024-02-20T00:12:53.924 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:53.871+0000 7fe80effd700 1 -- 172.21.15.132:0/2322911332 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe810014e50 con 0x7fe8200f5410 2024-02-20T00:12:53.924 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:53.872+0000 7fe824ca3700 1 -- 172.21.15.132:0/2322911332 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe820106290 con 0x7fe8200f5410 2024-02-20T00:12:53.924 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:53.872+0000 7fe80effd700 1 -- 172.21.15.132:0/2322911332 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe810014420 con 0x7fe8200f5410 2024-02-20T00:12:53.924 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:53.872+0000 7fe824ca3700 1 -- 172.21.15.132:0/2322911332 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe820045ad0 con 0x7fe8200f5410 2024-02-20T00:12:53.924 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:53.872+0000 7fe80effd700 1 --2- 172.21.15.132:0/2322911332 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fe80802ff10 0x7fe8080323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:12:53.924 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:53.872+0000 7fe80effd700 1 -- 172.21.15.132:0/2322911332 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe810040f40 con 0x7fe8200f5410 2024-02-20T00:12:53.924 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:53.873+0000 7fe81dd9b700 1 --2- 172.21.15.132:0/2322911332 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fe80802ff10 0x7fe8080323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:12:53.924 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:53.873+0000 7fe81dd9b700 1 --2- 172.21.15.132:0/2322911332 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fe80802ff10 0x7fe8080323d0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7fe8140096a0 tx=0x7fe814006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:12:53.924 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:53.877+0000 7fe80effd700 1 -- 172.21.15.132:0/2322911332 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe8100095f0 con 0x7fe8200f5410 2024-02-20T00:12:54.085 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:54.084+0000 7fe824ca3700 1 -- 172.21.15.132:0/2322911332 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe820107140 con 0x7fe8200f5410 2024-02-20T00:12:54.086 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:54.085+0000 7fe80effd700 1 -- 172.21.15.132:0/2322911332 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe81001db70 con 0x7fe8200f5410 2024-02-20T00:12:54.086 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:12:54.086 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:12:54.088 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:54.086+0000 7fe824ca3700 1 -- 172.21.15.132:0/2322911332 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fe80802ff10 msgr2=0x7fe8080323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:12:54.088 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:54.086+0000 7fe824ca3700 1 --2- 172.21.15.132:0/2322911332 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fe80802ff10 0x7fe8080323d0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7fe8140096a0 tx=0x7fe814006b40).stop 2024-02-20T00:12:54.088 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:54.086+0000 7fe824ca3700 1 -- 172.21.15.132:0/2322911332 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe8200f5410 msgr2=0x7fe820105710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:12:54.088 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:54.086+0000 7fe824ca3700 1 --2- 172.21.15.132:0/2322911332 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe8200f5410 0x7fe820105710 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7fe810005010 tx=0x7fe81000b040).stop 2024-02-20T00:12:54.088 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:54.086+0000 7fe824ca3700 1 -- 172.21.15.132:0/2322911332 shutdown_connections 2024-02-20T00:12:54.088 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:54.086+0000 7fe824ca3700 1 --2- 172.21.15.132:0/2322911332 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fe80802ff10 0x7fe8080323d0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:12:54.088 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:54.086+0000 7fe824ca3700 1 --2- 172.21.15.132:0/2322911332 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe8200f5410 0x7fe820105710 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:12:54.088 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:54.086+0000 7fe824ca3700 1 -- 172.21.15.132:0/2322911332 >> 172.21.15.132:0/2322911332 conn(0x7fe8200f0b70 msgr2=0x7fe8200f15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:12:54.088 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:54.086+0000 7fe824ca3700 1 -- 172.21.15.132:0/2322911332 shutdown_connections 2024-02-20T00:12:54.088 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:54.086+0000 7fe824ca3700 1 -- 172.21.15.132:0/2322911332 wait complete. 2024-02-20T00:12:54.089 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:12:54.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:12:53 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:12:52.862576+0000 mgr.smithi084.bmqrnf (mgr.14162) 89 : cluster [DBG] pgmap v82: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:12:55.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:12:54 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:12:54.085441+0000 mon.smithi084 (mon.0) 159 : audit [DBG] from='client.? 172.21.15.132:0/2322911332' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:12:55.709 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:12:55.710 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:12:56.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:12:55 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:12:54.862848+0000 mgr.smithi084.bmqrnf (mgr.14162) 90 : cluster [DBG] pgmap v83: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:12:57.520 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:57.517+0000 7fd8390e7700 1 -- 172.21.15.132:0/4154194262 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd8340f51a0 msgr2=0x7fd8340f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:12:57.520 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:57.517+0000 7fd8390e7700 1 --2- 172.21.15.132:0/4154194262 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd8340f51a0 0x7fd8340f55c0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7fd824009fb0 tx=0x7fd824009b30).stop 2024-02-20T00:12:57.520 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:57.517+0000 7fd8390e7700 1 -- 172.21.15.132:0/4154194262 shutdown_connections 2024-02-20T00:12:57.521 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:57.517+0000 7fd8390e7700 1 --2- 172.21.15.132:0/4154194262 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd8340f51a0 0x7fd8340f55c0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:12:57.521 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:57.517+0000 7fd8390e7700 1 -- 172.21.15.132:0/4154194262 >> 172.21.15.132:0/4154194262 conn(0x7fd8340f09a0 msgr2=0x7fd8340f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:12:57.521 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:57.517+0000 7fd8390e7700 1 -- 172.21.15.132:0/4154194262 shutdown_connections 2024-02-20T00:12:57.521 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:57.517+0000 7fd8390e7700 1 -- 172.21.15.132:0/4154194262 wait complete. 2024-02-20T00:12:57.521 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:57.517+0000 7fd8390e7700 1 Processor -- start 2024-02-20T00:12:57.521 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:57.518+0000 7fd8390e7700 1 -- start start 2024-02-20T00:12:57.521 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:57.518+0000 7fd8390e7700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd8340f51a0 0x7fd8341065c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:12:57.521 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:57.518+0000 7fd8390e7700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd834106ab0 con 0x7fd8340f51a0 2024-02-20T00:12:57.521 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:57.518+0000 7fd833fff700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd8340f51a0 0x7fd8341065c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:12:57.521 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:57.518+0000 7fd833fff700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd8340f51a0 0x7fd8341065c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:34612/0 (socket says 172.21.15.132:34612) 2024-02-20T00:12:57.521 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:57.518+0000 7fd833fff700 1 -- 172.21.15.132:0/1620116859 learned_addr learned my addr 172.21.15.132:0/1620116859 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:12:57.521 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:57.518+0000 7fd833fff700 1 -- 172.21.15.132:0/1620116859 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd824005040 con 0x7fd8340f51a0 2024-02-20T00:12:57.521 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:57.519+0000 7fd833fff700 1 --2- 172.21.15.132:0/1620116859 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd8340f51a0 0x7fd8341065c0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7fd8240089f0 tx=0x7fd82400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:12:57.521 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:57.519+0000 7fd830ff9700 1 -- 172.21.15.132:0/1620116859 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd824009320 con 0x7fd8340f51a0 2024-02-20T00:12:57.522 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:57.519+0000 7fd8390e7700 1 -- 172.21.15.132:0/1620116859 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd834106cb0 con 0x7fd8340f51a0 2024-02-20T00:12:57.522 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:57.519+0000 7fd830ff9700 1 -- 172.21.15.132:0/1620116859 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd824013380 con 0x7fd8340f51a0 2024-02-20T00:12:57.522 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:57.519+0000 7fd830ff9700 1 -- 172.21.15.132:0/1620116859 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd824014e20 con 0x7fd8340f51a0 2024-02-20T00:12:57.522 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:57.519+0000 7fd8390e7700 1 -- 172.21.15.132:0/1620116859 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd834107140 con 0x7fd8340f51a0 2024-02-20T00:12:57.523 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:57.519+0000 7fd830ff9700 1 -- 172.21.15.132:0/1620116859 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd824014420 con 0x7fd8340f51a0 2024-02-20T00:12:57.523 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:57.520+0000 7fd8390e7700 1 -- 172.21.15.132:0/1620116859 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd834045ad0 con 0x7fd8340f51a0 2024-02-20T00:12:57.523 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:57.520+0000 7fd830ff9700 1 --2- 172.21.15.132:0/1620116859 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fd81c02ff10 0x7fd81c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:12:57.523 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:57.520+0000 7fd830ff9700 1 -- 172.21.15.132:0/1620116859 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd824040ff0 con 0x7fd8340f51a0 2024-02-20T00:12:57.523 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:57.520+0000 7fd8337fe700 1 --2- 172.21.15.132:0/1620116859 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fd81c02ff10 0x7fd81c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:12:57.523 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:57.520+0000 7fd8337fe700 1 --2- 172.21.15.132:0/1620116859 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fd81c02ff10 0x7fd81c0323d0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7fd8280096a0 tx=0x7fd828006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:12:57.525 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:57.524+0000 7fd830ff9700 1 -- 172.21.15.132:0/1620116859 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd8240095f0 con 0x7fd8340f51a0 2024-02-20T00:12:57.730 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:57.729+0000 7fd8390e7700 1 -- 172.21.15.132:0/1620116859 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd834107ff0 con 0x7fd8340f51a0 2024-02-20T00:12:57.731 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:57.729+0000 7fd830ff9700 1 -- 172.21.15.132:0/1620116859 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd82401db70 con 0x7fd8340f51a0 2024-02-20T00:12:57.731 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:12:57.731 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:12:57.732 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:57.731+0000 7fd8390e7700 1 -- 172.21.15.132:0/1620116859 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fd81c02ff10 msgr2=0x7fd81c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:12:57.732 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:57.731+0000 7fd8390e7700 1 --2- 172.21.15.132:0/1620116859 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fd81c02ff10 0x7fd81c0323d0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7fd8280096a0 tx=0x7fd828006b40).stop 2024-02-20T00:12:57.732 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:57.731+0000 7fd8390e7700 1 -- 172.21.15.132:0/1620116859 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd8340f51a0 msgr2=0x7fd8341065c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:12:57.732 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:57.731+0000 7fd8390e7700 1 --2- 172.21.15.132:0/1620116859 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd8340f51a0 0x7fd8341065c0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7fd8240089f0 tx=0x7fd82400b040).stop 2024-02-20T00:12:57.732 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:57.731+0000 7fd8390e7700 1 -- 172.21.15.132:0/1620116859 shutdown_connections 2024-02-20T00:12:57.733 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:57.731+0000 7fd8390e7700 1 --2- 172.21.15.132:0/1620116859 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fd81c02ff10 0x7fd81c0323d0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:12:57.733 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:57.731+0000 7fd8390e7700 1 --2- 172.21.15.132:0/1620116859 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd8340f51a0 0x7fd8341065c0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:12:57.733 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:57.731+0000 7fd8390e7700 1 -- 172.21.15.132:0/1620116859 >> 172.21.15.132:0/1620116859 conn(0x7fd8340f09a0 msgr2=0x7fd83405fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:12:57.733 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:57.731+0000 7fd8390e7700 1 -- 172.21.15.132:0/1620116859 shutdown_connections 2024-02-20T00:12:57.733 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:12:57.731+0000 7fd8390e7700 1 -- 172.21.15.132:0/1620116859 wait complete. 2024-02-20T00:12:57.733 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:12:58.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:12:57 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:12:56.863227+0000 mgr.smithi084.bmqrnf (mgr.14162) 91 : cluster [DBG] pgmap v84: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:12:58.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:12:57 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:12:56.972119+0000 mon.smithi084 (mon.0) 160 : audit [INF] from='mgr.14162 172.21.15.84:0/2152933216' entity='mgr.smithi084.bmqrnf' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi084.bmqrnf/mirror_snapshot_schedule"}]: dispatch 2024-02-20T00:12:58.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:12:57 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:12:56.978536+0000 mon.smithi084 (mon.0) 161 : audit [INF] from='mgr.14162 172.21.15.84:0/2152933216' entity='mgr.smithi084.bmqrnf' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi084.bmqrnf/trash_purge_schedule"}]: dispatch 2024-02-20T00:12:58.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:12:57 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:12:57.730057+0000 mon.smithi084 (mon.0) 162 : audit [DBG] from='client.? 172.21.15.132:0/1620116859' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:12:59.390 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:12:59.391 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:13:00.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:12:59 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:12:58.863674+0000 mgr.smithi084.bmqrnf (mgr.14162) 92 : cluster [DBG] pgmap v85: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:13:01.169 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:01.167+0000 7f6bf7fff700 1 -- 172.21.15.132:0/596084299 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6bf80f62a0 msgr2=0x7f6bf80f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:13:01.169 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:01.167+0000 7f6bf7fff700 1 --2- 172.21.15.132:0/596084299 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6bf80f62a0 0x7f6bf80f66c0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7f6be8009fb0 tx=0x7f6be8009b30).stop 2024-02-20T00:13:01.169 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:01.167+0000 7f6bf7fff700 1 -- 172.21.15.132:0/596084299 shutdown_connections 2024-02-20T00:13:01.169 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:01.167+0000 7f6bf7fff700 1 --2- 172.21.15.132:0/596084299 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6bf80f62a0 0x7f6bf80f66c0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:13:01.169 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:01.167+0000 7f6bf7fff700 1 -- 172.21.15.132:0/596084299 >> 172.21.15.132:0/596084299 conn(0x7f6bf80f1aa0 msgr2=0x7f6bf80f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:13:01.169 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:01.167+0000 7f6bf7fff700 1 -- 172.21.15.132:0/596084299 shutdown_connections 2024-02-20T00:13:01.169 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:01.167+0000 7f6bf7fff700 1 -- 172.21.15.132:0/596084299 wait complete. 2024-02-20T00:13:01.169 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:01.167+0000 7f6bf7fff700 1 Processor -- start 2024-02-20T00:13:01.169 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:01.167+0000 7f6bf7fff700 1 -- start start 2024-02-20T00:13:01.170 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:01.168+0000 7f6bf7fff700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6bf80f62a0 0x7f6bf81065d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:13:01.170 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:01.168+0000 7f6bf7fff700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bf8106ac0 con 0x7f6bf80f62a0 2024-02-20T00:13:01.170 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:01.168+0000 7f6bf6ffd700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6bf80f62a0 0x7f6bf81065d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:13:01.170 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:01.168+0000 7f6bf6ffd700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6bf80f62a0 0x7f6bf81065d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:34636/0 (socket says 172.21.15.132:34636) 2024-02-20T00:13:01.170 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:01.168+0000 7f6bf6ffd700 1 -- 172.21.15.132:0/526653836 learned_addr learned my addr 172.21.15.132:0/526653836 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:13:01.171 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:01.168+0000 7f6bf6ffd700 1 -- 172.21.15.132:0/526653836 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6be8005040 con 0x7f6bf80f62a0 2024-02-20T00:13:01.171 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:01.168+0000 7f6bf6ffd700 1 --2- 172.21.15.132:0/526653836 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6bf80f62a0 0x7f6bf81065d0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f6be80089f0 tx=0x7f6be800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:13:01.171 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:01.168+0000 7f6bdf7fe700 1 -- 172.21.15.132:0/526653836 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6be8009320 con 0x7f6bf80f62a0 2024-02-20T00:13:01.171 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:01.168+0000 7f6bf7fff700 1 -- 172.21.15.132:0/526653836 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bf8106cc0 con 0x7f6bf80f62a0 2024-02-20T00:13:01.171 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:01.168+0000 7f6bdf7fe700 1 -- 172.21.15.132:0/526653836 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6be8013380 con 0x7f6bf80f62a0 2024-02-20T00:13:01.171 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:01.168+0000 7f6bdf7fe700 1 -- 172.21.15.132:0/526653836 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6be8014e20 con 0x7f6bf80f62a0 2024-02-20T00:13:01.172 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:01.168+0000 7f6bf7fff700 1 -- 172.21.15.132:0/526653836 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bf8107150 con 0x7f6bf80f62a0 2024-02-20T00:13:01.172 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:01.169+0000 7f6bdf7fe700 1 -- 172.21.15.132:0/526653836 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6be8014420 con 0x7f6bf80f62a0 2024-02-20T00:13:01.172 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:01.169+0000 7f6bf7fff700 1 -- 172.21.15.132:0/526653836 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bf8045ad0 con 0x7f6bf80f62a0 2024-02-20T00:13:01.172 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:01.169+0000 7f6bdf7fe700 1 --2- 172.21.15.132:0/526653836 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f6be002ff10 0x7f6be00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:13:01.172 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:01.170+0000 7f6bdf7fe700 1 -- 172.21.15.132:0/526653836 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6be8040ed0 con 0x7f6bf80f62a0 2024-02-20T00:13:01.172 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:01.170+0000 7f6bf67fc700 1 --2- 172.21.15.132:0/526653836 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f6be002ff10 0x7f6be00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:13:01.172 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:01.170+0000 7f6bf67fc700 1 --2- 172.21.15.132:0/526653836 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f6be002ff10 0x7f6be00323d0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f6bec0096a0 tx=0x7f6bec006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:13:01.175 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:01.174+0000 7f6bdf7fe700 1 -- 172.21.15.132:0/526653836 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6be8014790 con 0x7f6bf80f62a0 2024-02-20T00:13:01.378 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:01.377+0000 7f6bf7fff700 1 -- 172.21.15.132:0/526653836 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6bf8107fa0 con 0x7f6bf80f62a0 2024-02-20T00:13:01.379 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:01.377+0000 7f6bdf7fe700 1 -- 172.21.15.132:0/526653836 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6be801d7f0 con 0x7f6bf80f62a0 2024-02-20T00:13:01.379 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:13:01.379 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:13:01.380 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:01.379+0000 7f6bf7fff700 1 -- 172.21.15.132:0/526653836 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f6be002ff10 msgr2=0x7f6be00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:13:01.380 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:01.379+0000 7f6bf7fff700 1 --2- 172.21.15.132:0/526653836 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f6be002ff10 0x7f6be00323d0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f6bec0096a0 tx=0x7f6bec006b40).stop 2024-02-20T00:13:01.380 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:01.379+0000 7f6bf7fff700 1 -- 172.21.15.132:0/526653836 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6bf80f62a0 msgr2=0x7f6bf81065d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:13:01.381 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:01.379+0000 7f6bf7fff700 1 --2- 172.21.15.132:0/526653836 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6bf80f62a0 0x7f6bf81065d0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f6be80089f0 tx=0x7f6be800b040).stop 2024-02-20T00:13:01.381 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:01.379+0000 7f6bf7fff700 1 -- 172.21.15.132:0/526653836 shutdown_connections 2024-02-20T00:13:01.381 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:01.379+0000 7f6bf7fff700 1 --2- 172.21.15.132:0/526653836 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f6be002ff10 0x7f6be00323d0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:13:01.381 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:01.379+0000 7f6bf7fff700 1 --2- 172.21.15.132:0/526653836 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6bf80f62a0 0x7f6bf81065d0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:13:01.381 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:01.379+0000 7f6bf7fff700 1 -- 172.21.15.132:0/526653836 >> 172.21.15.132:0/526653836 conn(0x7f6bf80f1aa0 msgr2=0x7f6bf80f2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:13:01.381 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:01.379+0000 7f6bf7fff700 1 -- 172.21.15.132:0/526653836 shutdown_connections 2024-02-20T00:13:01.381 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:01.379+0000 7f6bf7fff700 1 -- 172.21.15.132:0/526653836 wait complete. 2024-02-20T00:13:01.381 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:13:02.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:13:01 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:13:00.863917+0000 mgr.smithi084.bmqrnf (mgr.14162) 93 : cluster [DBG] pgmap v86: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:13:02.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:13:01 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:13:01.378054+0000 mon.smithi084 (mon.0) 163 : audit [DBG] from='client.? 172.21.15.132:0/526653836' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:13:02.965 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:13:02.966 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:13:04.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:13:03 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:13:02.864254+0000 mgr.smithi084.bmqrnf (mgr.14162) 94 : cluster [DBG] pgmap v87: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:13:04.928 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:04.924+0000 7f4867739700 1 -- 172.21.15.132:0/3331335161 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f48600f5330 msgr2=0x7f48600f5750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:13:04.928 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:04.924+0000 7f4867739700 1 --2- 172.21.15.132:0/3331335161 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f48600f5330 0x7f48600f5750 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7f4850009fb0 tx=0x7f4850009b30).stop 2024-02-20T00:13:04.929 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:04.924+0000 7f4867739700 1 -- 172.21.15.132:0/3331335161 shutdown_connections 2024-02-20T00:13:04.929 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:04.924+0000 7f4867739700 1 --2- 172.21.15.132:0/3331335161 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f48600f5330 0x7f48600f5750 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:13:04.929 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:04.924+0000 7f4867739700 1 -- 172.21.15.132:0/3331335161 >> 172.21.15.132:0/3331335161 conn(0x7f48600f0a90 msgr2=0x7f48600f2eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:13:04.929 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:04.924+0000 7f4867739700 1 -- 172.21.15.132:0/3331335161 shutdown_connections 2024-02-20T00:13:04.929 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:04.924+0000 7f4867739700 1 -- 172.21.15.132:0/3331335161 wait complete. 2024-02-20T00:13:04.929 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:04.925+0000 7f4867739700 1 Processor -- start 2024-02-20T00:13:04.929 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:04.925+0000 7f4867739700 1 -- start start 2024-02-20T00:13:04.929 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:04.925+0000 7f4867739700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f48600f5330 0x7f4860109780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:13:04.929 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:04.925+0000 7f4867739700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4860060240 con 0x7f48600f5330 2024-02-20T00:13:04.929 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:04.925+0000 7f48654d5700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f48600f5330 0x7f4860109780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:13:04.929 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:04.925+0000 7f48654d5700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f48600f5330 0x7f4860109780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:34644/0 (socket says 172.21.15.132:34644) 2024-02-20T00:13:04.929 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:04.925+0000 7f48654d5700 1 -- 172.21.15.132:0/1336175743 learned_addr learned my addr 172.21.15.132:0/1336175743 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:13:04.929 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:04.926+0000 7f48654d5700 1 -- 172.21.15.132:0/1336175743 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4850005040 con 0x7f48600f5330 2024-02-20T00:13:04.930 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:04.926+0000 7f48654d5700 1 --2- 172.21.15.132:0/1336175743 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f48600f5330 0x7f4860109780 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f4850005010 tx=0x7f485000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:13:04.930 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:04.926+0000 7f4855ffb700 1 -- 172.21.15.132:0/1336175743 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4850009320 con 0x7f48600f5330 2024-02-20T00:13:04.930 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:04.926+0000 7f4867739700 1 -- 172.21.15.132:0/1336175743 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4860060440 con 0x7f48600f5330 2024-02-20T00:13:04.930 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:04.926+0000 7f4855ffb700 1 -- 172.21.15.132:0/1336175743 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4850013380 con 0x7f48600f5330 2024-02-20T00:13:04.930 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:04.926+0000 7f4855ffb700 1 -- 172.21.15.132:0/1336175743 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4850014e50 con 0x7f48600f5330 2024-02-20T00:13:04.930 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:04.926+0000 7f4867739700 1 -- 172.21.15.132:0/1336175743 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4860109e30 con 0x7f48600f5330 2024-02-20T00:13:04.931 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:04.927+0000 7f4855ffb700 1 -- 172.21.15.132:0/1336175743 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4850014420 con 0x7f48600f5330 2024-02-20T00:13:04.931 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:04.927+0000 7f4867739700 1 -- 172.21.15.132:0/1336175743 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4860045ad0 con 0x7f48600f5330 2024-02-20T00:13:04.931 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:04.927+0000 7f4855ffb700 1 --2- 172.21.15.132:0/1336175743 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f484c02ff10 0x7f484c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:13:04.931 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:04.927+0000 7f4855ffb700 1 -- 172.21.15.132:0/1336175743 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4850040f70 con 0x7f48600f5330 2024-02-20T00:13:04.931 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:04.927+0000 7f4864cd4700 1 --2- 172.21.15.132:0/1336175743 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f484c02ff10 0x7f484c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:13:04.931 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:04.928+0000 7f4864cd4700 1 --2- 172.21.15.132:0/1336175743 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f484c02ff10 0x7f484c0323d0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7f485c0096a0 tx=0x7f485c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:13:04.933 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:04.931+0000 7f4855ffb700 1 -- 172.21.15.132:0/1336175743 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f48500095f0 con 0x7f48600f5330 2024-02-20T00:13:05.139 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:05.137+0000 7f4867739700 1 -- 172.21.15.132:0/1336175743 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f486005ea50 con 0x7f48600f5330 2024-02-20T00:13:05.140 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:05.137+0000 7f4855ffb700 1 -- 172.21.15.132:0/1336175743 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f485001db70 con 0x7f48600f5330 2024-02-20T00:13:05.140 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:13:05.140 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:13:05.141 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:05.139+0000 7f4867739700 1 -- 172.21.15.132:0/1336175743 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f484c02ff10 msgr2=0x7f484c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:13:05.141 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:05.139+0000 7f4867739700 1 --2- 172.21.15.132:0/1336175743 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f484c02ff10 0x7f484c0323d0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7f485c0096a0 tx=0x7f485c006b40).stop 2024-02-20T00:13:05.142 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:05.139+0000 7f4867739700 1 -- 172.21.15.132:0/1336175743 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f48600f5330 msgr2=0x7f4860109780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:13:05.142 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:05.139+0000 7f4867739700 1 --2- 172.21.15.132:0/1336175743 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f48600f5330 0x7f4860109780 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f4850005010 tx=0x7f485000b040).stop 2024-02-20T00:13:05.142 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:05.139+0000 7f4867739700 1 -- 172.21.15.132:0/1336175743 shutdown_connections 2024-02-20T00:13:05.142 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:05.139+0000 7f4867739700 1 --2- 172.21.15.132:0/1336175743 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f484c02ff10 0x7f484c0323d0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:13:05.142 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:05.139+0000 7f4867739700 1 --2- 172.21.15.132:0/1336175743 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f48600f5330 0x7f4860109780 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:13:05.142 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:05.139+0000 7f4867739700 1 -- 172.21.15.132:0/1336175743 >> 172.21.15.132:0/1336175743 conn(0x7f48600f0a90 msgr2=0x7f48600f1580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:13:05.142 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:05.139+0000 7f4867739700 1 -- 172.21.15.132:0/1336175743 shutdown_connections 2024-02-20T00:13:05.142 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:05.139+0000 7f4867739700 1 -- 172.21.15.132:0/1336175743 wait complete. 2024-02-20T00:13:05.143 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:13:06.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:13:05 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:13:04.864645+0000 mgr.smithi084.bmqrnf (mgr.14162) 95 : cluster [DBG] pgmap v88: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:13:06.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:13:05 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:13:05.139101+0000 mon.smithi084 (mon.0) 164 : audit [DBG] from='client.? 172.21.15.132:0/1336175743' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:13:06.838 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:13:06.838 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:13:08.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:13:07 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:13:06.865070+0000 mgr.smithi084.bmqrnf (mgr.14162) 96 : cluster [DBG] pgmap v89: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:13:08.683 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:08.679+0000 7f748272c700 1 -- 172.21.15.132:0/3250355590 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f747c0f51a0 msgr2=0x7f747c0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:13:08.684 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:08.679+0000 7f748272c700 1 --2- 172.21.15.132:0/3250355590 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f747c0f51a0 0x7f747c0f55c0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7f746c009fb0 tx=0x7f746c009b30).stop 2024-02-20T00:13:08.684 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:08.679+0000 7f748272c700 1 -- 172.21.15.132:0/3250355590 shutdown_connections 2024-02-20T00:13:08.684 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:08.679+0000 7f748272c700 1 --2- 172.21.15.132:0/3250355590 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f747c0f51a0 0x7f747c0f55c0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:13:08.684 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:08.679+0000 7f748272c700 1 -- 172.21.15.132:0/3250355590 >> 172.21.15.132:0/3250355590 conn(0x7f747c0f09a0 msgr2=0x7f747c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:13:08.684 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:08.680+0000 7f748272c700 1 -- 172.21.15.132:0/3250355590 shutdown_connections 2024-02-20T00:13:08.684 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:08.680+0000 7f748272c700 1 -- 172.21.15.132:0/3250355590 wait complete. 2024-02-20T00:13:08.684 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:08.680+0000 7f748272c700 1 Processor -- start 2024-02-20T00:13:08.684 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:08.680+0000 7f748272c700 1 -- start start 2024-02-20T00:13:08.684 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:08.680+0000 7f748272c700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f747c0f51a0 0x7f747c0f2f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:13:08.684 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:08.680+0000 7f748272c700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f747c0f13a0 con 0x7f747c0f51a0 2024-02-20T00:13:08.684 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:08.680+0000 7f748172a700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f747c0f51a0 0x7f747c0f2f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:13:08.684 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:08.680+0000 7f748172a700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f747c0f51a0 0x7f747c0f2f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:35240/0 (socket says 172.21.15.132:35240) 2024-02-20T00:13:08.685 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:08.680+0000 7f748172a700 1 -- 172.21.15.132:0/685405222 learned_addr learned my addr 172.21.15.132:0/685405222 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:13:08.685 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:08.681+0000 7f748172a700 1 -- 172.21.15.132:0/685405222 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f746c005040 con 0x7f747c0f51a0 2024-02-20T00:13:08.685 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:08.681+0000 7f748172a700 1 --2- 172.21.15.132:0/685405222 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f747c0f51a0 0x7f747c0f2f20 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f746c0089f0 tx=0x7f746c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:13:08.685 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:08.681+0000 7f7471ffb700 1 -- 172.21.15.132:0/685405222 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f746c009320 con 0x7f747c0f51a0 2024-02-20T00:13:08.685 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:08.681+0000 7f748272c700 1 -- 172.21.15.132:0/685405222 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f747c0f1540 con 0x7f747c0f51a0 2024-02-20T00:13:08.685 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:08.681+0000 7f7471ffb700 1 -- 172.21.15.132:0/685405222 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f746c013380 con 0x7f747c0f51a0 2024-02-20T00:13:08.685 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:08.681+0000 7f7471ffb700 1 -- 172.21.15.132:0/685405222 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f746c014e20 con 0x7f747c0f51a0 2024-02-20T00:13:08.685 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:08.681+0000 7f748272c700 1 -- 172.21.15.132:0/685405222 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f747c0f19d0 con 0x7f747c0f51a0 2024-02-20T00:13:08.686 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:08.682+0000 7f7471ffb700 1 -- 172.21.15.132:0/685405222 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f746c0095f0 con 0x7f747c0f51a0 2024-02-20T00:13:08.686 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:08.682+0000 7f748272c700 1 -- 172.21.15.132:0/685405222 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f747c059100 con 0x7f747c0f51a0 2024-02-20T00:13:08.686 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:08.682+0000 7f7471ffb700 1 --2- 172.21.15.132:0/685405222 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f746802ff10 0x7f74680323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:13:08.686 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:08.682+0000 7f7471ffb700 1 -- 172.21.15.132:0/685405222 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f746c040f50 con 0x7f747c0f51a0 2024-02-20T00:13:08.686 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:08.682+0000 7f7480f29700 1 --2- 172.21.15.132:0/685405222 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f746802ff10 0x7f74680323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:13:08.686 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:08.683+0000 7f7480f29700 1 --2- 172.21.15.132:0/685405222 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f746802ff10 0x7f74680323d0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7f74780096a0 tx=0x7f7478006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:13:08.689 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:08.687+0000 7f7471ffb700 1 -- 172.21.15.132:0/685405222 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f746c014420 con 0x7f747c0f51a0 2024-02-20T00:13:08.894 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:08.892+0000 7f748272c700 1 -- 172.21.15.132:0/685405222 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f747c045ad0 con 0x7f747c0f51a0 2024-02-20T00:13:08.895 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:08.893+0000 7f7471ffb700 1 -- 172.21.15.132:0/685405222 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f746c01d980 con 0x7f747c0f51a0 2024-02-20T00:13:08.895 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:13:08.895 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:13:08.897 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:08.894+0000 7f748272c700 1 -- 172.21.15.132:0/685405222 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f746802ff10 msgr2=0x7f74680323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:13:08.897 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:08.894+0000 7f748272c700 1 --2- 172.21.15.132:0/685405222 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f746802ff10 0x7f74680323d0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7f74780096a0 tx=0x7f7478006b40).stop 2024-02-20T00:13:08.897 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:08.894+0000 7f748272c700 1 -- 172.21.15.132:0/685405222 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f747c0f51a0 msgr2=0x7f747c0f2f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:13:08.897 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:08.894+0000 7f748272c700 1 --2- 172.21.15.132:0/685405222 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f747c0f51a0 0x7f747c0f2f20 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f746c0089f0 tx=0x7f746c00b040).stop 2024-02-20T00:13:08.897 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:08.894+0000 7f748272c700 1 -- 172.21.15.132:0/685405222 shutdown_connections 2024-02-20T00:13:08.897 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:08.894+0000 7f748272c700 1 --2- 172.21.15.132:0/685405222 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f746802ff10 0x7f74680323d0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:13:08.897 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:08.894+0000 7f748272c700 1 --2- 172.21.15.132:0/685405222 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f747c0f51a0 0x7f747c0f2f20 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:13:08.897 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:08.894+0000 7f748272c700 1 -- 172.21.15.132:0/685405222 >> 172.21.15.132:0/685405222 conn(0x7f747c0f09a0 msgr2=0x7f747c05fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:13:08.897 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:08.894+0000 7f748272c700 1 -- 172.21.15.132:0/685405222 shutdown_connections 2024-02-20T00:13:08.897 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:08.894+0000 7f748272c700 1 -- 172.21.15.132:0/685405222 wait complete. 2024-02-20T00:13:08.898 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:13:09.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:13:08 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:13:08.894321+0000 mon.smithi084 (mon.0) 165 : audit [DBG] from='client.? 172.21.15.132:0/685405222' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:13:10.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:13:09 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:13:08.865429+0000 mgr.smithi084.bmqrnf (mgr.14162) 97 : cluster [DBG] pgmap v90: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:13:10.489 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:13:10.490 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:13:12.295 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:12.292+0000 7f679129e700 1 -- 172.21.15.132:0/499924254 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f678c0f5410 msgr2=0x7f678c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:13:12.295 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:12.292+0000 7f679129e700 1 --2- 172.21.15.132:0/499924254 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f678c0f5410 0x7f678c0f5830 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f6780009fb0 tx=0x7f6780009b30).stop 2024-02-20T00:13:12.295 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:12.292+0000 7f679129e700 1 -- 172.21.15.132:0/499924254 shutdown_connections 2024-02-20T00:13:12.295 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:12.292+0000 7f679129e700 1 --2- 172.21.15.132:0/499924254 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f678c0f5410 0x7f678c0f5830 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:13:12.295 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:12.292+0000 7f679129e700 1 -- 172.21.15.132:0/499924254 >> 172.21.15.132:0/499924254 conn(0x7f678c0f0b70 msgr2=0x7f678c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:13:12.295 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:12.292+0000 7f679129e700 1 -- 172.21.15.132:0/499924254 shutdown_connections 2024-02-20T00:13:12.295 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:12.292+0000 7f679129e700 1 -- 172.21.15.132:0/499924254 wait complete. 2024-02-20T00:13:12.297 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:12.293+0000 7f679129e700 1 Processor -- start 2024-02-20T00:13:12.297 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:12.293+0000 7f679129e700 1 -- start start 2024-02-20T00:13:12.297 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:12.293+0000 7f679129e700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f678c0f5410 0x7f678c1056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:13:12.297 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:12.293+0000 7f679129e700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f678c105bd0 con 0x7f678c0f5410 2024-02-20T00:13:12.297 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:12.293+0000 7f678affd700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f678c0f5410 0x7f678c1056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:13:12.298 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:12.293+0000 7f678affd700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f678c0f5410 0x7f678c1056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:35258/0 (socket says 172.21.15.132:35258) 2024-02-20T00:13:12.298 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:12.293+0000 7f678affd700 1 -- 172.21.15.132:0/941257369 learned_addr learned my addr 172.21.15.132:0/941257369 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:13:12.298 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:12.294+0000 7f678affd700 1 -- 172.21.15.132:0/941257369 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6780005040 con 0x7f678c0f5410 2024-02-20T00:13:12.298 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:12.294+0000 7f678affd700 1 --2- 172.21.15.132:0/941257369 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f678c0f5410 0x7f678c1056e0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f6780005010 tx=0x7f678000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:13:12.298 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:12.294+0000 7f67737fe700 1 -- 172.21.15.132:0/941257369 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6780009320 con 0x7f678c0f5410 2024-02-20T00:13:12.298 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:12.294+0000 7f679129e700 1 -- 172.21.15.132:0/941257369 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f678c105dd0 con 0x7f678c0f5410 2024-02-20T00:13:12.298 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:12.294+0000 7f67737fe700 1 -- 172.21.15.132:0/941257369 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6780013380 con 0x7f678c0f5410 2024-02-20T00:13:12.298 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:12.294+0000 7f67737fe700 1 -- 172.21.15.132:0/941257369 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6780014e50 con 0x7f678c0f5410 2024-02-20T00:13:12.298 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:12.294+0000 7f679129e700 1 -- 172.21.15.132:0/941257369 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f678c106260 con 0x7f678c0f5410 2024-02-20T00:13:12.298 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:12.295+0000 7f67737fe700 1 -- 172.21.15.132:0/941257369 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6780014420 con 0x7f678c0f5410 2024-02-20T00:13:12.298 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:12.295+0000 7f679129e700 1 -- 172.21.15.132:0/941257369 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f678c045ad0 con 0x7f678c0f5410 2024-02-20T00:13:12.298 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:12.295+0000 7f67737fe700 1 --2- 172.21.15.132:0/941257369 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f677402ff10 0x7f67740323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:13:12.298 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:12.295+0000 7f67737fe700 1 -- 172.21.15.132:0/941257369 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6780040f40 con 0x7f678c0f5410 2024-02-20T00:13:12.298 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:12.295+0000 7f678a7fc700 1 --2- 172.21.15.132:0/941257369 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f677402ff10 0x7f67740323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:13:12.299 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:12.296+0000 7f678a7fc700 1 --2- 172.21.15.132:0/941257369 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f677402ff10 0x7f67740323d0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7f677c0096a0 tx=0x7f677c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:13:12.301 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:12.300+0000 7f67737fe700 1 -- 172.21.15.132:0/941257369 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f67800095f0 con 0x7f678c0f5410 2024-02-20T00:13:12.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:13:11 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:13:10.865767+0000 mgr.smithi084.bmqrnf (mgr.14162) 98 : cluster [DBG] pgmap v91: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:13:12.506 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:12.504+0000 7f679129e700 1 -- 172.21.15.132:0/941257369 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f678c107110 con 0x7f678c0f5410 2024-02-20T00:13:12.507 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:12.505+0000 7f67737fe700 1 -- 172.21.15.132:0/941257369 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f678001db70 con 0x7f678c0f5410 2024-02-20T00:13:12.507 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:13:12.507 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:13:12.508 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:12.506+0000 7f679129e700 1 -- 172.21.15.132:0/941257369 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f677402ff10 msgr2=0x7f67740323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:13:12.509 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:12.506+0000 7f679129e700 1 --2- 172.21.15.132:0/941257369 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f677402ff10 0x7f67740323d0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7f677c0096a0 tx=0x7f677c006b40).stop 2024-02-20T00:13:12.509 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:12.506+0000 7f679129e700 1 -- 172.21.15.132:0/941257369 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f678c0f5410 msgr2=0x7f678c1056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:13:12.509 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:12.506+0000 7f679129e700 1 --2- 172.21.15.132:0/941257369 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f678c0f5410 0x7f678c1056e0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f6780005010 tx=0x7f678000b040).stop 2024-02-20T00:13:12.509 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:12.506+0000 7f679129e700 1 -- 172.21.15.132:0/941257369 shutdown_connections 2024-02-20T00:13:12.509 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:12.506+0000 7f679129e700 1 --2- 172.21.15.132:0/941257369 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f677402ff10 0x7f67740323d0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:13:12.509 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:12.506+0000 7f679129e700 1 --2- 172.21.15.132:0/941257369 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f678c0f5410 0x7f678c1056e0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:13:12.509 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:12.506+0000 7f679129e700 1 -- 172.21.15.132:0/941257369 >> 172.21.15.132:0/941257369 conn(0x7f678c0f0b70 msgr2=0x7f678c0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:13:12.509 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:12.506+0000 7f679129e700 1 -- 172.21.15.132:0/941257369 shutdown_connections 2024-02-20T00:13:12.509 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:12.506+0000 7f679129e700 1 -- 172.21.15.132:0/941257369 wait complete. 2024-02-20T00:13:12.509 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:13:13.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:13:12 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:13:12.506198+0000 mon.smithi084 (mon.0) 166 : audit [DBG] from='client.? 172.21.15.132:0/941257369' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:13:14.200 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:13:14.200 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:13:14.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:13:13 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:13:12.866000+0000 mgr.smithi084.bmqrnf (mgr.14162) 99 : cluster [DBG] pgmap v92: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:13:16.000 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:15.997+0000 7f2feb26e700 1 -- 172.21.15.132:0/3543672282 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2fe40f5410 msgr2=0x7f2fe40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:13:16.000 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:15.997+0000 7f2feb26e700 1 --2- 172.21.15.132:0/3543672282 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2fe40f5410 0x7f2fe40f5830 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7f2fd4009fb0 tx=0x7f2fd4009b30).stop 2024-02-20T00:13:16.000 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:15.997+0000 7f2feb26e700 1 -- 172.21.15.132:0/3543672282 shutdown_connections 2024-02-20T00:13:16.000 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:15.997+0000 7f2feb26e700 1 --2- 172.21.15.132:0/3543672282 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2fe40f5410 0x7f2fe40f5830 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:13:16.000 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:15.997+0000 7f2feb26e700 1 -- 172.21.15.132:0/3543672282 >> 172.21.15.132:0/3543672282 conn(0x7f2fe40f0b70 msgr2=0x7f2fe40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:13:16.000 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:15.998+0000 7f2feb26e700 1 -- 172.21.15.132:0/3543672282 shutdown_connections 2024-02-20T00:13:16.000 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:15.998+0000 7f2feb26e700 1 -- 172.21.15.132:0/3543672282 wait complete. 2024-02-20T00:13:16.002 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:15.998+0000 7f2feb26e700 1 Processor -- start 2024-02-20T00:13:16.002 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:15.998+0000 7f2feb26e700 1 -- start start 2024-02-20T00:13:16.002 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:15.998+0000 7f2feb26e700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2fe40f5410 0x7f2fe41056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:13:16.002 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:15.999+0000 7f2feb26e700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fe4105bd0 con 0x7f2fe40f5410 2024-02-20T00:13:16.002 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:15.999+0000 7f2fe900a700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2fe40f5410 0x7f2fe41056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:13:16.002 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:15.999+0000 7f2fe900a700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2fe40f5410 0x7f2fe41056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:35282/0 (socket says 172.21.15.132:35282) 2024-02-20T00:13:16.002 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:15.999+0000 7f2fe900a700 1 -- 172.21.15.132:0/3947430191 learned_addr learned my addr 172.21.15.132:0/3947430191 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:13:16.002 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:15.999+0000 7f2fe900a700 1 -- 172.21.15.132:0/3947430191 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fd4005040 con 0x7f2fe40f5410 2024-02-20T00:13:16.002 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:15.999+0000 7f2fe900a700 1 --2- 172.21.15.132:0/3947430191 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2fe40f5410 0x7f2fe41056e0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f2fd4005010 tx=0x7f2fd400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:13:16.002 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:15.999+0000 7f2fd9ffb700 1 -- 172.21.15.132:0/3947430191 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2fd4009320 con 0x7f2fe40f5410 2024-02-20T00:13:16.002 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:15.999+0000 7f2feb26e700 1 -- 172.21.15.132:0/3947430191 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fe4105dd0 con 0x7f2fe40f5410 2024-02-20T00:13:16.002 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:15.999+0000 7f2fd9ffb700 1 -- 172.21.15.132:0/3947430191 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2fd4013380 con 0x7f2fe40f5410 2024-02-20T00:13:16.003 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:15.999+0000 7f2fd9ffb700 1 -- 172.21.15.132:0/3947430191 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2fd4014e50 con 0x7f2fe40f5410 2024-02-20T00:13:16.003 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:15.999+0000 7f2feb26e700 1 -- 172.21.15.132:0/3947430191 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fe4106260 con 0x7f2fe40f5410 2024-02-20T00:13:16.003 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:16.000+0000 7f2fd9ffb700 1 -- 172.21.15.132:0/3947430191 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2fd40095f0 con 0x7f2fe40f5410 2024-02-20T00:13:16.004 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:16.000+0000 7f2feb26e700 1 -- 172.21.15.132:0/3947430191 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fe4045ad0 con 0x7f2fe40f5410 2024-02-20T00:13:16.004 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:16.000+0000 7f2fd9ffb700 1 --2- 172.21.15.132:0/3947430191 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f2fd002ff10 0x7f2fd00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:13:16.004 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:16.000+0000 7f2fd9ffb700 1 -- 172.21.15.132:0/3947430191 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2fd4040ef0 con 0x7f2fe40f5410 2024-02-20T00:13:16.004 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:16.001+0000 7f2fe8809700 1 --2- 172.21.15.132:0/3947430191 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f2fd002ff10 0x7f2fd00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:13:16.004 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:16.001+0000 7f2fe8809700 1 --2- 172.21.15.132:0/3947430191 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f2fd002ff10 0x7f2fd00323d0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7f2fe00096a0 tx=0x7f2fe0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:13:16.007 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:16.005+0000 7f2fd9ffb700 1 -- 172.21.15.132:0/3947430191 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2fd4014420 con 0x7f2fe40f5410 2024-02-20T00:13:16.212 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:16.210+0000 7f2feb26e700 1 -- 172.21.15.132:0/3947430191 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2fe4107110 con 0x7f2fe40f5410 2024-02-20T00:13:16.213 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:16.211+0000 7f2fd9ffb700 1 -- 172.21.15.132:0/3947430191 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2fd4010230 con 0x7f2fe40f5410 2024-02-20T00:13:16.213 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:13:16.213 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:13:16.215 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:16.213+0000 7f2feb26e700 1 -- 172.21.15.132:0/3947430191 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f2fd002ff10 msgr2=0x7f2fd00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:13:16.215 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:16.213+0000 7f2feb26e700 1 --2- 172.21.15.132:0/3947430191 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f2fd002ff10 0x7f2fd00323d0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7f2fe00096a0 tx=0x7f2fe0006b40).stop 2024-02-20T00:13:16.215 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:16.213+0000 7f2feb26e700 1 -- 172.21.15.132:0/3947430191 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2fe40f5410 msgr2=0x7f2fe41056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:13:16.215 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:16.213+0000 7f2feb26e700 1 --2- 172.21.15.132:0/3947430191 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2fe40f5410 0x7f2fe41056e0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f2fd4005010 tx=0x7f2fd400b040).stop 2024-02-20T00:13:16.215 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:16.213+0000 7f2feb26e700 1 -- 172.21.15.132:0/3947430191 shutdown_connections 2024-02-20T00:13:16.215 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:16.213+0000 7f2feb26e700 1 --2- 172.21.15.132:0/3947430191 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f2fd002ff10 0x7f2fd00323d0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:13:16.215 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:16.213+0000 7f2feb26e700 1 --2- 172.21.15.132:0/3947430191 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2fe40f5410 0x7f2fe41056e0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:13:16.215 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:16.213+0000 7f2feb26e700 1 -- 172.21.15.132:0/3947430191 >> 172.21.15.132:0/3947430191 conn(0x7f2fe40f0b70 msgr2=0x7f2fe40f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:13:16.216 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:16.213+0000 7f2feb26e700 1 -- 172.21.15.132:0/3947430191 shutdown_connections 2024-02-20T00:13:16.216 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:16.213+0000 7f2feb26e700 1 -- 172.21.15.132:0/3947430191 wait complete. 2024-02-20T00:13:16.216 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:13:16.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:13:15 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:13:14.866329+0000 mgr.smithi084.bmqrnf (mgr.14162) 100 : cluster [DBG] pgmap v93: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:13:17.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:13:16 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:13:16.212573+0000 mon.smithi084 (mon.0) 167 : audit [DBG] from='client.? 172.21.15.132:0/3947430191' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:13:17.937 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:13:17.937 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:13:18.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:13:17 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:13:16.866729+0000 mgr.smithi084.bmqrnf (mgr.14162) 101 : cluster [DBG] pgmap v94: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:13:19.599 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:19.596+0000 7f414586b700 1 -- 172.21.15.132:0/1241642451 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4140060160 msgr2=0x7f41400f8fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:13:19.600 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:19.596+0000 7f414586b700 1 --2- 172.21.15.132:0/1241642451 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4140060160 0x7f41400f8fa0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7f4134009fb0 tx=0x7f4134009b30).stop 2024-02-20T00:13:19.600 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:19.596+0000 7f414586b700 1 -- 172.21.15.132:0/1241642451 shutdown_connections 2024-02-20T00:13:19.600 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:19.596+0000 7f414586b700 1 --2- 172.21.15.132:0/1241642451 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4140060160 0x7f41400f8fa0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:13:19.600 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:19.596+0000 7f414586b700 1 -- 172.21.15.132:0/1241642451 >> 172.21.15.132:0/1241642451 conn(0x7f41400f09a0 msgr2=0x7f41400f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:13:19.600 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:19.596+0000 7f414586b700 1 -- 172.21.15.132:0/1241642451 shutdown_connections 2024-02-20T00:13:19.600 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:19.596+0000 7f414586b700 1 -- 172.21.15.132:0/1241642451 wait complete. 2024-02-20T00:13:19.600 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:19.596+0000 7f414586b700 1 Processor -- start 2024-02-20T00:13:19.600 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:19.596+0000 7f414586b700 1 -- start start 2024-02-20T00:13:19.600 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:19.597+0000 7f414586b700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4140060160 0x7f41401065d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:13:19.600 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:19.597+0000 7f414586b700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4140106ac0 con 0x7f4140060160 2024-02-20T00:13:19.600 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:19.597+0000 7f4144869700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4140060160 0x7f41401065d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:13:19.600 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:19.597+0000 7f4144869700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4140060160 0x7f41401065d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:38930/0 (socket says 172.21.15.132:38930) 2024-02-20T00:13:19.601 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:19.597+0000 7f4144869700 1 -- 172.21.15.132:0/1692887490 learned_addr learned my addr 172.21.15.132:0/1692887490 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:13:19.601 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:19.597+0000 7f4144869700 1 -- 172.21.15.132:0/1692887490 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4134005040 con 0x7f4140060160 2024-02-20T00:13:19.601 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:19.598+0000 7f4144869700 1 --2- 172.21.15.132:0/1692887490 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4140060160 0x7f41401065d0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f4134005010 tx=0x7f413400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:13:19.602 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:19.598+0000 7f413d7fa700 1 -- 172.21.15.132:0/1692887490 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4134009320 con 0x7f4140060160 2024-02-20T00:13:19.602 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:19.598+0000 7f414586b700 1 -- 172.21.15.132:0/1692887490 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4140106cc0 con 0x7f4140060160 2024-02-20T00:13:19.602 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:19.598+0000 7f413d7fa700 1 -- 172.21.15.132:0/1692887490 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4134013380 con 0x7f4140060160 2024-02-20T00:13:19.602 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:19.598+0000 7f413d7fa700 1 -- 172.21.15.132:0/1692887490 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4134014e50 con 0x7f4140060160 2024-02-20T00:13:19.602 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:19.598+0000 7f414586b700 1 -- 172.21.15.132:0/1692887490 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4140107150 con 0x7f4140060160 2024-02-20T00:13:19.602 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:19.599+0000 7f413d7fa700 1 -- 172.21.15.132:0/1692887490 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f41340095f0 con 0x7f4140060160 2024-02-20T00:13:19.602 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:19.599+0000 7f414586b700 1 -- 172.21.15.132:0/1692887490 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4140045ad0 con 0x7f4140060160 2024-02-20T00:13:19.602 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:19.599+0000 7f413d7fa700 1 --2- 172.21.15.132:0/1692887490 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f413002ff10 0x7f41300323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:13:19.602 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:19.599+0000 7f413d7fa700 1 -- 172.21.15.132:0/1692887490 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4134040fa0 con 0x7f4140060160 2024-02-20T00:13:19.602 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:19.599+0000 7f413ffff700 1 --2- 172.21.15.132:0/1692887490 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f413002ff10 0x7f41300323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:13:19.602 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:19.599+0000 7f413ffff700 1 --2- 172.21.15.132:0/1692887490 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f413002ff10 0x7f41300323d0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7f412c0096a0 tx=0x7f412c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:13:19.605 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:19.603+0000 7f413d7fa700 1 -- 172.21.15.132:0/1692887490 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f41340263a0 con 0x7f4140060160 2024-02-20T00:13:19.811 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:19.808+0000 7f414586b700 1 -- 172.21.15.132:0/1692887490 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4140108000 con 0x7f4140060160 2024-02-20T00:13:19.811 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:19.808+0000 7f413d7fa700 1 -- 172.21.15.132:0/1692887490 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4134010230 con 0x7f4140060160 2024-02-20T00:13:19.811 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:13:19.811 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:13:19.812 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:19.810+0000 7f414586b700 1 -- 172.21.15.132:0/1692887490 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f413002ff10 msgr2=0x7f41300323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:13:19.812 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:19.810+0000 7f414586b700 1 --2- 172.21.15.132:0/1692887490 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f413002ff10 0x7f41300323d0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7f412c0096a0 tx=0x7f412c006b40).stop 2024-02-20T00:13:19.812 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:19.810+0000 7f414586b700 1 -- 172.21.15.132:0/1692887490 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4140060160 msgr2=0x7f41401065d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:13:19.812 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:19.810+0000 7f414586b700 1 --2- 172.21.15.132:0/1692887490 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4140060160 0x7f41401065d0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f4134005010 tx=0x7f413400b040).stop 2024-02-20T00:13:19.813 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:19.810+0000 7f414586b700 1 -- 172.21.15.132:0/1692887490 shutdown_connections 2024-02-20T00:13:19.813 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:19.810+0000 7f414586b700 1 --2- 172.21.15.132:0/1692887490 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f413002ff10 0x7f41300323d0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:13:19.813 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:19.810+0000 7f414586b700 1 --2- 172.21.15.132:0/1692887490 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4140060160 0x7f41401065d0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:13:19.813 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:19.810+0000 7f414586b700 1 -- 172.21.15.132:0/1692887490 >> 172.21.15.132:0/1692887490 conn(0x7f41400f09a0 msgr2=0x7f41400f1430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:13:19.813 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:19.810+0000 7f414586b700 1 -- 172.21.15.132:0/1692887490 shutdown_connections 2024-02-20T00:13:19.813 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:19.810+0000 7f414586b700 1 -- 172.21.15.132:0/1692887490 wait complete. 2024-02-20T00:13:19.813 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:13:20.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:13:19 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:13:18.866997+0000 mgr.smithi084.bmqrnf (mgr.14162) 102 : cluster [DBG] pgmap v95: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:13:20.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:13:19 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:13:19.809933+0000 mon.smithi084 (mon.0) 168 : audit [DBG] from='client.? 172.21.15.132:0/1692887490' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:13:21.485 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:13:21.485 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:13:22.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:13:21 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:13:20.867271+0000 mgr.smithi084.bmqrnf (mgr.14162) 103 : cluster [DBG] pgmap v96: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:13:23.295 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:23.292+0000 7f6cd759e700 1 -- 172.21.15.132:0/1291579805 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6cd80f51a0 msgr2=0x7f6cd80f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:13:23.295 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:23.292+0000 7f6cd759e700 1 --2- 172.21.15.132:0/1291579805 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6cd80f51a0 0x7f6cd80f55c0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7f6cc0009fb0 tx=0x7f6cc0009b30).stop 2024-02-20T00:13:23.295 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:23.292+0000 7f6cd759e700 1 -- 172.21.15.132:0/1291579805 shutdown_connections 2024-02-20T00:13:23.295 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:23.292+0000 7f6cd759e700 1 --2- 172.21.15.132:0/1291579805 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6cd80f51a0 0x7f6cd80f55c0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:13:23.295 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:23.292+0000 7f6cd759e700 1 -- 172.21.15.132:0/1291579805 >> 172.21.15.132:0/1291579805 conn(0x7f6cd80f09a0 msgr2=0x7f6cd80f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:13:23.296 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:23.292+0000 7f6cd759e700 1 -- 172.21.15.132:0/1291579805 shutdown_connections 2024-02-20T00:13:23.296 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:23.292+0000 7f6cd759e700 1 -- 172.21.15.132:0/1291579805 wait complete. 2024-02-20T00:13:23.296 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:23.292+0000 7f6cd759e700 1 Processor -- start 2024-02-20T00:13:23.296 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:23.292+0000 7f6cd759e700 1 -- start start 2024-02-20T00:13:23.296 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:23.292+0000 7f6cd759e700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6cd80f51a0 0x7f6cd8108820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:13:23.296 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:23.292+0000 7f6cd759e700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cd8108d10 con 0x7f6cd80f51a0 2024-02-20T00:13:23.296 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:23.293+0000 7f6cd659c700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6cd80f51a0 0x7f6cd8108820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:13:23.296 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:23.293+0000 7f6cd659c700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6cd80f51a0 0x7f6cd8108820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:38952/0 (socket says 172.21.15.132:38952) 2024-02-20T00:13:23.296 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:23.293+0000 7f6cd659c700 1 -- 172.21.15.132:0/318808784 learned_addr learned my addr 172.21.15.132:0/318808784 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:13:23.296 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:23.293+0000 7f6cd659c700 1 -- 172.21.15.132:0/318808784 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cc0005040 con 0x7f6cd80f51a0 2024-02-20T00:13:23.296 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:23.293+0000 7f6cd659c700 1 --2- 172.21.15.132:0/318808784 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6cd80f51a0 0x7f6cd8108820 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f6cc00089f0 tx=0x7f6cc000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:13:23.297 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:23.293+0000 7f6cceffd700 1 -- 172.21.15.132:0/318808784 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6cc0009320 con 0x7f6cd80f51a0 2024-02-20T00:13:23.298 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:23.293+0000 7f6cd759e700 1 -- 172.21.15.132:0/318808784 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6cd8108f10 con 0x7f6cd80f51a0 2024-02-20T00:13:23.298 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:23.293+0000 7f6cceffd700 1 -- 172.21.15.132:0/318808784 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6cc0013380 con 0x7f6cd80f51a0 2024-02-20T00:13:23.298 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:23.293+0000 7f6cceffd700 1 -- 172.21.15.132:0/318808784 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6cc001ecb0 con 0x7f6cd80f51a0 2024-02-20T00:13:23.298 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:23.293+0000 7f6cd759e700 1 -- 172.21.15.132:0/318808784 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cd81093a0 con 0x7f6cd80f51a0 2024-02-20T00:13:23.298 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:23.294+0000 7f6cceffd700 1 -- 172.21.15.132:0/318808784 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6cc00095f0 con 0x7f6cd80f51a0 2024-02-20T00:13:23.298 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:23.294+0000 7f6cd759e700 1 -- 172.21.15.132:0/318808784 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6cd8045ad0 con 0x7f6cd80f51a0 2024-02-20T00:13:23.298 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:23.294+0000 7f6cceffd700 1 --2- 172.21.15.132:0/318808784 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f6cc402ffb0 0x7f6cc4032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:13:23.298 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:23.294+0000 7f6cceffd700 1 -- 172.21.15.132:0/318808784 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6cc0040ff0 con 0x7f6cd80f51a0 2024-02-20T00:13:23.298 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:23.296+0000 7f6cd5d9b700 1 --2- 172.21.15.132:0/318808784 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f6cc402ffb0 0x7f6cc4032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:13:23.300 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:23.298+0000 7f6cd5d9b700 1 --2- 172.21.15.132:0/318808784 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f6cc402ffb0 0x7f6cc4032470 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f6cc80096a0 tx=0x7f6cc8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:13:23.301 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:23.299+0000 7f6cceffd700 1 -- 172.21.15.132:0/318808784 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6cc006d2b0 con 0x7f6cd80f51a0 2024-02-20T00:13:23.507 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:23.505+0000 7f6cd759e700 1 -- 172.21.15.132:0/318808784 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6cd80fad30 con 0x7f6cd80f51a0 2024-02-20T00:13:23.507 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:23.505+0000 7f6cceffd700 1 -- 172.21.15.132:0/318808784 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6cc001e420 con 0x7f6cd80f51a0 2024-02-20T00:13:23.508 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:13:23.508 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:13:23.509 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:23.507+0000 7f6cd759e700 1 -- 172.21.15.132:0/318808784 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f6cc402ffb0 msgr2=0x7f6cc4032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:13:23.509 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:23.507+0000 7f6cd759e700 1 --2- 172.21.15.132:0/318808784 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f6cc402ffb0 0x7f6cc4032470 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f6cc80096a0 tx=0x7f6cc8006b40).stop 2024-02-20T00:13:23.509 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:23.507+0000 7f6cd759e700 1 -- 172.21.15.132:0/318808784 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6cd80f51a0 msgr2=0x7f6cd8108820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:13:23.510 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:23.507+0000 7f6cd759e700 1 --2- 172.21.15.132:0/318808784 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6cd80f51a0 0x7f6cd8108820 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f6cc00089f0 tx=0x7f6cc000b040).stop 2024-02-20T00:13:23.510 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:23.507+0000 7f6cd759e700 1 -- 172.21.15.132:0/318808784 shutdown_connections 2024-02-20T00:13:23.510 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:23.507+0000 7f6cd759e700 1 --2- 172.21.15.132:0/318808784 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f6cc402ffb0 0x7f6cc4032470 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:13:23.510 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:23.507+0000 7f6cd759e700 1 --2- 172.21.15.132:0/318808784 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6cd80f51a0 0x7f6cd8108820 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:13:23.510 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:23.507+0000 7f6cd759e700 1 -- 172.21.15.132:0/318808784 >> 172.21.15.132:0/318808784 conn(0x7f6cd80f09a0 msgr2=0x7f6cd805fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:13:23.510 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:23.507+0000 7f6cd759e700 1 -- 172.21.15.132:0/318808784 shutdown_connections 2024-02-20T00:13:23.510 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:23.507+0000 7f6cd759e700 1 -- 172.21.15.132:0/318808784 wait complete. 2024-02-20T00:13:23.510 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:13:24.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:13:23 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:13:22.867500+0000 mgr.smithi084.bmqrnf (mgr.14162) 104 : cluster [DBG] pgmap v97: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:13:24.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:13:23 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:13:23.506995+0000 mon.smithi084 (mon.0) 169 : audit [DBG] from='client.? 172.21.15.132:0/318808784' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:13:25.100 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:13:25.100 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:13:26.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:13:25 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:13:24.867835+0000 mgr.smithi084.bmqrnf (mgr.14162) 105 : cluster [DBG] pgmap v98: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:13:26.845 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:26.841+0000 7fe93e079700 1 -- 172.21.15.132:0/3300274690 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe9380f6510 msgr2=0x7fe9380f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:13:26.845 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:26.841+0000 7fe93e079700 1 --2- 172.21.15.132:0/3300274690 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe9380f6510 0x7fe9380f6930 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7fe92c009fb0 tx=0x7fe92c009b30).stop 2024-02-20T00:13:26.845 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:26.841+0000 7fe93e079700 1 -- 172.21.15.132:0/3300274690 shutdown_connections 2024-02-20T00:13:26.845 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:26.841+0000 7fe93e079700 1 --2- 172.21.15.132:0/3300274690 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe9380f6510 0x7fe9380f6930 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:13:26.845 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:26.841+0000 7fe93e079700 1 -- 172.21.15.132:0/3300274690 >> 172.21.15.132:0/3300274690 conn(0x7fe9380f1c70 msgr2=0x7fe9380f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:13:26.845 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:26.842+0000 7fe93e079700 1 -- 172.21.15.132:0/3300274690 shutdown_connections 2024-02-20T00:13:26.845 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:26.842+0000 7fe93e079700 1 -- 172.21.15.132:0/3300274690 wait complete. 2024-02-20T00:13:26.845 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:26.842+0000 7fe93e079700 1 Processor -- start 2024-02-20T00:13:26.845 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:26.842+0000 7fe93e079700 1 -- start start 2024-02-20T00:13:26.845 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:26.842+0000 7fe93e079700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe9380f6510 0x7fe9381067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:13:26.845 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:26.842+0000 7fe93e079700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe938106cd0 con 0x7fe9380f6510 2024-02-20T00:13:26.845 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:26.843+0000 7fe9377fe700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe9380f6510 0x7fe9381067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:13:26.846 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:26.843+0000 7fe9377fe700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe9380f6510 0x7fe9381067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:38974/0 (socket says 172.21.15.132:38974) 2024-02-20T00:13:26.846 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:26.843+0000 7fe9377fe700 1 -- 172.21.15.132:0/2534731115 learned_addr learned my addr 172.21.15.132:0/2534731115 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:13:26.846 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:26.843+0000 7fe9377fe700 1 -- 172.21.15.132:0/2534731115 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe92c005040 con 0x7fe9380f6510 2024-02-20T00:13:26.846 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:26.843+0000 7fe9377fe700 1 --2- 172.21.15.132:0/2534731115 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe9380f6510 0x7fe9381067e0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7fe92c0089f0 tx=0x7fe92c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:13:26.846 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:26.843+0000 7fe91ffff700 1 -- 172.21.15.132:0/2534731115 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe92c009320 con 0x7fe9380f6510 2024-02-20T00:13:26.846 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:26.843+0000 7fe93e079700 1 -- 172.21.15.132:0/2534731115 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe938106ed0 con 0x7fe9380f6510 2024-02-20T00:13:26.846 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:26.843+0000 7fe91ffff700 1 -- 172.21.15.132:0/2534731115 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe92c013380 con 0x7fe9380f6510 2024-02-20T00:13:26.846 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:26.843+0000 7fe91ffff700 1 -- 172.21.15.132:0/2534731115 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe92c01ed80 con 0x7fe9380f6510 2024-02-20T00:13:26.846 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:26.843+0000 7fe93e079700 1 -- 172.21.15.132:0/2534731115 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe938107360 con 0x7fe9380f6510 2024-02-20T00:13:26.847 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:26.844+0000 7fe91ffff700 1 -- 172.21.15.132:0/2534731115 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe92c01e420 con 0x7fe9380f6510 2024-02-20T00:13:26.847 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:26.844+0000 7fe93e079700 1 -- 172.21.15.132:0/2534731115 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe938045ad0 con 0x7fe9380f6510 2024-02-20T00:13:26.847 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:26.844+0000 7fe91ffff700 1 --2- 172.21.15.132:0/2534731115 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fe92002ff60 0x7fe920032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:13:26.847 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:26.844+0000 7fe91ffff700 1 -- 172.21.15.132:0/2534731115 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe92c041110 con 0x7fe9380f6510 2024-02-20T00:13:26.847 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:26.844+0000 7fe936ffd700 1 --2- 172.21.15.132:0/2534731115 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fe92002ff60 0x7fe920032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:13:26.847 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:26.845+0000 7fe936ffd700 1 --2- 172.21.15.132:0/2534731115 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fe92002ff60 0x7fe920032420 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7fe9280096a0 tx=0x7fe928006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:13:26.850 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:26.849+0000 7fe91ffff700 1 -- 172.21.15.132:0/2534731115 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe92c01cb10 con 0x7fe9380f6510 2024-02-20T00:13:27.055 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:27.053+0000 7fe93e079700 1 -- 172.21.15.132:0/2534731115 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe938108210 con 0x7fe9380f6510 2024-02-20T00:13:27.055 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:27.054+0000 7fe91ffff700 1 -- 172.21.15.132:0/2534731115 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe92c004490 con 0x7fe9380f6510 2024-02-20T00:13:27.055 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:13:27.055 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:13:27.057 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:27.056+0000 7fe93e079700 1 -- 172.21.15.132:0/2534731115 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fe92002ff60 msgr2=0x7fe920032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:13:27.057 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:27.056+0000 7fe93e079700 1 --2- 172.21.15.132:0/2534731115 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fe92002ff60 0x7fe920032420 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7fe9280096a0 tx=0x7fe928006b40).stop 2024-02-20T00:13:27.057 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:27.056+0000 7fe93e079700 1 -- 172.21.15.132:0/2534731115 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe9380f6510 msgr2=0x7fe9381067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:13:27.057 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:27.056+0000 7fe93e079700 1 --2- 172.21.15.132:0/2534731115 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe9380f6510 0x7fe9381067e0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7fe92c0089f0 tx=0x7fe92c00b040).stop 2024-02-20T00:13:27.057 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:27.056+0000 7fe93e079700 1 -- 172.21.15.132:0/2534731115 shutdown_connections 2024-02-20T00:13:27.057 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:27.056+0000 7fe93e079700 1 --2- 172.21.15.132:0/2534731115 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fe92002ff60 0x7fe920032420 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:13:27.058 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:27.056+0000 7fe93e079700 1 --2- 172.21.15.132:0/2534731115 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe9380f6510 0x7fe9381067e0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:13:27.058 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:27.056+0000 7fe93e079700 1 -- 172.21.15.132:0/2534731115 >> 172.21.15.132:0/2534731115 conn(0x7fe9380f1c70 msgr2=0x7fe9380f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:13:27.058 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:27.056+0000 7fe93e079700 1 -- 172.21.15.132:0/2534731115 shutdown_connections 2024-02-20T00:13:27.058 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:27.056+0000 7fe93e079700 1 -- 172.21.15.132:0/2534731115 wait complete. 2024-02-20T00:13:27.058 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:13:28.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:13:27 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:13:26.868162+0000 mgr.smithi084.bmqrnf (mgr.14162) 106 : cluster [DBG] pgmap v99: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:13:28.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:13:27 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:13:27.054708+0000 mon.smithi084 (mon.0) 170 : audit [DBG] from='client.? 172.21.15.132:0/2534731115' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:13:28.652 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:13:28.652 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:13:30.356 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:30.353+0000 7fe9ac24f700 1 -- 172.21.15.132:0/3973742683 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe9a40f5410 msgr2=0x7fe9a40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:13:30.356 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:30.353+0000 7fe9ac24f700 1 --2- 172.21.15.132:0/3973742683 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe9a40f5410 0x7fe9a40f5830 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7fe9a0009fb0 tx=0x7fe9a0009b30).stop 2024-02-20T00:13:30.356 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:30.353+0000 7fe9ac24f700 1 -- 172.21.15.132:0/3973742683 shutdown_connections 2024-02-20T00:13:30.356 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:30.353+0000 7fe9ac24f700 1 --2- 172.21.15.132:0/3973742683 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe9a40f5410 0x7fe9a40f5830 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:13:30.356 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:30.353+0000 7fe9ac24f700 1 -- 172.21.15.132:0/3973742683 >> 172.21.15.132:0/3973742683 conn(0x7fe9a40f0b70 msgr2=0x7fe9a40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:13:30.357 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:30.353+0000 7fe9ac24f700 1 -- 172.21.15.132:0/3973742683 shutdown_connections 2024-02-20T00:13:30.357 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:30.353+0000 7fe9ac24f700 1 -- 172.21.15.132:0/3973742683 wait complete. 2024-02-20T00:13:30.357 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:30.353+0000 7fe9ac24f700 1 Processor -- start 2024-02-20T00:13:30.357 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:30.353+0000 7fe9ac24f700 1 -- start start 2024-02-20T00:13:30.357 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:30.354+0000 7fe9ac24f700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe9a40f5410 0x7fe9a4109500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:13:30.357 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:30.354+0000 7fe9ac24f700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9a4060220 con 0x7fe9a40f5410 2024-02-20T00:13:30.357 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:30.354+0000 7fe9a9feb700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe9a40f5410 0x7fe9a4109500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:13:30.357 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:30.354+0000 7fe9a9feb700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe9a40f5410 0x7fe9a4109500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:42256/0 (socket says 172.21.15.132:42256) 2024-02-20T00:13:30.357 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:30.354+0000 7fe9a9feb700 1 -- 172.21.15.132:0/469623824 learned_addr learned my addr 172.21.15.132:0/469623824 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:13:30.357 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:30.354+0000 7fe9a9feb700 1 -- 172.21.15.132:0/469623824 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9a0005040 con 0x7fe9a40f5410 2024-02-20T00:13:30.358 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:30.354+0000 7fe9a9feb700 1 --2- 172.21.15.132:0/469623824 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe9a40f5410 0x7fe9a4109500 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7fe9a0005010 tx=0x7fe9a000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:13:30.358 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:30.354+0000 7fe99a7fc700 1 -- 172.21.15.132:0/469623824 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe9a0009320 con 0x7fe9a40f5410 2024-02-20T00:13:30.358 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:30.354+0000 7fe9ac24f700 1 -- 172.21.15.132:0/469623824 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9a4060420 con 0x7fe9a40f5410 2024-02-20T00:13:30.358 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:30.354+0000 7fe99a7fc700 1 -- 172.21.15.132:0/469623824 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe9a0013380 con 0x7fe9a40f5410 2024-02-20T00:13:30.358 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:30.354+0000 7fe99a7fc700 1 -- 172.21.15.132:0/469623824 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe9a0014e50 con 0x7fe9a40f5410 2024-02-20T00:13:30.358 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:30.355+0000 7fe9ac24f700 1 -- 172.21.15.132:0/469623824 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9a4109bb0 con 0x7fe9a40f5410 2024-02-20T00:13:30.359 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:13:29 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:13:28.868431+0000 mgr.smithi084.bmqrnf (mgr.14162) 107 : cluster [DBG] pgmap v100: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:13:30.359 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:30.355+0000 7fe99a7fc700 1 -- 172.21.15.132:0/469623824 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe9a0014420 con 0x7fe9a40f5410 2024-02-20T00:13:30.360 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:30.355+0000 7fe99a7fc700 1 --2- 172.21.15.132:0/469623824 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fe99002ff10 0x7fe9900323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:13:30.360 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:30.355+0000 7fe99a7fc700 1 -- 172.21.15.132:0/469623824 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe9a0040e30 con 0x7fe9a40f5410 2024-02-20T00:13:30.360 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:30.355+0000 7fe9ac24f700 1 -- 172.21.15.132:0/469623824 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9a4059100 con 0x7fe9a40f5410 2024-02-20T00:13:30.360 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:30.356+0000 7fe9a97ea700 1 --2- 172.21.15.132:0/469623824 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fe99002ff10 0x7fe9900323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:13:30.360 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:30.356+0000 7fe9a97ea700 1 --2- 172.21.15.132:0/469623824 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fe99002ff10 0x7fe9900323d0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7fe9940096a0 tx=0x7fe994006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:13:30.361 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:30.360+0000 7fe99a7fc700 1 -- 172.21.15.132:0/469623824 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe9a00095f0 con 0x7fe9a40f5410 2024-02-20T00:13:30.565 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:30.564+0000 7fe9ac24f700 1 -- 172.21.15.132:0/469623824 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe9a410a2a0 con 0x7fe9a40f5410 2024-02-20T00:13:30.565 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:30.564+0000 7fe99a7fc700 1 -- 172.21.15.132:0/469623824 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe9a002b020 con 0x7fe9a40f5410 2024-02-20T00:13:30.566 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:13:30.566 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:13:30.567 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:30.566+0000 7fe9ac24f700 1 -- 172.21.15.132:0/469623824 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fe99002ff10 msgr2=0x7fe9900323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:13:30.567 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:30.566+0000 7fe9ac24f700 1 --2- 172.21.15.132:0/469623824 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fe99002ff10 0x7fe9900323d0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7fe9940096a0 tx=0x7fe994006b40).stop 2024-02-20T00:13:30.567 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:30.566+0000 7fe9ac24f700 1 -- 172.21.15.132:0/469623824 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe9a40f5410 msgr2=0x7fe9a4109500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:13:30.567 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:30.566+0000 7fe9ac24f700 1 --2- 172.21.15.132:0/469623824 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe9a40f5410 0x7fe9a4109500 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7fe9a0005010 tx=0x7fe9a000b040).stop 2024-02-20T00:13:30.568 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:30.566+0000 7fe9ac24f700 1 -- 172.21.15.132:0/469623824 shutdown_connections 2024-02-20T00:13:30.568 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:30.566+0000 7fe9ac24f700 1 --2- 172.21.15.132:0/469623824 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fe99002ff10 0x7fe9900323d0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:13:30.568 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:30.566+0000 7fe9ac24f700 1 --2- 172.21.15.132:0/469623824 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe9a40f5410 0x7fe9a4109500 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:13:30.568 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:30.566+0000 7fe9ac24f700 1 -- 172.21.15.132:0/469623824 >> 172.21.15.132:0/469623824 conn(0x7fe9a40f0b70 msgr2=0x7fe9a40f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:13:30.568 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:30.566+0000 7fe9ac24f700 1 -- 172.21.15.132:0/469623824 shutdown_connections 2024-02-20T00:13:30.568 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:30.566+0000 7fe9ac24f700 1 -- 172.21.15.132:0/469623824 wait complete. 2024-02-20T00:13:30.568 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:13:31.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:13:30 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:13:30.564864+0000 mon.smithi084 (mon.0) 171 : audit [DBG] from='client.? 172.21.15.132:0/469623824' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:13:32.132 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:13:32.132 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:13:32.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:13:31 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:13:30.868809+0000 mgr.smithi084.bmqrnf (mgr.14162) 108 : cluster [DBG] pgmap v101: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:13:33.855 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:33.852+0000 7fe51fc13700 1 -- 172.21.15.132:0/1650531762 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe5180f47c0 msgr2=0x7fe5180f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:13:33.855 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:33.852+0000 7fe51fc13700 1 --2- 172.21.15.132:0/1650531762 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe5180f47c0 0x7fe5180f4be0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7fe508009fb0 tx=0x7fe508009b30).stop 2024-02-20T00:13:33.856 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:33.852+0000 7fe51fc13700 1 -- 172.21.15.132:0/1650531762 shutdown_connections 2024-02-20T00:13:33.856 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:33.852+0000 7fe51fc13700 1 --2- 172.21.15.132:0/1650531762 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe5180f47c0 0x7fe5180f4be0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:13:33.856 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:33.852+0000 7fe51fc13700 1 -- 172.21.15.132:0/1650531762 >> 172.21.15.132:0/1650531762 conn(0x7fe5180f02e0 msgr2=0x7fe5180f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:13:33.856 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:33.853+0000 7fe51fc13700 1 -- 172.21.15.132:0/1650531762 shutdown_connections 2024-02-20T00:13:33.856 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:33.853+0000 7fe51fc13700 1 -- 172.21.15.132:0/1650531762 wait complete. 2024-02-20T00:13:33.856 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:33.853+0000 7fe51fc13700 1 Processor -- start 2024-02-20T00:13:33.856 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:33.853+0000 7fe51fc13700 1 -- start start 2024-02-20T00:13:33.856 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:33.853+0000 7fe51fc13700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe5180f47c0 0x7fe5180f7370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:13:33.856 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:33.853+0000 7fe51fc13700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5180f57f0 con 0x7fe5180f47c0 2024-02-20T00:13:33.856 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:33.853+0000 7fe51d9af700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe5180f47c0 0x7fe5180f7370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:13:33.856 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:33.853+0000 7fe51d9af700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe5180f47c0 0x7fe5180f7370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:42260/0 (socket says 172.21.15.132:42260) 2024-02-20T00:13:33.856 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:33.853+0000 7fe51d9af700 1 -- 172.21.15.132:0/2808769580 learned_addr learned my addr 172.21.15.132:0/2808769580 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:13:33.857 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:33.854+0000 7fe51d9af700 1 -- 172.21.15.132:0/2808769580 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe508005040 con 0x7fe5180f47c0 2024-02-20T00:13:33.857 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:33.854+0000 7fe51d9af700 1 --2- 172.21.15.132:0/2808769580 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe5180f47c0 0x7fe5180f7370 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7fe508005010 tx=0x7fe50800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:13:33.857 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:33.854+0000 7fe50e7fc700 1 -- 172.21.15.132:0/2808769580 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe508009320 con 0x7fe5180f47c0 2024-02-20T00:13:33.857 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:33.854+0000 7fe51fc13700 1 -- 172.21.15.132:0/2808769580 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5180f5990 con 0x7fe5180f47c0 2024-02-20T00:13:33.857 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:33.854+0000 7fe50e7fc700 1 -- 172.21.15.132:0/2808769580 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe508013380 con 0x7fe5180f47c0 2024-02-20T00:13:33.857 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:33.854+0000 7fe50e7fc700 1 -- 172.21.15.132:0/2808769580 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe508014e50 con 0x7fe5180f47c0 2024-02-20T00:13:33.857 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:33.854+0000 7fe51fc13700 1 -- 172.21.15.132:0/2808769580 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5180f5e20 con 0x7fe5180f47c0 2024-02-20T00:13:33.858 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:33.855+0000 7fe50e7fc700 1 -- 172.21.15.132:0/2808769580 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe5080263a0 con 0x7fe5180f47c0 2024-02-20T00:13:33.858 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:33.855+0000 7fe51fc13700 1 -- 172.21.15.132:0/2808769580 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe518059100 con 0x7fe5180f47c0 2024-02-20T00:13:33.858 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:33.855+0000 7fe50e7fc700 1 --2- 172.21.15.132:0/2808769580 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fe50402ff10 0x7fe5040323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:13:33.858 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:33.855+0000 7fe50e7fc700 1 -- 172.21.15.132:0/2808769580 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe508040f80 con 0x7fe5180f47c0 2024-02-20T00:13:33.858 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:33.855+0000 7fe51d1ae700 1 --2- 172.21.15.132:0/2808769580 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fe50402ff10 0x7fe5040323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:13:33.858 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:33.856+0000 7fe51d1ae700 1 --2- 172.21.15.132:0/2808769580 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fe50402ff10 0x7fe5040323d0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7fe5140096a0 tx=0x7fe514006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:13:33.861 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:33.860+0000 7fe50e7fc700 1 -- 172.21.15.132:0/2808769580 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe5080126f0 con 0x7fe5180f47c0 2024-02-20T00:13:34.064 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:34.063+0000 7fe51fc13700 1 -- 172.21.15.132:0/2808769580 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe5180fc090 con 0x7fe5180f47c0 2024-02-20T00:13:34.065 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:34.063+0000 7fe50e7fc700 1 -- 172.21.15.132:0/2808769580 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe508012330 con 0x7fe5180f47c0 2024-02-20T00:13:34.065 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:13:34.065 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:13:34.066 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:34.065+0000 7fe51fc13700 1 -- 172.21.15.132:0/2808769580 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fe50402ff10 msgr2=0x7fe5040323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:13:34.066 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:34.065+0000 7fe51fc13700 1 --2- 172.21.15.132:0/2808769580 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fe50402ff10 0x7fe5040323d0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7fe5140096a0 tx=0x7fe514006b40).stop 2024-02-20T00:13:34.067 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:34.065+0000 7fe51fc13700 1 -- 172.21.15.132:0/2808769580 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe5180f47c0 msgr2=0x7fe5180f7370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:13:34.067 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:34.065+0000 7fe51fc13700 1 --2- 172.21.15.132:0/2808769580 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe5180f47c0 0x7fe5180f7370 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7fe508005010 tx=0x7fe50800b040).stop 2024-02-20T00:13:34.067 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:34.065+0000 7fe51fc13700 1 -- 172.21.15.132:0/2808769580 shutdown_connections 2024-02-20T00:13:34.067 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:34.065+0000 7fe51fc13700 1 --2- 172.21.15.132:0/2808769580 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fe50402ff10 0x7fe5040323d0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:13:34.067 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:34.065+0000 7fe51fc13700 1 --2- 172.21.15.132:0/2808769580 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe5180f47c0 0x7fe5180f7370 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:13:34.067 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:34.065+0000 7fe51fc13700 1 -- 172.21.15.132:0/2808769580 >> 172.21.15.132:0/2808769580 conn(0x7fe5180f02e0 msgr2=0x7fe5180f0d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:13:34.067 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:34.065+0000 7fe51fc13700 1 -- 172.21.15.132:0/2808769580 shutdown_connections 2024-02-20T00:13:34.067 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:34.065+0000 7fe51fc13700 1 -- 172.21.15.132:0/2808769580 wait complete. 2024-02-20T00:13:34.067 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:13:34.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:13:33 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:13:32.869148+0000 mgr.smithi084.bmqrnf (mgr.14162) 109 : cluster [DBG] pgmap v102: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:13:35.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:13:34 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:13:34.063893+0000 mon.smithi084 (mon.0) 172 : audit [DBG] from='client.? 172.21.15.132:0/2808769580' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:13:35.654 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:13:35.654 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:13:36.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:13:35 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:13:34.869569+0000 mgr.smithi084.bmqrnf (mgr.14162) 110 : cluster [DBG] pgmap v103: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:13:37.347 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:37.344+0000 7f9c445be700 1 -- 172.21.15.132:0/3089556387 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9c3c0f6510 msgr2=0x7f9c3c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:13:37.347 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:37.344+0000 7f9c445be700 1 --2- 172.21.15.132:0/3089556387 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9c3c0f6510 0x7f9c3c0f6930 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f9c2c009fb0 tx=0x7f9c2c009b30).stop 2024-02-20T00:13:37.347 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:37.344+0000 7f9c445be700 1 -- 172.21.15.132:0/3089556387 shutdown_connections 2024-02-20T00:13:37.348 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:37.344+0000 7f9c445be700 1 --2- 172.21.15.132:0/3089556387 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9c3c0f6510 0x7f9c3c0f6930 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:13:37.348 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:37.344+0000 7f9c445be700 1 -- 172.21.15.132:0/3089556387 >> 172.21.15.132:0/3089556387 conn(0x7f9c3c0f1c70 msgr2=0x7f9c3c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:13:37.348 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:37.344+0000 7f9c445be700 1 -- 172.21.15.132:0/3089556387 shutdown_connections 2024-02-20T00:13:37.348 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:37.344+0000 7f9c445be700 1 -- 172.21.15.132:0/3089556387 wait complete. 2024-02-20T00:13:37.348 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:37.345+0000 7f9c445be700 1 Processor -- start 2024-02-20T00:13:37.348 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:37.345+0000 7f9c445be700 1 -- start start 2024-02-20T00:13:37.348 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:37.345+0000 7f9c445be700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9c3c0f6510 0x7f9c3c06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:13:37.348 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:37.345+0000 7f9c445be700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c3c069e20 con 0x7f9c3c0f6510 2024-02-20T00:13:37.348 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:37.345+0000 7f9c4235a700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9c3c0f6510 0x7f9c3c06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:13:37.348 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:37.345+0000 7f9c4235a700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9c3c0f6510 0x7f9c3c06b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:54774/0 (socket says 172.21.15.132:54774) 2024-02-20T00:13:37.348 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:37.345+0000 7f9c4235a700 1 -- 172.21.15.132:0/4234927533 learned_addr learned my addr 172.21.15.132:0/4234927533 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:13:37.348 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:37.346+0000 7f9c4235a700 1 -- 172.21.15.132:0/4234927533 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c2c005040 con 0x7f9c3c0f6510 2024-02-20T00:13:37.349 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:37.346+0000 7f9c4235a700 1 --2- 172.21.15.132:0/4234927533 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9c3c0f6510 0x7f9c3c06b9a0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f9c2c0089f0 tx=0x7f9c2c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:13:37.349 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:37.346+0000 7f9c32ffd700 1 -- 172.21.15.132:0/4234927533 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9c2c009320 con 0x7f9c3c0f6510 2024-02-20T00:13:37.349 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:37.346+0000 7f9c445be700 1 -- 172.21.15.132:0/4234927533 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c3c069fc0 con 0x7f9c3c0f6510 2024-02-20T00:13:37.349 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:37.346+0000 7f9c32ffd700 1 -- 172.21.15.132:0/4234927533 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9c2c013380 con 0x7f9c3c0f6510 2024-02-20T00:13:37.349 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:37.346+0000 7f9c32ffd700 1 -- 172.21.15.132:0/4234927533 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9c2c01ed80 con 0x7f9c3c0f6510 2024-02-20T00:13:37.349 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:37.346+0000 7f9c445be700 1 -- 172.21.15.132:0/4234927533 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c3c06a450 con 0x7f9c3c0f6510 2024-02-20T00:13:37.350 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:37.347+0000 7f9c32ffd700 1 -- 172.21.15.132:0/4234927533 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9c2c01e420 con 0x7f9c3c0f6510 2024-02-20T00:13:37.350 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:37.347+0000 7f9c445be700 1 -- 172.21.15.132:0/4234927533 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c3c059100 con 0x7f9c3c0f6510 2024-02-20T00:13:37.350 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:37.347+0000 7f9c32ffd700 1 --2- 172.21.15.132:0/4234927533 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f9c2802ff60 0x7f9c28032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:13:37.350 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:37.347+0000 7f9c32ffd700 1 -- 172.21.15.132:0/4234927533 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9c2c041080 con 0x7f9c3c0f6510 2024-02-20T00:13:37.350 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:37.347+0000 7f9c41b59700 1 --2- 172.21.15.132:0/4234927533 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f9c2802ff60 0x7f9c28032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:13:37.350 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:37.347+0000 7f9c41b59700 1 --2- 172.21.15.132:0/4234927533 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f9c2802ff60 0x7f9c28032420 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f9c380096a0 tx=0x7f9c38006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:13:37.352 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:37.351+0000 7f9c32ffd700 1 -- 172.21.15.132:0/4234927533 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9c2c0095f0 con 0x7f9c3c0f6510 2024-02-20T00:13:37.555 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:37.554+0000 7f9c445be700 1 -- 172.21.15.132:0/4234927533 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9c3c0fc0a0 con 0x7f9c3c0f6510 2024-02-20T00:13:37.556 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:37.555+0000 7f9c32ffd700 1 -- 172.21.15.132:0/4234927533 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9c2c01cb10 con 0x7f9c3c0f6510 2024-02-20T00:13:37.556 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:13:37.556 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:13:37.558 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:37.556+0000 7f9c445be700 1 -- 172.21.15.132:0/4234927533 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f9c2802ff60 msgr2=0x7f9c28032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:13:37.558 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:37.556+0000 7f9c445be700 1 --2- 172.21.15.132:0/4234927533 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f9c2802ff60 0x7f9c28032420 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f9c380096a0 tx=0x7f9c38006b40).stop 2024-02-20T00:13:37.558 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:37.556+0000 7f9c445be700 1 -- 172.21.15.132:0/4234927533 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9c3c0f6510 msgr2=0x7f9c3c06b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:13:37.558 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:37.556+0000 7f9c445be700 1 --2- 172.21.15.132:0/4234927533 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9c3c0f6510 0x7f9c3c06b9a0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f9c2c0089f0 tx=0x7f9c2c00b040).stop 2024-02-20T00:13:37.558 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:37.556+0000 7f9c445be700 1 -- 172.21.15.132:0/4234927533 shutdown_connections 2024-02-20T00:13:37.558 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:37.556+0000 7f9c445be700 1 --2- 172.21.15.132:0/4234927533 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f9c2802ff60 0x7f9c28032420 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:13:37.558 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:37.556+0000 7f9c445be700 1 --2- 172.21.15.132:0/4234927533 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9c3c0f6510 0x7f9c3c06b9a0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:13:37.558 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:37.556+0000 7f9c445be700 1 -- 172.21.15.132:0/4234927533 >> 172.21.15.132:0/4234927533 conn(0x7f9c3c0f1c70 msgr2=0x7f9c3c0f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:13:37.558 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:37.556+0000 7f9c445be700 1 -- 172.21.15.132:0/4234927533 shutdown_connections 2024-02-20T00:13:37.558 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:37.557+0000 7f9c445be700 1 -- 172.21.15.132:0/4234927533 wait complete. 2024-02-20T00:13:37.559 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:13:38.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:13:37 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:13:36.869861+0000 mgr.smithi084.bmqrnf (mgr.14162) 111 : cluster [DBG] pgmap v104: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:13:38.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:13:37 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:13:37.555526+0000 mon.smithi084 (mon.0) 173 : audit [DBG] from='client.? 172.21.15.132:0/4234927533' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:13:39.303 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:13:39.303 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:13:40.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:13:39 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:13:38.870222+0000 mgr.smithi084.bmqrnf (mgr.14162) 112 : cluster [DBG] pgmap v105: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:13:41.205 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:41.182+0000 7f4d73b49700 1 -- 172.21.15.132:0/3698083281 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4d6c0f5410 msgr2=0x7f4d6c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:13:41.205 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:41.182+0000 7f4d73b49700 1 --2- 172.21.15.132:0/3698083281 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4d6c0f5410 0x7f4d6c0f5830 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f4d68009fb0 tx=0x7f4d68009b30).stop 2024-02-20T00:13:41.205 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:41.182+0000 7f4d73b49700 1 -- 172.21.15.132:0/3698083281 shutdown_connections 2024-02-20T00:13:41.205 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:41.182+0000 7f4d73b49700 1 --2- 172.21.15.132:0/3698083281 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4d6c0f5410 0x7f4d6c0f5830 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:13:41.205 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:41.182+0000 7f4d73b49700 1 -- 172.21.15.132:0/3698083281 >> 172.21.15.132:0/3698083281 conn(0x7f4d6c0f0b70 msgr2=0x7f4d6c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:13:41.205 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:41.182+0000 7f4d73b49700 1 -- 172.21.15.132:0/3698083281 shutdown_connections 2024-02-20T00:13:41.205 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:41.182+0000 7f4d73b49700 1 -- 172.21.15.132:0/3698083281 wait complete. 2024-02-20T00:13:41.206 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:41.183+0000 7f4d73b49700 1 Processor -- start 2024-02-20T00:13:41.206 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:41.183+0000 7f4d73b49700 1 -- start start 2024-02-20T00:13:41.206 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:41.183+0000 7f4d73b49700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4d6c0f5410 0x7f4d6c1067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:13:41.206 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:41.183+0000 7f4d73b49700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d6c106cd0 con 0x7f4d6c0f5410 2024-02-20T00:13:41.206 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:41.183+0000 7f4d718e5700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4d6c0f5410 0x7f4d6c1067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:13:41.206 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:41.183+0000 7f4d718e5700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4d6c0f5410 0x7f4d6c1067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:54792/0 (socket says 172.21.15.132:54792) 2024-02-20T00:13:41.206 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:41.183+0000 7f4d718e5700 1 -- 172.21.15.132:0/3291791697 learned_addr learned my addr 172.21.15.132:0/3291791697 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:13:41.206 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:41.183+0000 7f4d718e5700 1 -- 172.21.15.132:0/3291791697 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d68005040 con 0x7f4d6c0f5410 2024-02-20T00:13:41.206 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:41.184+0000 7f4d718e5700 1 --2- 172.21.15.132:0/3291791697 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4d6c0f5410 0x7f4d6c1067e0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f4d680089f0 tx=0x7f4d6800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:13:41.206 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:41.184+0000 7f4d627fc700 1 -- 172.21.15.132:0/3291791697 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4d68009320 con 0x7f4d6c0f5410 2024-02-20T00:13:41.206 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:41.184+0000 7f4d73b49700 1 -- 172.21.15.132:0/3291791697 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d6c106ed0 con 0x7f4d6c0f5410 2024-02-20T00:13:41.206 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:41.184+0000 7f4d627fc700 1 -- 172.21.15.132:0/3291791697 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4d68013380 con 0x7f4d6c0f5410 2024-02-20T00:13:41.207 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:41.184+0000 7f4d627fc700 1 -- 172.21.15.132:0/3291791697 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4d6801ed80 con 0x7f4d6c0f5410 2024-02-20T00:13:41.207 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:41.184+0000 7f4d73b49700 1 -- 172.21.15.132:0/3291791697 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d6c107360 con 0x7f4d6c0f5410 2024-02-20T00:13:41.207 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:41.185+0000 7f4d627fc700 1 -- 172.21.15.132:0/3291791697 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4d6801e420 con 0x7f4d6c0f5410 2024-02-20T00:13:41.207 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:41.185+0000 7f4d73b49700 1 -- 172.21.15.132:0/3291791697 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d6c045ad0 con 0x7f4d6c0f5410 2024-02-20T00:13:41.207 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:41.185+0000 7f4d627fc700 1 --2- 172.21.15.132:0/3291791697 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f4d5802ff60 0x7f4d58032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:13:41.207 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:41.185+0000 7f4d627fc700 1 -- 172.21.15.132:0/3291791697 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4d68040f30 con 0x7f4d6c0f5410 2024-02-20T00:13:41.207 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:41.185+0000 7f4d710e4700 1 --2- 172.21.15.132:0/3291791697 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f4d5802ff60 0x7f4d58032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:13:41.207 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:41.185+0000 7f4d710e4700 1 --2- 172.21.15.132:0/3291791697 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f4d5802ff60 0x7f4d58032420 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f4d5c0096a0 tx=0x7f4d5c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:13:41.207 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:41.189+0000 7f4d627fc700 1 -- 172.21.15.132:0/3291791697 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4d6800e2a0 con 0x7f4d6c0f5410 2024-02-20T00:13:41.397 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:41.395+0000 7f4d73b49700 1 -- 172.21.15.132:0/3291791697 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4d6c108210 con 0x7f4d6c0f5410 2024-02-20T00:13:41.398 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:41.396+0000 7f4d627fc700 1 -- 172.21.15.132:0/3291791697 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4d6800e2a0 con 0x7f4d6c0f5410 2024-02-20T00:13:41.398 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:13:41.398 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:13:41.400 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:41.398+0000 7f4d73b49700 1 -- 172.21.15.132:0/3291791697 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f4d5802ff60 msgr2=0x7f4d58032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:13:41.400 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:41.398+0000 7f4d73b49700 1 --2- 172.21.15.132:0/3291791697 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f4d5802ff60 0x7f4d58032420 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f4d5c0096a0 tx=0x7f4d5c006b40).stop 2024-02-20T00:13:41.400 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:41.398+0000 7f4d73b49700 1 -- 172.21.15.132:0/3291791697 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4d6c0f5410 msgr2=0x7f4d6c1067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:13:41.400 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:41.398+0000 7f4d73b49700 1 --2- 172.21.15.132:0/3291791697 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4d6c0f5410 0x7f4d6c1067e0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f4d680089f0 tx=0x7f4d6800b040).stop 2024-02-20T00:13:41.400 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:41.398+0000 7f4d73b49700 1 -- 172.21.15.132:0/3291791697 shutdown_connections 2024-02-20T00:13:41.400 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:41.398+0000 7f4d73b49700 1 --2- 172.21.15.132:0/3291791697 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f4d5802ff60 0x7f4d58032420 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:13:41.401 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:41.398+0000 7f4d73b49700 1 --2- 172.21.15.132:0/3291791697 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4d6c0f5410 0x7f4d6c1067e0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:13:41.401 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:41.398+0000 7f4d73b49700 1 -- 172.21.15.132:0/3291791697 >> 172.21.15.132:0/3291791697 conn(0x7f4d6c0f0b70 msgr2=0x7f4d6c0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:13:41.401 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:41.398+0000 7f4d73b49700 1 -- 172.21.15.132:0/3291791697 shutdown_connections 2024-02-20T00:13:41.401 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:41.398+0000 7f4d73b49700 1 -- 172.21.15.132:0/3291791697 wait complete. 2024-02-20T00:13:41.401 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:13:42.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:13:41 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:13:40.870565+0000 mgr.smithi084.bmqrnf (mgr.14162) 113 : cluster [DBG] pgmap v106: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:13:42.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:13:41 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:13:41.397554+0000 mon.smithi084 (mon.0) 174 : audit [DBG] from='client.? 172.21.15.132:0/3291791697' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:13:43.085 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:13:43.085 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:13:44.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:13:43 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:13:42.870915+0000 mgr.smithi084.bmqrnf (mgr.14162) 114 : cluster [DBG] pgmap v107: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:13:44.962 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:44.958+0000 7f53be5b6700 1 -- 172.21.15.132:0/1579518143 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f53b8060280 msgr2=0x7f53b80f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:13:44.962 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:44.958+0000 7f53be5b6700 1 --2- 172.21.15.132:0/1579518143 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f53b8060280 0x7f53b80f90f0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f53a0009fb0 tx=0x7f53a0009b30).stop 2024-02-20T00:13:44.962 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:44.958+0000 7f53be5b6700 1 -- 172.21.15.132:0/1579518143 shutdown_connections 2024-02-20T00:13:44.962 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:44.958+0000 7f53be5b6700 1 --2- 172.21.15.132:0/1579518143 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f53b8060280 0x7f53b80f90f0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:13:44.962 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:44.958+0000 7f53be5b6700 1 -- 172.21.15.132:0/1579518143 >> 172.21.15.132:0/1579518143 conn(0x7f53b80f0b70 msgr2=0x7f53b80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:13:44.962 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:44.958+0000 7f53be5b6700 1 -- 172.21.15.132:0/1579518143 shutdown_connections 2024-02-20T00:13:44.962 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:44.958+0000 7f53be5b6700 1 -- 172.21.15.132:0/1579518143 wait complete. 2024-02-20T00:13:44.963 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:44.958+0000 7f53be5b6700 1 Processor -- start 2024-02-20T00:13:44.963 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:44.959+0000 7f53be5b6700 1 -- start start 2024-02-20T00:13:44.963 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:44.959+0000 7f53be5b6700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f53b8060280 0x7f53b81067a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:13:44.963 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:44.959+0000 7f53be5b6700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53b8106c90 con 0x7f53b8060280 2024-02-20T00:13:44.963 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:44.959+0000 7f53b7fff700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f53b8060280 0x7f53b81067a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:13:44.963 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:44.959+0000 7f53b7fff700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f53b8060280 0x7f53b81067a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:54796/0 (socket says 172.21.15.132:54796) 2024-02-20T00:13:44.963 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:44.959+0000 7f53b7fff700 1 -- 172.21.15.132:0/3958252445 learned_addr learned my addr 172.21.15.132:0/3958252445 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:13:44.963 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:44.959+0000 7f53b7fff700 1 -- 172.21.15.132:0/3958252445 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53a0005040 con 0x7f53b8060280 2024-02-20T00:13:44.963 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:44.959+0000 7f53b7fff700 1 --2- 172.21.15.132:0/3958252445 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f53b8060280 0x7f53b81067a0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f53a00089f0 tx=0x7f53a000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:13:44.963 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:44.960+0000 7f53b4ff9700 1 -- 172.21.15.132:0/3958252445 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f53a0009320 con 0x7f53b8060280 2024-02-20T00:13:44.963 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:44.960+0000 7f53be5b6700 1 -- 172.21.15.132:0/3958252445 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53b8106e90 con 0x7f53b8060280 2024-02-20T00:13:44.963 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:44.960+0000 7f53b4ff9700 1 -- 172.21.15.132:0/3958252445 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f53a0013380 con 0x7f53b8060280 2024-02-20T00:13:44.963 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:44.960+0000 7f53b4ff9700 1 -- 172.21.15.132:0/3958252445 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f53a0014e20 con 0x7f53b8060280 2024-02-20T00:13:44.964 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:44.960+0000 7f53be5b6700 1 -- 172.21.15.132:0/3958252445 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53b8107320 con 0x7f53b8060280 2024-02-20T00:13:44.964 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:44.960+0000 7f53b4ff9700 1 -- 172.21.15.132:0/3958252445 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f53a00095f0 con 0x7f53b8060280 2024-02-20T00:13:44.965 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:44.960+0000 7f53be5b6700 1 -- 172.21.15.132:0/3958252445 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53b8045ad0 con 0x7f53b8060280 2024-02-20T00:13:44.965 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:44.961+0000 7f53b4ff9700 1 --2- 172.21.15.132:0/3958252445 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f53a402ff10 0x7f53a40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:13:44.965 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:44.961+0000 7f53b4ff9700 1 -- 172.21.15.132:0/3958252445 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f53a0040f50 con 0x7f53b8060280 2024-02-20T00:13:44.965 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:44.961+0000 7f53b77fe700 1 --2- 172.21.15.132:0/3958252445 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f53a402ff10 0x7f53a40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:13:44.965 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:44.961+0000 7f53b77fe700 1 --2- 172.21.15.132:0/3958252445 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f53a402ff10 0x7f53a40323d0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f53a80096a0 tx=0x7f53a8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:13:44.967 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:44.965+0000 7f53b4ff9700 1 -- 172.21.15.132:0/3958252445 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f53a0014420 con 0x7f53b8060280 2024-02-20T00:13:45.172 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:45.170+0000 7f53be5b6700 1 -- 172.21.15.132:0/3958252445 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f53b81081d0 con 0x7f53b8060280 2024-02-20T00:13:45.173 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:45.170+0000 7f53b4ff9700 1 -- 172.21.15.132:0/3958252445 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f53a001d980 con 0x7f53b8060280 2024-02-20T00:13:45.173 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:13:45.173 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:13:45.174 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:45.172+0000 7f53be5b6700 1 -- 172.21.15.132:0/3958252445 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f53a402ff10 msgr2=0x7f53a40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:13:45.175 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:45.172+0000 7f53be5b6700 1 --2- 172.21.15.132:0/3958252445 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f53a402ff10 0x7f53a40323d0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f53a80096a0 tx=0x7f53a8006b40).stop 2024-02-20T00:13:45.175 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:45.172+0000 7f53be5b6700 1 -- 172.21.15.132:0/3958252445 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f53b8060280 msgr2=0x7f53b81067a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:13:45.175 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:45.172+0000 7f53be5b6700 1 --2- 172.21.15.132:0/3958252445 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f53b8060280 0x7f53b81067a0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f53a00089f0 tx=0x7f53a000b040).stop 2024-02-20T00:13:45.175 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:45.172+0000 7f53be5b6700 1 -- 172.21.15.132:0/3958252445 shutdown_connections 2024-02-20T00:13:45.175 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:45.172+0000 7f53be5b6700 1 --2- 172.21.15.132:0/3958252445 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f53a402ff10 0x7f53a40323d0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:13:45.175 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:45.172+0000 7f53be5b6700 1 --2- 172.21.15.132:0/3958252445 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f53b8060280 0x7f53b81067a0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:13:45.175 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:45.172+0000 7f53be5b6700 1 -- 172.21.15.132:0/3958252445 >> 172.21.15.132:0/3958252445 conn(0x7f53b80f0b70 msgr2=0x7f53b80f1600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:13:45.175 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:45.172+0000 7f53be5b6700 1 -- 172.21.15.132:0/3958252445 shutdown_connections 2024-02-20T00:13:45.175 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:45.172+0000 7f53be5b6700 1 -- 172.21.15.132:0/3958252445 wait complete. 2024-02-20T00:13:45.176 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:13:46.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:13:45 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:13:44.871259+0000 mgr.smithi084.bmqrnf (mgr.14162) 115 : cluster [DBG] pgmap v108: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:13:46.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:13:45 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:13:45.172023+0000 mon.smithi084 (mon.0) 175 : audit [DBG] from='client.? 172.21.15.132:0/3958252445' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:13:46.880 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:13:46.880 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:13:48.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:13:47 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:13:46.871550+0000 mgr.smithi084.bmqrnf (mgr.14162) 116 : cluster [DBG] pgmap v109: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:13:48.690 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:48.686+0000 7febaa5bd700 1 -- 172.21.15.132:0/3053541817 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7feba40f6510 msgr2=0x7feba40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:13:48.690 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:48.686+0000 7febaa5bd700 1 --2- 172.21.15.132:0/3053541817 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7feba40f6510 0x7feba40f6930 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7feb8c009fb0 tx=0x7feb8c009b30).stop 2024-02-20T00:13:48.690 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:48.686+0000 7febaa5bd700 1 -- 172.21.15.132:0/3053541817 shutdown_connections 2024-02-20T00:13:48.690 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:48.686+0000 7febaa5bd700 1 --2- 172.21.15.132:0/3053541817 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7feba40f6510 0x7feba40f6930 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:13:48.691 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:48.686+0000 7febaa5bd700 1 -- 172.21.15.132:0/3053541817 >> 172.21.15.132:0/3053541817 conn(0x7feba40f1c70 msgr2=0x7feba40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:13:48.691 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:48.686+0000 7febaa5bd700 1 -- 172.21.15.132:0/3053541817 shutdown_connections 2024-02-20T00:13:48.691 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:48.686+0000 7febaa5bd700 1 -- 172.21.15.132:0/3053541817 wait complete. 2024-02-20T00:13:48.691 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:48.687+0000 7febaa5bd700 1 Processor -- start 2024-02-20T00:13:48.691 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:48.687+0000 7febaa5bd700 1 -- start start 2024-02-20T00:13:48.691 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:48.687+0000 7febaa5bd700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7feba40f6510 0x7feba406b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:13:48.691 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:48.687+0000 7febaa5bd700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feba4069e20 con 0x7feba40f6510 2024-02-20T00:13:48.691 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:48.687+0000 7feba3fff700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7feba40f6510 0x7feba406b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:13:48.691 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:48.687+0000 7feba3fff700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7feba40f6510 0x7feba406b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:57802/0 (socket says 172.21.15.132:57802) 2024-02-20T00:13:48.691 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:48.687+0000 7feba3fff700 1 -- 172.21.15.132:0/292513809 learned_addr learned my addr 172.21.15.132:0/292513809 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:13:48.691 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:48.688+0000 7feba3fff700 1 -- 172.21.15.132:0/292513809 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb8c005040 con 0x7feba40f6510 2024-02-20T00:13:48.691 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:48.688+0000 7feba3fff700 1 --2- 172.21.15.132:0/292513809 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7feba40f6510 0x7feba406b9a0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7feb8c005010 tx=0x7feb8c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:13:48.691 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:48.688+0000 7feba0ff9700 1 -- 172.21.15.132:0/292513809 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feb8c009320 con 0x7feba40f6510 2024-02-20T00:13:48.692 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:48.688+0000 7febaa5bd700 1 -- 172.21.15.132:0/292513809 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feba4069fc0 con 0x7feba40f6510 2024-02-20T00:13:48.692 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:48.688+0000 7feba0ff9700 1 -- 172.21.15.132:0/292513809 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7feb8c013380 con 0x7feba40f6510 2024-02-20T00:13:48.692 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:48.688+0000 7feba0ff9700 1 -- 172.21.15.132:0/292513809 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feb8c014e50 con 0x7feba40f6510 2024-02-20T00:13:48.692 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:48.688+0000 7febaa5bd700 1 -- 172.21.15.132:0/292513809 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feba406a450 con 0x7feba40f6510 2024-02-20T00:13:48.693 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:48.689+0000 7feba0ff9700 1 -- 172.21.15.132:0/292513809 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7feb8c014420 con 0x7feba40f6510 2024-02-20T00:13:48.693 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:48.689+0000 7febaa5bd700 1 -- 172.21.15.132:0/292513809 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feba4059100 con 0x7feba40f6510 2024-02-20T00:13:48.693 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:48.689+0000 7feba0ff9700 1 --2- 172.21.15.132:0/292513809 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7feb9002ff10 0x7feb900323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:13:48.693 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:48.689+0000 7feba0ff9700 1 -- 172.21.15.132:0/292513809 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7feb8c040f40 con 0x7feba40f6510 2024-02-20T00:13:48.693 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:48.689+0000 7feba37fe700 1 --2- 172.21.15.132:0/292513809 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7feb9002ff10 0x7feb900323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:13:48.693 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:48.690+0000 7feba37fe700 1 --2- 172.21.15.132:0/292513809 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7feb9002ff10 0x7feb900323d0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7feb940096a0 tx=0x7feb94006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:13:48.696 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:48.694+0000 7feba0ff9700 1 -- 172.21.15.132:0/292513809 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7feb8c0095f0 con 0x7feba40f6510 2024-02-20T00:13:48.900 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:48.898+0000 7febaa5bd700 1 -- 172.21.15.132:0/292513809 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7feba40fc0a0 con 0x7feba40f6510 2024-02-20T00:13:48.901 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:48.899+0000 7feba0ff9700 1 -- 172.21.15.132:0/292513809 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7feb8c01db70 con 0x7feba40f6510 2024-02-20T00:13:48.901 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:13:48.901 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:13:48.903 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:48.900+0000 7febaa5bd700 1 -- 172.21.15.132:0/292513809 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7feb9002ff10 msgr2=0x7feb900323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:13:48.903 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:48.900+0000 7febaa5bd700 1 --2- 172.21.15.132:0/292513809 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7feb9002ff10 0x7feb900323d0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7feb940096a0 tx=0x7feb94006b40).stop 2024-02-20T00:13:48.903 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:48.900+0000 7febaa5bd700 1 -- 172.21.15.132:0/292513809 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7feba40f6510 msgr2=0x7feba406b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:13:48.903 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:48.900+0000 7febaa5bd700 1 --2- 172.21.15.132:0/292513809 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7feba40f6510 0x7feba406b9a0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7feb8c005010 tx=0x7feb8c00b040).stop 2024-02-20T00:13:48.903 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:48.900+0000 7febaa5bd700 1 -- 172.21.15.132:0/292513809 shutdown_connections 2024-02-20T00:13:48.904 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:48.900+0000 7febaa5bd700 1 --2- 172.21.15.132:0/292513809 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7feb9002ff10 0x7feb900323d0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:13:48.904 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:48.900+0000 7febaa5bd700 1 --2- 172.21.15.132:0/292513809 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7feba40f6510 0x7feba406b9a0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:13:48.904 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:48.900+0000 7febaa5bd700 1 -- 172.21.15.132:0/292513809 >> 172.21.15.132:0/292513809 conn(0x7feba40f1c70 msgr2=0x7feba40f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:13:48.904 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:48.900+0000 7febaa5bd700 1 -- 172.21.15.132:0/292513809 shutdown_connections 2024-02-20T00:13:48.904 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:48.900+0000 7febaa5bd700 1 -- 172.21.15.132:0/292513809 wait complete. 2024-02-20T00:13:48.905 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:13:49.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:13:48 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:13:48.900270+0000 mon.smithi084 (mon.0) 176 : audit [DBG] from='client.? 172.21.15.132:0/292513809' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:13:50.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:13:49 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:13:48.871893+0000 mgr.smithi084.bmqrnf (mgr.14162) 117 : cluster [DBG] pgmap v110: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:13:50.514 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:13:50.515 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:13:52.267 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:52.264+0000 7f4a6188e700 1 -- 172.21.15.132:0/894656118 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4a5c0f62a0 msgr2=0x7f4a5c0f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:13:52.267 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:52.264+0000 7f4a6188e700 1 --2- 172.21.15.132:0/894656118 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4a5c0f62a0 0x7f4a5c0f66c0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7f4a50009fb0 tx=0x7f4a50009b30).stop 2024-02-20T00:13:52.267 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:52.264+0000 7f4a6188e700 1 -- 172.21.15.132:0/894656118 shutdown_connections 2024-02-20T00:13:52.267 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:52.264+0000 7f4a6188e700 1 --2- 172.21.15.132:0/894656118 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4a5c0f62a0 0x7f4a5c0f66c0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:13:52.267 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:52.264+0000 7f4a6188e700 1 -- 172.21.15.132:0/894656118 >> 172.21.15.132:0/894656118 conn(0x7f4a5c0f1aa0 msgr2=0x7f4a5c0f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:13:52.268 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:52.264+0000 7f4a6188e700 1 -- 172.21.15.132:0/894656118 shutdown_connections 2024-02-20T00:13:52.268 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:52.264+0000 7f4a6188e700 1 -- 172.21.15.132:0/894656118 wait complete. 2024-02-20T00:13:52.268 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:52.264+0000 7f4a6188e700 1 Processor -- start 2024-02-20T00:13:52.268 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:52.264+0000 7f4a6188e700 1 -- start start 2024-02-20T00:13:52.268 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:52.265+0000 7f4a6188e700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4a5c0f62a0 0x7f4a5c06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:13:52.268 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:52.265+0000 7f4a6188e700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a5c069e20 con 0x7f4a5c0f62a0 2024-02-20T00:13:52.270 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:52.265+0000 7f4a6088c700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4a5c0f62a0 0x7f4a5c06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:13:52.270 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:52.265+0000 7f4a6088c700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4a5c0f62a0 0x7f4a5c06b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:57830/0 (socket says 172.21.15.132:57830) 2024-02-20T00:13:52.270 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:52.265+0000 7f4a6088c700 1 -- 172.21.15.132:0/3830975588 learned_addr learned my addr 172.21.15.132:0/3830975588 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:13:52.270 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:52.265+0000 7f4a6088c700 1 -- 172.21.15.132:0/3830975588 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a50005040 con 0x7f4a5c0f62a0 2024-02-20T00:13:52.270 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:52.265+0000 7f4a6088c700 1 --2- 172.21.15.132:0/3830975588 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4a5c0f62a0 0x7f4a5c06b9a0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7f4a500089f0 tx=0x7f4a5000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:13:52.270 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:52.266+0000 7f4a597fa700 1 -- 172.21.15.132:0/3830975588 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4a50009320 con 0x7f4a5c0f62a0 2024-02-20T00:13:52.270 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:52.266+0000 7f4a6188e700 1 -- 172.21.15.132:0/3830975588 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a5c069fc0 con 0x7f4a5c0f62a0 2024-02-20T00:13:52.270 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:52.266+0000 7f4a597fa700 1 -- 172.21.15.132:0/3830975588 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4a50013380 con 0x7f4a5c0f62a0 2024-02-20T00:13:52.271 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:52.266+0000 7f4a597fa700 1 -- 172.21.15.132:0/3830975588 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4a50014e20 con 0x7f4a5c0f62a0 2024-02-20T00:13:52.271 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:52.266+0000 7f4a6188e700 1 -- 172.21.15.132:0/3830975588 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a5c06a450 con 0x7f4a5c0f62a0 2024-02-20T00:13:52.271 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:52.266+0000 7f4a597fa700 1 -- 172.21.15.132:0/3830975588 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4a500263a0 con 0x7f4a5c0f62a0 2024-02-20T00:13:52.271 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:52.266+0000 7f4a6188e700 1 -- 172.21.15.132:0/3830975588 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a5c059100 con 0x7f4a5c0f62a0 2024-02-20T00:13:52.271 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:52.267+0000 7f4a597fa700 1 --2- 172.21.15.132:0/3830975588 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f4a4c02ff10 0x7f4a4c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:13:52.271 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:52.267+0000 7f4a597fa700 1 -- 172.21.15.132:0/3830975588 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4a50040f60 con 0x7f4a5c0f62a0 2024-02-20T00:13:52.271 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:52.267+0000 7f4a5bfff700 1 --2- 172.21.15.132:0/3830975588 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f4a4c02ff10 0x7f4a4c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:13:52.271 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:52.267+0000 7f4a5bfff700 1 --2- 172.21.15.132:0/3830975588 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f4a4c02ff10 0x7f4a4c0323d0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f4a480096a0 tx=0x7f4a48006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:13:52.273 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:52.271+0000 7f4a597fa700 1 -- 172.21.15.132:0/3830975588 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4a500126f0 con 0x7f4a5c0f62a0 2024-02-20T00:13:52.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:13:51 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:13:50.872231+0000 mgr.smithi084.bmqrnf (mgr.14162) 118 : cluster [DBG] pgmap v111: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:13:52.479 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:52.477+0000 7f4a6188e700 1 -- 172.21.15.132:0/3830975588 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4a5c0fbe30 con 0x7f4a5c0f62a0 2024-02-20T00:13:52.480 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:52.478+0000 7f4a597fa700 1 -- 172.21.15.132:0/3830975588 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4a500126f0 con 0x7f4a5c0f62a0 2024-02-20T00:13:52.480 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:13:52.480 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:13:52.481 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:52.479+0000 7f4a6188e700 1 -- 172.21.15.132:0/3830975588 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f4a4c02ff10 msgr2=0x7f4a4c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:13:52.481 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:52.479+0000 7f4a6188e700 1 --2- 172.21.15.132:0/3830975588 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f4a4c02ff10 0x7f4a4c0323d0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f4a480096a0 tx=0x7f4a48006b40).stop 2024-02-20T00:13:52.482 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:52.479+0000 7f4a6188e700 1 -- 172.21.15.132:0/3830975588 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4a5c0f62a0 msgr2=0x7f4a5c06b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:13:52.482 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:52.479+0000 7f4a6188e700 1 --2- 172.21.15.132:0/3830975588 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4a5c0f62a0 0x7f4a5c06b9a0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7f4a500089f0 tx=0x7f4a5000b040).stop 2024-02-20T00:13:52.482 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:52.479+0000 7f4a6188e700 1 -- 172.21.15.132:0/3830975588 shutdown_connections 2024-02-20T00:13:52.482 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:52.479+0000 7f4a6188e700 1 --2- 172.21.15.132:0/3830975588 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f4a4c02ff10 0x7f4a4c0323d0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:13:52.482 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:52.479+0000 7f4a6188e700 1 --2- 172.21.15.132:0/3830975588 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4a5c0f62a0 0x7f4a5c06b9a0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:13:52.482 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:52.479+0000 7f4a6188e700 1 -- 172.21.15.132:0/3830975588 >> 172.21.15.132:0/3830975588 conn(0x7f4a5c0f1aa0 msgr2=0x7f4a5c0f2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:13:52.482 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:52.479+0000 7f4a6188e700 1 -- 172.21.15.132:0/3830975588 shutdown_connections 2024-02-20T00:13:52.482 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:52.479+0000 7f4a6188e700 1 -- 172.21.15.132:0/3830975588 wait complete. 2024-02-20T00:13:52.483 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:13:53.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:13:52 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:13:52.479388+0000 mon.smithi084 (mon.0) 177 : audit [DBG] from='client.? 172.21.15.132:0/3830975588' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:13:54.205 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:13:54.206 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:13:54.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:13:53 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:13:52.872611+0000 mgr.smithi084.bmqrnf (mgr.14162) 119 : cluster [DBG] pgmap v112: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:13:56.071 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:56.068+0000 7fdf6d1de700 1 -- 172.21.15.132:0/1661013811 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdf680f62a0 msgr2=0x7fdf680f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:13:56.071 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:56.068+0000 7fdf6d1de700 1 --2- 172.21.15.132:0/1661013811 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdf680f62a0 0x7fdf680f66c0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 rx=0x7fdf58009fb0 tx=0x7fdf58009b30).stop 2024-02-20T00:13:56.072 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:56.068+0000 7fdf6d1de700 1 -- 172.21.15.132:0/1661013811 shutdown_connections 2024-02-20T00:13:56.072 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:56.068+0000 7fdf6d1de700 1 --2- 172.21.15.132:0/1661013811 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdf680f62a0 0x7fdf680f66c0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:13:56.072 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:56.068+0000 7fdf6d1de700 1 -- 172.21.15.132:0/1661013811 >> 172.21.15.132:0/1661013811 conn(0x7fdf680f1aa0 msgr2=0x7fdf680f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:13:56.072 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:56.068+0000 7fdf6d1de700 1 -- 172.21.15.132:0/1661013811 shutdown_connections 2024-02-20T00:13:56.072 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:56.068+0000 7fdf6d1de700 1 -- 172.21.15.132:0/1661013811 wait complete. 2024-02-20T00:13:56.072 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:56.069+0000 7fdf6d1de700 1 Processor -- start 2024-02-20T00:13:56.072 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:56.069+0000 7fdf6d1de700 1 -- start start 2024-02-20T00:13:56.072 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:56.069+0000 7fdf6d1de700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdf680f62a0 0x7fdf681065d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:13:56.072 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:56.069+0000 7fdf6d1de700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf68106ac0 con 0x7fdf680f62a0 2024-02-20T00:13:56.074 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:56.070+0000 7fdf67fff700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdf680f62a0 0x7fdf681065d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:13:56.074 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:56.070+0000 7fdf67fff700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdf680f62a0 0x7fdf681065d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:57854/0 (socket says 172.21.15.132:57854) 2024-02-20T00:13:56.074 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:56.070+0000 7fdf67fff700 1 -- 172.21.15.132:0/638711242 learned_addr learned my addr 172.21.15.132:0/638711242 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:13:56.074 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:56.070+0000 7fdf67fff700 1 -- 172.21.15.132:0/638711242 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf58005040 con 0x7fdf680f62a0 2024-02-20T00:13:56.074 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:56.070+0000 7fdf67fff700 1 --2- 172.21.15.132:0/638711242 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdf680f62a0 0x7fdf681065d0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7fdf580089f0 tx=0x7fdf5800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:13:56.074 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:56.070+0000 7fdf64ff9700 1 -- 172.21.15.132:0/638711242 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdf58009320 con 0x7fdf680f62a0 2024-02-20T00:13:56.074 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:56.070+0000 7fdf6d1de700 1 -- 172.21.15.132:0/638711242 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf68106cc0 con 0x7fdf680f62a0 2024-02-20T00:13:56.074 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:56.070+0000 7fdf64ff9700 1 -- 172.21.15.132:0/638711242 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdf58013380 con 0x7fdf680f62a0 2024-02-20T00:13:56.074 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:56.070+0000 7fdf64ff9700 1 -- 172.21.15.132:0/638711242 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdf58014e20 con 0x7fdf680f62a0 2024-02-20T00:13:56.075 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:56.070+0000 7fdf6d1de700 1 -- 172.21.15.132:0/638711242 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf68107150 con 0x7fdf680f62a0 2024-02-20T00:13:56.075 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:56.071+0000 7fdf64ff9700 1 -- 172.21.15.132:0/638711242 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdf58014420 con 0x7fdf680f62a0 2024-02-20T00:13:56.075 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:56.071+0000 7fdf6d1de700 1 -- 172.21.15.132:0/638711242 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf68045ad0 con 0x7fdf680f62a0 2024-02-20T00:13:56.075 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:56.071+0000 7fdf64ff9700 1 --2- 172.21.15.132:0/638711242 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fdf5002ff60 0x7fdf50032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:13:56.075 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:56.071+0000 7fdf64ff9700 1 -- 172.21.15.132:0/638711242 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdf58040fc0 con 0x7fdf680f62a0 2024-02-20T00:13:56.075 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:56.071+0000 7fdf677fe700 1 --2- 172.21.15.132:0/638711242 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fdf5002ff60 0x7fdf50032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:13:56.075 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:56.072+0000 7fdf677fe700 1 --2- 172.21.15.132:0/638711242 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fdf5002ff60 0x7fdf50032420 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7fdf5c0096a0 tx=0x7fdf5c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:13:56.078 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:56.076+0000 7fdf64ff9700 1 -- 172.21.15.132:0/638711242 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdf580095f0 con 0x7fdf680f62a0 2024-02-20T00:13:56.281 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:56.279+0000 7fdf6d1de700 1 -- 172.21.15.132:0/638711242 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdf68108000 con 0x7fdf680f62a0 2024-02-20T00:13:56.282 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:56.280+0000 7fdf64ff9700 1 -- 172.21.15.132:0/638711242 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fdf5801db70 con 0x7fdf680f62a0 2024-02-20T00:13:56.282 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:13:56.282 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:13:56.284 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:56.281+0000 7fdf6d1de700 1 -- 172.21.15.132:0/638711242 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fdf5002ff60 msgr2=0x7fdf50032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:13:56.284 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:56.281+0000 7fdf6d1de700 1 --2- 172.21.15.132:0/638711242 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fdf5002ff60 0x7fdf50032420 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7fdf5c0096a0 tx=0x7fdf5c006b40).stop 2024-02-20T00:13:56.285 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:56.281+0000 7fdf6d1de700 1 -- 172.21.15.132:0/638711242 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdf680f62a0 msgr2=0x7fdf681065d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:13:56.285 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:56.281+0000 7fdf6d1de700 1 --2- 172.21.15.132:0/638711242 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdf680f62a0 0x7fdf681065d0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7fdf580089f0 tx=0x7fdf5800b040).stop 2024-02-20T00:13:56.285 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:56.282+0000 7fdf6d1de700 1 -- 172.21.15.132:0/638711242 shutdown_connections 2024-02-20T00:13:56.285 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:56.282+0000 7fdf6d1de700 1 --2- 172.21.15.132:0/638711242 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fdf5002ff60 0x7fdf50032420 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:13:56.285 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:56.282+0000 7fdf6d1de700 1 --2- 172.21.15.132:0/638711242 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdf680f62a0 0x7fdf681065d0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:13:56.285 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:56.282+0000 7fdf6d1de700 1 -- 172.21.15.132:0/638711242 >> 172.21.15.132:0/638711242 conn(0x7fdf680f1aa0 msgr2=0x7fdf680f2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:13:56.285 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:56.282+0000 7fdf6d1de700 1 -- 172.21.15.132:0/638711242 shutdown_connections 2024-02-20T00:13:56.285 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:56.282+0000 7fdf6d1de700 1 -- 172.21.15.132:0/638711242 wait complete. 2024-02-20T00:13:56.285 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:13:56.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:13:55 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:13:54.872916+0000 mgr.smithi084.bmqrnf (mgr.14162) 120 : cluster [DBG] pgmap v113: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:13:57.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:13:56 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:13:56.281155+0000 mon.smithi084 (mon.0) 178 : audit [DBG] from='client.? 172.21.15.132:0/638711242' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:13:57.930 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:13:57.931 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:13:58.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:13:57 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:13:56.873266+0000 mgr.smithi084.bmqrnf (mgr.14162) 121 : cluster [DBG] pgmap v114: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:13:58.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:13:57 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:13:56.973266+0000 mon.smithi084 (mon.0) 179 : audit [INF] from='mgr.14162 172.21.15.84:0/2152933216' entity='mgr.smithi084.bmqrnf' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi084.bmqrnf/mirror_snapshot_schedule"}]: dispatch 2024-02-20T00:13:58.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:13:57 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:13:56.979558+0000 mon.smithi084 (mon.0) 180 : audit [INF] from='mgr.14162 172.21.15.84:0/2152933216' entity='mgr.smithi084.bmqrnf' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi084.bmqrnf/trash_purge_schedule"}]: dispatch 2024-02-20T00:13:59.722 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:59.717+0000 7ff33ae5b700 1 -- 172.21.15.132:0/4201893752 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff3340f6480 msgr2=0x7ff3340f68a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:13:59.722 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:59.717+0000 7ff33ae5b700 1 --2- 172.21.15.132:0/4201893752 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff3340f6480 0x7ff3340f68a0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 rx=0x7ff328009fb0 tx=0x7ff328009b30).stop 2024-02-20T00:13:59.723 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:59.717+0000 7ff33ae5b700 1 -- 172.21.15.132:0/4201893752 shutdown_connections 2024-02-20T00:13:59.723 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:59.717+0000 7ff33ae5b700 1 --2- 172.21.15.132:0/4201893752 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff3340f6480 0x7ff3340f68a0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:13:59.723 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:59.717+0000 7ff33ae5b700 1 -- 172.21.15.132:0/4201893752 >> 172.21.15.132:0/4201893752 conn(0x7ff3340f1c00 msgr2=0x7ff3340f4020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:13:59.723 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:59.717+0000 7ff33ae5b700 1 -- 172.21.15.132:0/4201893752 shutdown_connections 2024-02-20T00:13:59.723 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:59.717+0000 7ff33ae5b700 1 -- 172.21.15.132:0/4201893752 wait complete. 2024-02-20T00:13:59.723 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:59.718+0000 7ff33ae5b700 1 Processor -- start 2024-02-20T00:13:59.723 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:59.718+0000 7ff33ae5b700 1 -- start start 2024-02-20T00:13:59.723 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:59.718+0000 7ff33ae5b700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff3340f6480 0x7ff33406b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:13:59.723 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:59.718+0000 7ff33ae5b700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff334069e20 con 0x7ff3340f6480 2024-02-20T00:13:59.723 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:59.718+0000 7ff338bf7700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff3340f6480 0x7ff33406b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:13:59.724 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:59.718+0000 7ff338bf7700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff3340f6480 0x7ff33406b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:51358/0 (socket says 172.21.15.132:51358) 2024-02-20T00:13:59.724 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:59.718+0000 7ff338bf7700 1 -- 172.21.15.132:0/2094177125 learned_addr learned my addr 172.21.15.132:0/2094177125 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:13:59.724 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:59.719+0000 7ff338bf7700 1 -- 172.21.15.132:0/2094177125 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff328005040 con 0x7ff3340f6480 2024-02-20T00:13:59.724 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:59.719+0000 7ff338bf7700 1 --2- 172.21.15.132:0/2094177125 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff3340f6480 0x7ff33406b9a0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7ff328005010 tx=0x7ff32800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:13:59.724 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:59.719+0000 7ff3317fa700 1 -- 172.21.15.132:0/2094177125 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff328009320 con 0x7ff3340f6480 2024-02-20T00:13:59.724 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:59.719+0000 7ff3317fa700 1 -- 172.21.15.132:0/2094177125 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff328013380 con 0x7ff3340f6480 2024-02-20T00:13:59.724 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:59.719+0000 7ff33ae5b700 1 -- 172.21.15.132:0/2094177125 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff334069fc0 con 0x7ff3340f6480 2024-02-20T00:13:59.724 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:59.719+0000 7ff3317fa700 1 -- 172.21.15.132:0/2094177125 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff328014e50 con 0x7ff3340f6480 2024-02-20T00:13:59.724 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:59.719+0000 7ff33ae5b700 1 -- 172.21.15.132:0/2094177125 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff33406a450 con 0x7ff3340f6480 2024-02-20T00:13:59.725 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:59.720+0000 7ff3317fa700 1 -- 172.21.15.132:0/2094177125 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff3280263a0 con 0x7ff3340f6480 2024-02-20T00:13:59.725 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:59.720+0000 7ff33ae5b700 1 -- 172.21.15.132:0/2094177125 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff334059100 con 0x7ff3340f6480 2024-02-20T00:13:59.725 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:59.720+0000 7ff3317fa700 1 --2- 172.21.15.132:0/2094177125 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7ff32402ff10 0x7ff3240323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:13:59.725 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:59.720+0000 7ff3317fa700 1 -- 172.21.15.132:0/2094177125 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff328041250 con 0x7ff3340f6480 2024-02-20T00:13:59.725 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:59.720+0000 7ff333fff700 1 --2- 172.21.15.132:0/2094177125 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7ff32402ff10 0x7ff3240323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:13:59.725 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:59.721+0000 7ff333fff700 1 --2- 172.21.15.132:0/2094177125 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7ff32402ff10 0x7ff3240323d0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7ff3200096a0 tx=0x7ff320006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:13:59.727 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:59.725+0000 7ff3317fa700 1 -- 172.21.15.132:0/2094177125 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff328026e60 con 0x7ff3340f6480 2024-02-20T00:13:59.932 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:59.930+0000 7ff33ae5b700 1 -- 172.21.15.132:0/2094177125 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff3340fc090 con 0x7ff3340f6480 2024-02-20T00:13:59.934 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:59.932+0000 7ff3317fa700 1 -- 172.21.15.132:0/2094177125 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff328012330 con 0x7ff3340f6480 2024-02-20T00:13:59.934 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:13:59.934 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:13:59.935 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:59.933+0000 7ff33ae5b700 1 -- 172.21.15.132:0/2094177125 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7ff32402ff10 msgr2=0x7ff3240323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:13:59.935 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:59.933+0000 7ff33ae5b700 1 --2- 172.21.15.132:0/2094177125 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7ff32402ff10 0x7ff3240323d0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7ff3200096a0 tx=0x7ff320006b40).stop 2024-02-20T00:13:59.936 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:59.933+0000 7ff33ae5b700 1 -- 172.21.15.132:0/2094177125 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff3340f6480 msgr2=0x7ff33406b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:13:59.936 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:59.933+0000 7ff33ae5b700 1 --2- 172.21.15.132:0/2094177125 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff3340f6480 0x7ff33406b9a0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7ff328005010 tx=0x7ff32800b040).stop 2024-02-20T00:13:59.936 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:59.933+0000 7ff33ae5b700 1 -- 172.21.15.132:0/2094177125 shutdown_connections 2024-02-20T00:13:59.936 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:59.933+0000 7ff33ae5b700 1 --2- 172.21.15.132:0/2094177125 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7ff32402ff10 0x7ff3240323d0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:13:59.936 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:59.933+0000 7ff33ae5b700 1 --2- 172.21.15.132:0/2094177125 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff3340f6480 0x7ff33406b9a0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:13:59.936 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:59.933+0000 7ff33ae5b700 1 -- 172.21.15.132:0/2094177125 >> 172.21.15.132:0/2094177125 conn(0x7ff3340f1c00 msgr2=0x7ff3340f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:13:59.936 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:59.933+0000 7ff33ae5b700 1 -- 172.21.15.132:0/2094177125 shutdown_connections 2024-02-20T00:13:59.936 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:13:59.933+0000 7ff33ae5b700 1 -- 172.21.15.132:0/2094177125 wait complete. 2024-02-20T00:13:59.936 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:14:00.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:13:59 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:13:58.873637+0000 mgr.smithi084.bmqrnf (mgr.14162) 122 : cluster [DBG] pgmap v115: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:14:01.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:14:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:13:59.933178+0000 mon.smithi084 (mon.0) 181 : audit [DBG] from='client.? 172.21.15.132:0/2094177125' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:14:01.515 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:14:01.515 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:14:02.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:14:01 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:14:00.873937+0000 mgr.smithi084.bmqrnf (mgr.14162) 123 : cluster [DBG] pgmap v116: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:14:03.427 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:03.425+0000 7f547b90c700 1 -- 172.21.15.132:0/1345049397 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f54740f6510 msgr2=0x7f54740f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:14:03.427 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:03.425+0000 7f547b90c700 1 --2- 172.21.15.132:0/1345049397 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f54740f6510 0x7f54740f6930 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 rx=0x7f5464009fb0 tx=0x7f5464009b30).stop 2024-02-20T00:14:03.428 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:03.425+0000 7f547b90c700 1 -- 172.21.15.132:0/1345049397 shutdown_connections 2024-02-20T00:14:03.428 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:03.425+0000 7f547b90c700 1 --2- 172.21.15.132:0/1345049397 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f54740f6510 0x7f54740f6930 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:14:03.428 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:03.425+0000 7f547b90c700 1 -- 172.21.15.132:0/1345049397 >> 172.21.15.132:0/1345049397 conn(0x7f54740f1c70 msgr2=0x7f54740f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:14:03.428 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:03.425+0000 7f547b90c700 1 -- 172.21.15.132:0/1345049397 shutdown_connections 2024-02-20T00:14:03.428 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:03.425+0000 7f547b90c700 1 -- 172.21.15.132:0/1345049397 wait complete. 2024-02-20T00:14:03.428 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:03.426+0000 7f547b90c700 1 Processor -- start 2024-02-20T00:14:03.428 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:03.426+0000 7f547b90c700 1 -- start start 2024-02-20T00:14:03.428 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:03.426+0000 7f547b90c700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f54740f6510 0x7f54741067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:14:03.428 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:03.426+0000 7f547b90c700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5474106cd0 con 0x7f54740f6510 2024-02-20T00:14:03.428 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:03.426+0000 7f54796a8700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f54740f6510 0x7f54741067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:14:03.428 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:03.426+0000 7f54796a8700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f54740f6510 0x7f54741067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:51376/0 (socket says 172.21.15.132:51376) 2024-02-20T00:14:03.428 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:03.426+0000 7f54796a8700 1 -- 172.21.15.132:0/880351578 learned_addr learned my addr 172.21.15.132:0/880351578 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:14:03.430 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:03.427+0000 7f54796a8700 1 -- 172.21.15.132:0/880351578 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5464005040 con 0x7f54740f6510 2024-02-20T00:14:03.430 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:03.427+0000 7f54796a8700 1 --2- 172.21.15.132:0/880351578 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f54740f6510 0x7f54741067e0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7f54640089f0 tx=0x7f546400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:14:03.430 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:03.427+0000 7f5469ffb700 1 -- 172.21.15.132:0/880351578 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5464009320 con 0x7f54740f6510 2024-02-20T00:14:03.430 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:03.427+0000 7f547b90c700 1 -- 172.21.15.132:0/880351578 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5474106ed0 con 0x7f54740f6510 2024-02-20T00:14:03.430 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:03.427+0000 7f5469ffb700 1 -- 172.21.15.132:0/880351578 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5464013380 con 0x7f54740f6510 2024-02-20T00:14:03.430 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:03.427+0000 7f5469ffb700 1 -- 172.21.15.132:0/880351578 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f546401ed80 con 0x7f54740f6510 2024-02-20T00:14:03.430 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:03.427+0000 7f547b90c700 1 -- 172.21.15.132:0/880351578 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5474107360 con 0x7f54740f6510 2024-02-20T00:14:03.430 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:03.428+0000 7f547b90c700 1 -- 172.21.15.132:0/880351578 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5474045ad0 con 0x7f54740f6510 2024-02-20T00:14:03.431 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:03.428+0000 7f5469ffb700 1 -- 172.21.15.132:0/880351578 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f546401e420 con 0x7f54740f6510 2024-02-20T00:14:03.431 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:03.428+0000 7f5469ffb700 1 --2- 172.21.15.132:0/880351578 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f546002ff60 0x7f5460032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:14:03.431 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:03.428+0000 7f5469ffb700 1 -- 172.21.15.132:0/880351578 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5464040f00 con 0x7f54740f6510 2024-02-20T00:14:03.431 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:03.428+0000 7f5478ea7700 1 --2- 172.21.15.132:0/880351578 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f546002ff60 0x7f5460032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:14:03.431 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:03.429+0000 7f5478ea7700 1 --2- 172.21.15.132:0/880351578 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f546002ff60 0x7f5460032420 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f54700096a0 tx=0x7f5470006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:14:03.434 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:03.433+0000 7f5469ffb700 1 -- 172.21.15.132:0/880351578 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f54640095f0 con 0x7f54740f6510 2024-02-20T00:14:03.639 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:03.637+0000 7f547b90c700 1 -- 172.21.15.132:0/880351578 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5474108210 con 0x7f54740f6510 2024-02-20T00:14:03.639 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:03.638+0000 7f5469ffb700 1 -- 172.21.15.132:0/880351578 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f546401cb10 con 0x7f54740f6510 2024-02-20T00:14:03.640 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:14:03.640 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:14:03.641 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:03.640+0000 7f547b90c700 1 -- 172.21.15.132:0/880351578 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f546002ff60 msgr2=0x7f5460032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:14:03.641 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:03.640+0000 7f547b90c700 1 --2- 172.21.15.132:0/880351578 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f546002ff60 0x7f5460032420 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f54700096a0 tx=0x7f5470006b40).stop 2024-02-20T00:14:03.641 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:03.640+0000 7f547b90c700 1 -- 172.21.15.132:0/880351578 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f54740f6510 msgr2=0x7f54741067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:14:03.641 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:03.640+0000 7f547b90c700 1 --2- 172.21.15.132:0/880351578 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f54740f6510 0x7f54741067e0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7f54640089f0 tx=0x7f546400b040).stop 2024-02-20T00:14:03.642 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:03.640+0000 7f547b90c700 1 -- 172.21.15.132:0/880351578 shutdown_connections 2024-02-20T00:14:03.642 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:03.640+0000 7f547b90c700 1 --2- 172.21.15.132:0/880351578 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f546002ff60 0x7f5460032420 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:14:03.642 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:03.640+0000 7f547b90c700 1 --2- 172.21.15.132:0/880351578 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f54740f6510 0x7f54741067e0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:14:03.642 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:03.640+0000 7f547b90c700 1 -- 172.21.15.132:0/880351578 >> 172.21.15.132:0/880351578 conn(0x7f54740f1c70 msgr2=0x7f54740f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:14:03.642 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:03.640+0000 7f547b90c700 1 -- 172.21.15.132:0/880351578 shutdown_connections 2024-02-20T00:14:03.642 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:03.640+0000 7f547b90c700 1 -- 172.21.15.132:0/880351578 wait complete. 2024-02-20T00:14:03.642 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:14:04.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:14:03 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:14:02.874186+0000 mgr.smithi084.bmqrnf (mgr.14162) 124 : cluster [DBG] pgmap v117: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:14:04.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:14:03 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:14:03.638770+0000 mon.smithi084 (mon.0) 182 : audit [DBG] from='client.? 172.21.15.132:0/880351578' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:14:05.282 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:14:05.283 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:14:06.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:14:05 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:14:04.874557+0000 mgr.smithi084.bmqrnf (mgr.14162) 125 : cluster [DBG] pgmap v118: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:14:07.062 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:07.059+0000 7fd585d41700 1 -- 172.21.15.132:0/191103019 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd5800f6510 msgr2=0x7fd5800f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:14:07.062 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:07.059+0000 7fd585d41700 1 --2- 172.21.15.132:0/191103019 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd5800f6510 0x7fd5800f6930 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0x7fd570009fb0 tx=0x7fd570009b30).stop 2024-02-20T00:14:07.062 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:07.059+0000 7fd585d41700 1 -- 172.21.15.132:0/191103019 shutdown_connections 2024-02-20T00:14:07.062 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:07.059+0000 7fd585d41700 1 --2- 172.21.15.132:0/191103019 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd5800f6510 0x7fd5800f6930 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:14:07.062 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:07.059+0000 7fd585d41700 1 -- 172.21.15.132:0/191103019 >> 172.21.15.132:0/191103019 conn(0x7fd5800f1c70 msgr2=0x7fd5800f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:14:07.062 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:07.059+0000 7fd585d41700 1 -- 172.21.15.132:0/191103019 shutdown_connections 2024-02-20T00:14:07.062 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:07.059+0000 7fd585d41700 1 -- 172.21.15.132:0/191103019 wait complete. 2024-02-20T00:14:07.063 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:07.059+0000 7fd585d41700 1 Processor -- start 2024-02-20T00:14:07.063 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:07.060+0000 7fd585d41700 1 -- start start 2024-02-20T00:14:07.063 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:07.060+0000 7fd585d41700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd5800f6510 0x7fd5801067c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:14:07.063 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:07.060+0000 7fd585d41700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd580106cb0 con 0x7fd5800f6510 2024-02-20T00:14:07.063 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:07.060+0000 7fd57f7fe700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd5800f6510 0x7fd5801067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:14:07.063 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:07.060+0000 7fd57f7fe700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd5800f6510 0x7fd5801067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:51392/0 (socket says 172.21.15.132:51392) 2024-02-20T00:14:07.063 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:07.060+0000 7fd57f7fe700 1 -- 172.21.15.132:0/3721004863 learned_addr learned my addr 172.21.15.132:0/3721004863 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:14:07.063 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:07.060+0000 7fd57f7fe700 1 -- 172.21.15.132:0/3721004863 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd570005040 con 0x7fd5800f6510 2024-02-20T00:14:07.063 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:07.060+0000 7fd57f7fe700 1 --2- 172.21.15.132:0/3721004863 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd5800f6510 0x7fd5801067c0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7fd570005010 tx=0x7fd57000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:14:07.063 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:07.061+0000 7fd567fff700 1 -- 172.21.15.132:0/3721004863 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd570009320 con 0x7fd5800f6510 2024-02-20T00:14:07.063 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:07.061+0000 7fd585d41700 1 -- 172.21.15.132:0/3721004863 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd580106eb0 con 0x7fd5800f6510 2024-02-20T00:14:07.063 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:07.061+0000 7fd567fff700 1 -- 172.21.15.132:0/3721004863 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd570013380 con 0x7fd5800f6510 2024-02-20T00:14:07.063 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:07.061+0000 7fd567fff700 1 -- 172.21.15.132:0/3721004863 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd570014e50 con 0x7fd5800f6510 2024-02-20T00:14:07.063 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:07.061+0000 7fd585d41700 1 -- 172.21.15.132:0/3721004863 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd580107340 con 0x7fd5800f6510 2024-02-20T00:14:07.064 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:07.061+0000 7fd567fff700 1 -- 172.21.15.132:0/3721004863 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd5700095f0 con 0x7fd5800f6510 2024-02-20T00:14:07.065 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:07.061+0000 7fd585d41700 1 -- 172.21.15.132:0/3721004863 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd580045ad0 con 0x7fd5800f6510 2024-02-20T00:14:07.065 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:07.062+0000 7fd567fff700 1 --2- 172.21.15.132:0/3721004863 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fd56802ff10 0x7fd5680323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:14:07.065 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:07.062+0000 7fd567fff700 1 -- 172.21.15.132:0/3721004863 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd570040e70 con 0x7fd5800f6510 2024-02-20T00:14:07.065 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:07.062+0000 7fd57effd700 1 --2- 172.21.15.132:0/3721004863 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fd56802ff10 0x7fd5680323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:14:07.065 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:07.062+0000 7fd57effd700 1 --2- 172.21.15.132:0/3721004863 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fd56802ff10 0x7fd5680323d0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7fd5740096a0 tx=0x7fd574006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:14:07.067 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:07.066+0000 7fd567fff700 1 -- 172.21.15.132:0/3721004863 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd570014420 con 0x7fd5800f6510 2024-02-20T00:14:07.272 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:07.271+0000 7fd585d41700 1 -- 172.21.15.132:0/3721004863 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd580108190 con 0x7fd5800f6510 2024-02-20T00:14:07.272 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:07.271+0000 7fd567fff700 1 -- 172.21.15.132:0/3721004863 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd570010230 con 0x7fd5800f6510 2024-02-20T00:14:07.273 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:14:07.273 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:14:07.274 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:07.273+0000 7fd585d41700 1 -- 172.21.15.132:0/3721004863 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fd56802ff10 msgr2=0x7fd5680323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:14:07.274 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:07.273+0000 7fd585d41700 1 --2- 172.21.15.132:0/3721004863 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fd56802ff10 0x7fd5680323d0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7fd5740096a0 tx=0x7fd574006b40).stop 2024-02-20T00:14:07.274 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:07.273+0000 7fd585d41700 1 -- 172.21.15.132:0/3721004863 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd5800f6510 msgr2=0x7fd5801067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:14:07.274 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:07.273+0000 7fd585d41700 1 --2- 172.21.15.132:0/3721004863 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd5800f6510 0x7fd5801067c0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7fd570005010 tx=0x7fd57000b040).stop 2024-02-20T00:14:07.275 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:07.273+0000 7fd585d41700 1 -- 172.21.15.132:0/3721004863 shutdown_connections 2024-02-20T00:14:07.275 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:07.273+0000 7fd585d41700 1 --2- 172.21.15.132:0/3721004863 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fd56802ff10 0x7fd5680323d0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:14:07.275 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:07.273+0000 7fd585d41700 1 --2- 172.21.15.132:0/3721004863 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd5800f6510 0x7fd5801067c0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:14:07.275 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:07.273+0000 7fd585d41700 1 -- 172.21.15.132:0/3721004863 >> 172.21.15.132:0/3721004863 conn(0x7fd5800f1c70 msgr2=0x7fd5800f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:14:07.275 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:07.273+0000 7fd585d41700 1 -- 172.21.15.132:0/3721004863 shutdown_connections 2024-02-20T00:14:07.275 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:07.273+0000 7fd585d41700 1 -- 172.21.15.132:0/3721004863 wait complete. 2024-02-20T00:14:07.275 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:14:08.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:14:07 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:14:06.874876+0000 mgr.smithi084.bmqrnf (mgr.14162) 126 : cluster [DBG] pgmap v119: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:14:08.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:14:07 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:14:07.272015+0000 mon.smithi084 (mon.0) 183 : audit [DBG] from='client.? 172.21.15.132:0/3721004863' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:14:08.843 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:14:08.843 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:14:10.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:14:09 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:14:08.875170+0000 mgr.smithi084.bmqrnf (mgr.14162) 127 : cluster [DBG] pgmap v120: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:14:10.634 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:10.631+0000 7f684ca1d700 1 -- 172.21.15.132:0/3642707766 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f68480f51a0 msgr2=0x7f68480f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:14:10.634 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:10.631+0000 7f684ca1d700 1 --2- 172.21.15.132:0/3642707766 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f68480f51a0 0x7f68480f55c0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0x7f6838009fb0 tx=0x7f6838009b30).stop 2024-02-20T00:14:10.634 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:10.631+0000 7f684ca1d700 1 -- 172.21.15.132:0/3642707766 shutdown_connections 2024-02-20T00:14:10.634 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:10.631+0000 7f684ca1d700 1 --2- 172.21.15.132:0/3642707766 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f68480f51a0 0x7f68480f55c0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:14:10.634 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:10.631+0000 7f684ca1d700 1 -- 172.21.15.132:0/3642707766 >> 172.21.15.132:0/3642707766 conn(0x7f68480f09a0 msgr2=0x7f68480f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:14:10.634 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:10.631+0000 7f684ca1d700 1 -- 172.21.15.132:0/3642707766 shutdown_connections 2024-02-20T00:14:10.634 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:10.631+0000 7f684ca1d700 1 -- 172.21.15.132:0/3642707766 wait complete. 2024-02-20T00:14:10.634 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:10.632+0000 7f684ca1d700 1 Processor -- start 2024-02-20T00:14:10.634 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:10.632+0000 7f684ca1d700 1 -- start start 2024-02-20T00:14:10.634 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:10.632+0000 7f684ca1d700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6848060200 0x7f6848109410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:14:10.634 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:10.632+0000 7f684ca1d700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6848109900 con 0x7f6848060200 2024-02-20T00:14:10.635 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:10.633+0000 7f68477fe700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6848060200 0x7f6848109410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:14:10.635 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:10.633+0000 7f68477fe700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6848060200 0x7f6848109410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:54964/0 (socket says 172.21.15.132:54964) 2024-02-20T00:14:10.635 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:10.633+0000 7f68477fe700 1 -- 172.21.15.132:0/2509030595 learned_addr learned my addr 172.21.15.132:0/2509030595 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:14:10.636 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:10.633+0000 7f68477fe700 1 -- 172.21.15.132:0/2509030595 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6838005040 con 0x7f6848060200 2024-02-20T00:14:10.636 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:10.633+0000 7f68477fe700 1 --2- 172.21.15.132:0/2509030595 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6848060200 0x7f6848109410 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7f6838009f80 tx=0x7f6838009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:14:10.636 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:10.633+0000 7f682ffff700 1 -- 172.21.15.132:0/2509030595 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6838013380 con 0x7f6848060200 2024-02-20T00:14:10.636 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:10.633+0000 7f684ca1d700 1 -- 172.21.15.132:0/2509030595 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6848109a40 con 0x7f6848060200 2024-02-20T00:14:10.637 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:10.633+0000 7f682ffff700 1 -- 172.21.15.132:0/2509030595 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6838013790 con 0x7f6848060200 2024-02-20T00:14:10.637 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:10.633+0000 7f682ffff700 1 -- 172.21.15.132:0/2509030595 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6838014420 con 0x7f6848060200 2024-02-20T00:14:10.637 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:10.633+0000 7f684ca1d700 1 -- 172.21.15.132:0/2509030595 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6848109e70 con 0x7f6848060200 2024-02-20T00:14:10.637 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:10.634+0000 7f682ffff700 1 -- 172.21.15.132:0/2509030595 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6838025060 con 0x7f6848060200 2024-02-20T00:14:10.637 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:10.634+0000 7f684ca1d700 1 -- 172.21.15.132:0/2509030595 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6848045ad0 con 0x7f6848060200 2024-02-20T00:14:10.637 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:10.634+0000 7f682ffff700 1 --2- 172.21.15.132:0/2509030595 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f683002ff60 0x7f6830032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:14:10.637 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:10.634+0000 7f682ffff700 1 -- 172.21.15.132:0/2509030595 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6838040e00 con 0x7f6848060200 2024-02-20T00:14:10.637 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:10.634+0000 7f6846ffd700 1 --2- 172.21.15.132:0/2509030595 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f683002ff60 0x7f6830032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:14:10.637 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:10.635+0000 7f6846ffd700 1 --2- 172.21.15.132:0/2509030595 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f683002ff60 0x7f6830032420 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7f683c0096a0 tx=0x7f683c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:14:10.640 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:10.639+0000 7f682ffff700 1 -- 172.21.15.132:0/2509030595 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f683800f280 con 0x7f6848060200 2024-02-20T00:14:10.845 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:10.844+0000 7f684ca1d700 1 -- 172.21.15.132:0/2509030595 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f68480fad30 con 0x7f6848060200 2024-02-20T00:14:10.845 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:10.844+0000 7f682ffff700 1 -- 172.21.15.132:0/2509030595 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f683800f280 con 0x7f6848060200 2024-02-20T00:14:10.846 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:14:10.846 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:14:10.847 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:10.846+0000 7f684ca1d700 1 -- 172.21.15.132:0/2509030595 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f683002ff60 msgr2=0x7f6830032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:14:10.847 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:10.846+0000 7f684ca1d700 1 --2- 172.21.15.132:0/2509030595 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f683002ff60 0x7f6830032420 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7f683c0096a0 tx=0x7f683c006b40).stop 2024-02-20T00:14:10.847 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:10.846+0000 7f684ca1d700 1 -- 172.21.15.132:0/2509030595 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6848060200 msgr2=0x7f6848109410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:14:10.847 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:10.846+0000 7f684ca1d700 1 --2- 172.21.15.132:0/2509030595 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6848060200 0x7f6848109410 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7f6838009f80 tx=0x7f6838009b30).stop 2024-02-20T00:14:10.848 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:10.846+0000 7f684ca1d700 1 -- 172.21.15.132:0/2509030595 shutdown_connections 2024-02-20T00:14:10.848 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:10.846+0000 7f684ca1d700 1 --2- 172.21.15.132:0/2509030595 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f683002ff60 0x7f6830032420 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:14:10.848 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:10.846+0000 7f684ca1d700 1 --2- 172.21.15.132:0/2509030595 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6848060200 0x7f6848109410 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:14:10.848 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:10.846+0000 7f684ca1d700 1 -- 172.21.15.132:0/2509030595 >> 172.21.15.132:0/2509030595 conn(0x7f68480f09a0 msgr2=0x7f68480f14b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:14:10.848 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:10.846+0000 7f684ca1d700 1 -- 172.21.15.132:0/2509030595 shutdown_connections 2024-02-20T00:14:10.848 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:10.846+0000 7f684ca1d700 1 -- 172.21.15.132:0/2509030595 wait complete. 2024-02-20T00:14:10.848 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:14:11.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:14:10 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:14:10.844952+0000 mon.smithi084 (mon.0) 184 : audit [DBG] from='client.? 172.21.15.132:0/2509030595' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:14:12.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:14:11 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:14:10.875424+0000 mgr.smithi084.bmqrnf (mgr.14162) 128 : cluster [DBG] pgmap v121: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:14:12.550 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:14:12.551 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:14:14.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:14:13 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:14:12.875606+0000 mgr.smithi084.bmqrnf (mgr.14162) 129 : cluster [DBG] pgmap v122: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:14:14.462 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:14.460+0000 7f93beac6700 1 -- 172.21.15.132:0/4051839581 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f93b80f5410 msgr2=0x7f93b80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:14:14.462 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:14.460+0000 7f93beac6700 1 --2- 172.21.15.132:0/4051839581 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f93b80f5410 0x7f93b80f5830 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 rx=0x7f93ac009fb0 tx=0x7f93ac009b30).stop 2024-02-20T00:14:14.463 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:14.460+0000 7f93beac6700 1 -- 172.21.15.132:0/4051839581 shutdown_connections 2024-02-20T00:14:14.463 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:14.460+0000 7f93beac6700 1 --2- 172.21.15.132:0/4051839581 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f93b80f5410 0x7f93b80f5830 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:14:14.463 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:14.460+0000 7f93beac6700 1 -- 172.21.15.132:0/4051839581 >> 172.21.15.132:0/4051839581 conn(0x7f93b80f0b70 msgr2=0x7f93b80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:14:14.463 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:14.460+0000 7f93beac6700 1 -- 172.21.15.132:0/4051839581 shutdown_connections 2024-02-20T00:14:14.463 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:14.461+0000 7f93beac6700 1 -- 172.21.15.132:0/4051839581 wait complete. 2024-02-20T00:14:14.463 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:14.461+0000 7f93beac6700 1 Processor -- start 2024-02-20T00:14:14.463 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:14.461+0000 7f93beac6700 1 -- start start 2024-02-20T00:14:14.463 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:14.461+0000 7f93beac6700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f93b80f5410 0x7f93b810b940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:14:14.463 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:14.461+0000 7f93beac6700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93b8107330 con 0x7f93b80f5410 2024-02-20T00:14:14.465 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:14.462+0000 7f93bc862700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f93b80f5410 0x7f93b810b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:14:14.465 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:14.462+0000 7f93bc862700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f93b80f5410 0x7f93b810b940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:54982/0 (socket says 172.21.15.132:54982) 2024-02-20T00:14:14.465 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:14.462+0000 7f93bc862700 1 -- 172.21.15.132:0/1085808367 learned_addr learned my addr 172.21.15.132:0/1085808367 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:14:14.465 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:14.462+0000 7f93bc862700 1 -- 172.21.15.132:0/1085808367 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93ac005040 con 0x7f93b80f5410 2024-02-20T00:14:14.465 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:14.462+0000 7f93bc862700 1 --2- 172.21.15.132:0/1085808367 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f93b80f5410 0x7f93b810b940 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7f93ac0089f0 tx=0x7f93ac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:14:14.465 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:14.462+0000 7f93b57fa700 1 -- 172.21.15.132:0/1085808367 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f93ac009320 con 0x7f93b80f5410 2024-02-20T00:14:14.465 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:14.462+0000 7f93b57fa700 1 -- 172.21.15.132:0/1085808367 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f93ac013380 con 0x7f93b80f5410 2024-02-20T00:14:14.465 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:14.462+0000 7f93beac6700 1 -- 172.21.15.132:0/1085808367 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93b8060220 con 0x7f93b80f5410 2024-02-20T00:14:14.465 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:14.462+0000 7f93b57fa700 1 -- 172.21.15.132:0/1085808367 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f93ac01ed80 con 0x7f93b80f5410 2024-02-20T00:14:14.465 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:14.462+0000 7f93beac6700 1 -- 172.21.15.132:0/1085808367 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93b810be30 con 0x7f93b80f5410 2024-02-20T00:14:14.465 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:14.463+0000 7f93b57fa700 1 -- 172.21.15.132:0/1085808367 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f93ac01e420 con 0x7f93b80f5410 2024-02-20T00:14:14.466 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:14.463+0000 7f93beac6700 1 -- 172.21.15.132:0/1085808367 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93b8045ad0 con 0x7f93b80f5410 2024-02-20T00:14:14.466 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:14.463+0000 7f93b57fa700 1 --2- 172.21.15.132:0/1085808367 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f93a802ff60 0x7f93a8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:14:14.466 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:14.463+0000 7f93b57fa700 1 -- 172.21.15.132:0/1085808367 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f93ac040ef0 con 0x7f93b80f5410 2024-02-20T00:14:14.466 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:14.463+0000 7f93b7fff700 1 --2- 172.21.15.132:0/1085808367 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f93a802ff60 0x7f93a8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:14:14.466 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:14.464+0000 7f93b7fff700 1 --2- 172.21.15.132:0/1085808367 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f93a802ff60 0x7f93a8032420 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f93a40096a0 tx=0x7f93a4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:14:14.469 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:14.468+0000 7f93b57fa700 1 -- 172.21.15.132:0/1085808367 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f93ac00e2a0 con 0x7f93b80f5410 2024-02-20T00:14:14.674 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:14.672+0000 7f93beac6700 1 -- 172.21.15.132:0/1085808367 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f93b810c5f0 con 0x7f93b80f5410 2024-02-20T00:14:14.674 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:14.673+0000 7f93b57fa700 1 -- 172.21.15.132:0/1085808367 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f93ac00e2a0 con 0x7f93b80f5410 2024-02-20T00:14:14.674 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:14:14.675 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:14:14.676 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:14.675+0000 7f93beac6700 1 -- 172.21.15.132:0/1085808367 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f93a802ff60 msgr2=0x7f93a8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:14:14.676 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:14.675+0000 7f93beac6700 1 --2- 172.21.15.132:0/1085808367 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f93a802ff60 0x7f93a8032420 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f93a40096a0 tx=0x7f93a4006b40).stop 2024-02-20T00:14:14.676 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:14.675+0000 7f93beac6700 1 -- 172.21.15.132:0/1085808367 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f93b80f5410 msgr2=0x7f93b810b940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:14:14.676 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:14.675+0000 7f93beac6700 1 --2- 172.21.15.132:0/1085808367 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f93b80f5410 0x7f93b810b940 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7f93ac0089f0 tx=0x7f93ac00b040).stop 2024-02-20T00:14:14.676 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:14.675+0000 7f93beac6700 1 -- 172.21.15.132:0/1085808367 shutdown_connections 2024-02-20T00:14:14.676 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:14.675+0000 7f93beac6700 1 --2- 172.21.15.132:0/1085808367 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f93a802ff60 0x7f93a8032420 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:14:14.677 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:14.675+0000 7f93beac6700 1 --2- 172.21.15.132:0/1085808367 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f93b80f5410 0x7f93b810b940 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:14:14.677 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:14.675+0000 7f93beac6700 1 -- 172.21.15.132:0/1085808367 >> 172.21.15.132:0/1085808367 conn(0x7f93b80f0b70 msgr2=0x7f93b80f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:14:14.677 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:14.675+0000 7f93beac6700 1 -- 172.21.15.132:0/1085808367 shutdown_connections 2024-02-20T00:14:14.677 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:14.675+0000 7f93beac6700 1 -- 172.21.15.132:0/1085808367 wait complete. 2024-02-20T00:14:14.677 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:14:15.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:14:14 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:14:14.673674+0000 mon.smithi084 (mon.0) 185 : audit [DBG] from='client.? 172.21.15.132:0/1085808367' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:14:16.294 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:14:16.295 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:14:16.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:14:15 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:14:14.875923+0000 mgr.smithi084.bmqrnf (mgr.14162) 130 : cluster [DBG] pgmap v123: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:14:18.079 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:18.076+0000 7f6d6a784700 1 -- 172.21.15.132:0/1789361086 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6d640f51a0 msgr2=0x7f6d640f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:14:18.079 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:18.076+0000 7f6d6a784700 1 --2- 172.21.15.132:0/1789361086 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6d640f51a0 0x7f6d640f55c0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 rx=0x7f6d60009fb0 tx=0x7f6d60009b30).stop 2024-02-20T00:14:18.079 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:18.076+0000 7f6d6a784700 1 -- 172.21.15.132:0/1789361086 shutdown_connections 2024-02-20T00:14:18.079 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:18.076+0000 7f6d6a784700 1 --2- 172.21.15.132:0/1789361086 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6d640f51a0 0x7f6d640f55c0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:14:18.079 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:18.076+0000 7f6d6a784700 1 -- 172.21.15.132:0/1789361086 >> 172.21.15.132:0/1789361086 conn(0x7f6d640f09a0 msgr2=0x7f6d640f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:14:18.079 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:18.076+0000 7f6d6a784700 1 -- 172.21.15.132:0/1789361086 shutdown_connections 2024-02-20T00:14:18.079 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:18.076+0000 7f6d6a784700 1 -- 172.21.15.132:0/1789361086 wait complete. 2024-02-20T00:14:18.079 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:18.076+0000 7f6d6a784700 1 Processor -- start 2024-02-20T00:14:18.080 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:18.076+0000 7f6d6a784700 1 -- start start 2024-02-20T00:14:18.080 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:18.077+0000 7f6d6a784700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6d640f51a0 0x7f6d641054d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:14:18.080 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:18.077+0000 7f6d6a784700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d641059c0 con 0x7f6d640f51a0 2024-02-20T00:14:18.080 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:18.077+0000 7f6d69782700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6d640f51a0 0x7f6d641054d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:14:18.080 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:18.077+0000 7f6d69782700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6d640f51a0 0x7f6d641054d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:45920/0 (socket says 172.21.15.132:45920) 2024-02-20T00:14:18.080 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:18.077+0000 7f6d69782700 1 -- 172.21.15.132:0/2281309728 learned_addr learned my addr 172.21.15.132:0/2281309728 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:14:18.080 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:18.077+0000 7f6d69782700 1 -- 172.21.15.132:0/2281309728 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d60005040 con 0x7f6d640f51a0 2024-02-20T00:14:18.080 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:18.077+0000 7f6d69782700 1 --2- 172.21.15.132:0/2281309728 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6d640f51a0 0x7f6d641054d0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7f6d600089f0 tx=0x7f6d6000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:14:18.080 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:18.077+0000 7f6d59ffb700 1 -- 172.21.15.132:0/2281309728 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6d60009320 con 0x7f6d640f51a0 2024-02-20T00:14:18.080 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:18.077+0000 7f6d6a784700 1 -- 172.21.15.132:0/2281309728 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d64105bc0 con 0x7f6d640f51a0 2024-02-20T00:14:18.080 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:18.077+0000 7f6d59ffb700 1 -- 172.21.15.132:0/2281309728 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6d60013380 con 0x7f6d640f51a0 2024-02-20T00:14:18.080 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:18.077+0000 7f6d59ffb700 1 -- 172.21.15.132:0/2281309728 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6d60014e20 con 0x7f6d640f51a0 2024-02-20T00:14:18.080 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:18.078+0000 7f6d6a784700 1 -- 172.21.15.132:0/2281309728 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d64106050 con 0x7f6d640f51a0 2024-02-20T00:14:18.081 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:18.078+0000 7f6d59ffb700 1 -- 172.21.15.132:0/2281309728 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6d600095f0 con 0x7f6d640f51a0 2024-02-20T00:14:18.081 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:18.078+0000 7f6d6a784700 1 -- 172.21.15.132:0/2281309728 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d64045ad0 con 0x7f6d640f51a0 2024-02-20T00:14:18.082 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:18.078+0000 7f6d59ffb700 1 --2- 172.21.15.132:0/2281309728 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f6d5002ff10 0x7f6d500323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:14:18.082 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:18.079+0000 7f6d59ffb700 1 -- 172.21.15.132:0/2281309728 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6d60040ed0 con 0x7f6d640f51a0 2024-02-20T00:14:18.082 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:18.079+0000 7f6d68f81700 1 --2- 172.21.15.132:0/2281309728 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f6d5002ff10 0x7f6d500323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:14:18.082 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:18.079+0000 7f6d68f81700 1 --2- 172.21.15.132:0/2281309728 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f6d5002ff10 0x7f6d500323d0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f6d540096a0 tx=0x7f6d54006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:14:18.084 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:18.083+0000 7f6d59ffb700 1 -- 172.21.15.132:0/2281309728 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6d60014420 con 0x7f6d640f51a0 2024-02-20T00:14:18.289 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:18.287+0000 7f6d6a784700 1 -- 172.21.15.132:0/2281309728 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6d64106f00 con 0x7f6d640f51a0 2024-02-20T00:14:18.289 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:18.288+0000 7f6d59ffb700 1 -- 172.21.15.132:0/2281309728 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6d6001d980 con 0x7f6d640f51a0 2024-02-20T00:14:18.290 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:14:18.290 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:14:18.291 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:18.290+0000 7f6d6a784700 1 -- 172.21.15.132:0/2281309728 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f6d5002ff10 msgr2=0x7f6d500323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:14:18.291 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:18.290+0000 7f6d6a784700 1 --2- 172.21.15.132:0/2281309728 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f6d5002ff10 0x7f6d500323d0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f6d540096a0 tx=0x7f6d54006b40).stop 2024-02-20T00:14:18.292 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:18.290+0000 7f6d6a784700 1 -- 172.21.15.132:0/2281309728 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6d640f51a0 msgr2=0x7f6d641054d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:14:18.292 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:18.290+0000 7f6d6a784700 1 --2- 172.21.15.132:0/2281309728 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6d640f51a0 0x7f6d641054d0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7f6d600089f0 tx=0x7f6d6000b040).stop 2024-02-20T00:14:18.292 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:18.290+0000 7f6d6a784700 1 -- 172.21.15.132:0/2281309728 shutdown_connections 2024-02-20T00:14:18.292 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:18.290+0000 7f6d6a784700 1 --2- 172.21.15.132:0/2281309728 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f6d5002ff10 0x7f6d500323d0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:14:18.292 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:18.290+0000 7f6d6a784700 1 --2- 172.21.15.132:0/2281309728 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6d640f51a0 0x7f6d641054d0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:14:18.292 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:18.290+0000 7f6d6a784700 1 -- 172.21.15.132:0/2281309728 >> 172.21.15.132:0/2281309728 conn(0x7f6d640f09a0 msgr2=0x7f6d640f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:14:18.292 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:18.290+0000 7f6d6a784700 1 -- 172.21.15.132:0/2281309728 shutdown_connections 2024-02-20T00:14:18.292 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:18.290+0000 7f6d6a784700 1 -- 172.21.15.132:0/2281309728 wait complete. 2024-02-20T00:14:18.292 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:14:18.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:14:17 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:14:16.876292+0000 mgr.smithi084.bmqrnf (mgr.14162) 131 : cluster [DBG] pgmap v124: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:14:19.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:14:18 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:14:18.288723+0000 mon.smithi084 (mon.0) 186 : audit [DBG] from='client.? 172.21.15.132:0/2281309728' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:14:19.940 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:14:19.941 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:14:20.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:14:19 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:14:18.876634+0000 mgr.smithi084.bmqrnf (mgr.14162) 132 : cluster [DBG] pgmap v125: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:14:21.792 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:21.789+0000 7fc06e665700 1 -- 172.21.15.132:0/1070399978 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc0680f6510 msgr2=0x7fc0680f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:14:21.792 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:21.789+0000 7fc06e665700 1 --2- 172.21.15.132:0/1070399978 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc0680f6510 0x7fc0680f6930 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 rx=0x7fc05c009fb0 tx=0x7fc05c009b30).stop 2024-02-20T00:14:21.792 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:21.789+0000 7fc06e665700 1 -- 172.21.15.132:0/1070399978 shutdown_connections 2024-02-20T00:14:21.792 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:21.789+0000 7fc06e665700 1 --2- 172.21.15.132:0/1070399978 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc0680f6510 0x7fc0680f6930 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:14:21.792 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:21.789+0000 7fc06e665700 1 -- 172.21.15.132:0/1070399978 >> 172.21.15.132:0/1070399978 conn(0x7fc0680f1c70 msgr2=0x7fc0680f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:14:21.792 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:21.789+0000 7fc06e665700 1 -- 172.21.15.132:0/1070399978 shutdown_connections 2024-02-20T00:14:21.792 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:21.789+0000 7fc06e665700 1 -- 172.21.15.132:0/1070399978 wait complete. 2024-02-20T00:14:21.792 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:21.790+0000 7fc06e665700 1 Processor -- start 2024-02-20T00:14:21.792 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:21.790+0000 7fc06e665700 1 -- start start 2024-02-20T00:14:21.794 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:21.790+0000 7fc06e665700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc0680f6510 0x7fc06806b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:14:21.794 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:21.790+0000 7fc06e665700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc068069e20 con 0x7fc0680f6510 2024-02-20T00:14:21.794 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:21.790+0000 7fc067fff700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc0680f6510 0x7fc06806b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:14:21.794 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:21.790+0000 7fc067fff700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc0680f6510 0x7fc06806b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:45948/0 (socket says 172.21.15.132:45948) 2024-02-20T00:14:21.795 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:21.790+0000 7fc067fff700 1 -- 172.21.15.132:0/2024192184 learned_addr learned my addr 172.21.15.132:0/2024192184 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:14:21.795 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:21.791+0000 7fc067fff700 1 -- 172.21.15.132:0/2024192184 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc05c005040 con 0x7fc0680f6510 2024-02-20T00:14:21.795 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:21.791+0000 7fc067fff700 1 --2- 172.21.15.132:0/2024192184 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc0680f6510 0x7fc06806b9a0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7fc05c005010 tx=0x7fc05c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:14:21.795 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:21.791+0000 7fc064ff9700 1 -- 172.21.15.132:0/2024192184 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc05c009320 con 0x7fc0680f6510 2024-02-20T00:14:21.795 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:21.791+0000 7fc06e665700 1 -- 172.21.15.132:0/2024192184 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc068069fc0 con 0x7fc0680f6510 2024-02-20T00:14:21.795 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:21.791+0000 7fc064ff9700 1 -- 172.21.15.132:0/2024192184 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc05c013380 con 0x7fc0680f6510 2024-02-20T00:14:21.795 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:21.791+0000 7fc064ff9700 1 -- 172.21.15.132:0/2024192184 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc05c014e50 con 0x7fc0680f6510 2024-02-20T00:14:21.795 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:21.791+0000 7fc06e665700 1 -- 172.21.15.132:0/2024192184 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc06806a450 con 0x7fc0680f6510 2024-02-20T00:14:21.795 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:21.792+0000 7fc064ff9700 1 -- 172.21.15.132:0/2024192184 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc05c0095f0 con 0x7fc0680f6510 2024-02-20T00:14:21.795 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:21.792+0000 7fc06e665700 1 -- 172.21.15.132:0/2024192184 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc068059100 con 0x7fc0680f6510 2024-02-20T00:14:21.795 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:21.792+0000 7fc064ff9700 1 --2- 172.21.15.132:0/2024192184 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fc05002ff10 0x7fc0500323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:14:21.795 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:21.792+0000 7fc064ff9700 1 -- 172.21.15.132:0/2024192184 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc05c040e90 con 0x7fc0680f6510 2024-02-20T00:14:21.796 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:21.792+0000 7fc0677fe700 1 --2- 172.21.15.132:0/2024192184 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fc05002ff10 0x7fc0500323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:14:21.796 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:21.792+0000 7fc0677fe700 1 --2- 172.21.15.132:0/2024192184 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fc05002ff10 0x7fc0500323d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7fc0580096a0 tx=0x7fc058006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:14:21.798 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:21.796+0000 7fc064ff9700 1 -- 172.21.15.132:0/2024192184 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc05c0268c0 con 0x7fc0680f6510 2024-02-20T00:14:22.006 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:22.003+0000 7fc06e665700 1 -- 172.21.15.132:0/2024192184 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc0680fc0a0 con 0x7fc0680f6510 2024-02-20T00:14:22.006 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:22.004+0000 7fc064ff9700 1 -- 172.21.15.132:0/2024192184 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc05c010230 con 0x7fc0680f6510 2024-02-20T00:14:22.007 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:14:22.007 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:14:22.008 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:22.006+0000 7fc06e665700 1 -- 172.21.15.132:0/2024192184 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fc05002ff10 msgr2=0x7fc0500323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:14:22.008 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:22.006+0000 7fc06e665700 1 --2- 172.21.15.132:0/2024192184 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fc05002ff10 0x7fc0500323d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7fc0580096a0 tx=0x7fc058006b40).stop 2024-02-20T00:14:22.008 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:22.006+0000 7fc06e665700 1 -- 172.21.15.132:0/2024192184 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc0680f6510 msgr2=0x7fc06806b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:14:22.009 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:22.006+0000 7fc06e665700 1 --2- 172.21.15.132:0/2024192184 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc0680f6510 0x7fc06806b9a0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7fc05c005010 tx=0x7fc05c00b040).stop 2024-02-20T00:14:22.009 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:22.006+0000 7fc06e665700 1 -- 172.21.15.132:0/2024192184 shutdown_connections 2024-02-20T00:14:22.009 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:22.006+0000 7fc06e665700 1 --2- 172.21.15.132:0/2024192184 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fc05002ff10 0x7fc0500323d0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:14:22.009 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:22.006+0000 7fc06e665700 1 --2- 172.21.15.132:0/2024192184 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc0680f6510 0x7fc06806b9a0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:14:22.009 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:22.006+0000 7fc06e665700 1 -- 172.21.15.132:0/2024192184 >> 172.21.15.132:0/2024192184 conn(0x7fc0680f1c70 msgr2=0x7fc0680f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:14:22.009 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:22.006+0000 7fc06e665700 1 -- 172.21.15.132:0/2024192184 shutdown_connections 2024-02-20T00:14:22.009 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:22.006+0000 7fc06e665700 1 -- 172.21.15.132:0/2024192184 wait complete. 2024-02-20T00:14:22.010 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:14:22.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:14:21 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:14:20.876970+0000 mgr.smithi084.bmqrnf (mgr.14162) 133 : cluster [DBG] pgmap v126: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:14:23.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:14:22 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:14:22.005704+0000 mon.smithi084 (mon.0) 187 : audit [DBG] from='client.? 172.21.15.132:0/2024192184' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:14:23.568 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:14:23.568 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:14:24.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:14:23 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:14:22.877305+0000 mgr.smithi084.bmqrnf (mgr.14162) 134 : cluster [DBG] pgmap v127: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:14:25.482 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:25.479+0000 7f5cad59a700 1 -- 172.21.15.132:0/3864059041 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5ca0013990 con 0x7f5cb00f5410 2024-02-20T00:14:25.482 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:25.479+0000 7f5cb49f3700 1 -- 172.21.15.132:0/3864059041 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5cb00f5410 msgr2=0x7f5cb00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:14:25.483 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:25.479+0000 7f5cb49f3700 1 --2- 172.21.15.132:0/3864059041 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5cb00f5410 0x7f5cb00f5830 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 rx=0x7f5ca0009fb0 tx=0x7f5ca0009b30).stop 2024-02-20T00:14:25.483 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:25.479+0000 7f5cb49f3700 1 -- 172.21.15.132:0/3864059041 shutdown_connections 2024-02-20T00:14:25.483 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:25.479+0000 7f5cb49f3700 1 --2- 172.21.15.132:0/3864059041 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5cb00f5410 0x7f5cb00f5830 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:14:25.483 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:25.479+0000 7f5cb49f3700 1 -- 172.21.15.132:0/3864059041 >> 172.21.15.132:0/3864059041 conn(0x7f5cb00f0b70 msgr2=0x7f5cb00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:14:25.483 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:25.479+0000 7f5cb49f3700 1 -- 172.21.15.132:0/3864059041 shutdown_connections 2024-02-20T00:14:25.483 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:25.479+0000 7f5cb49f3700 1 -- 172.21.15.132:0/3864059041 wait complete. 2024-02-20T00:14:25.483 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:25.480+0000 7f5cb49f3700 1 Processor -- start 2024-02-20T00:14:25.483 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:25.480+0000 7f5cb49f3700 1 -- start start 2024-02-20T00:14:25.483 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:25.480+0000 7f5cb49f3700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5cb00f5410 0x7f5cb01056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:14:25.483 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:25.480+0000 7f5cb49f3700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cb0105bd0 con 0x7f5cb00f5410 2024-02-20T00:14:25.485 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:25.481+0000 7f5cae59c700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5cb00f5410 0x7f5cb01056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:14:25.485 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:25.481+0000 7f5cae59c700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5cb00f5410 0x7f5cb01056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:45962/0 (socket says 172.21.15.132:45962) 2024-02-20T00:14:25.485 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:25.481+0000 7f5cae59c700 1 -- 172.21.15.132:0/3458723272 learned_addr learned my addr 172.21.15.132:0/3458723272 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:14:25.485 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:25.481+0000 7f5cae59c700 1 -- 172.21.15.132:0/3458723272 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ca0005040 con 0x7f5cb00f5410 2024-02-20T00:14:25.485 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:25.481+0000 7f5cae59c700 1 --2- 172.21.15.132:0/3458723272 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5cb00f5410 0x7f5cb01056e0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7f5ca0005010 tx=0x7f5ca000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:14:25.485 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:25.481+0000 7f5ca6ffd700 1 -- 172.21.15.132:0/3458723272 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5ca0009320 con 0x7f5cb00f5410 2024-02-20T00:14:25.486 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:25.481+0000 7f5cb49f3700 1 -- 172.21.15.132:0/3458723272 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5cb0105dd0 con 0x7f5cb00f5410 2024-02-20T00:14:25.486 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:25.481+0000 7f5ca6ffd700 1 -- 172.21.15.132:0/3458723272 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5ca0013380 con 0x7f5cb00f5410 2024-02-20T00:14:25.486 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:25.481+0000 7f5ca6ffd700 1 -- 172.21.15.132:0/3458723272 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5ca0014e50 con 0x7f5cb00f5410 2024-02-20T00:14:25.486 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:25.481+0000 7f5cb49f3700 1 -- 172.21.15.132:0/3458723272 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5cb0106260 con 0x7f5cb00f5410 2024-02-20T00:14:25.486 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:25.482+0000 7f5ca6ffd700 1 -- 172.21.15.132:0/3458723272 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5ca0014420 con 0x7f5cb00f5410 2024-02-20T00:14:25.486 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:25.482+0000 7f5cb49f3700 1 -- 172.21.15.132:0/3458723272 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5cb0045ad0 con 0x7f5cb00f5410 2024-02-20T00:14:25.486 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:25.482+0000 7f5ca6ffd700 1 --2- 172.21.15.132:0/3458723272 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f5c9c02ff10 0x7f5c9c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:14:25.486 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:25.482+0000 7f5ca6ffd700 1 -- 172.21.15.132:0/3458723272 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5ca0042d30 con 0x7f5cb00f5410 2024-02-20T00:14:25.486 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:25.482+0000 7f5cadd9b700 1 --2- 172.21.15.132:0/3458723272 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f5c9c02ff10 0x7f5c9c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:14:25.486 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:25.483+0000 7f5cadd9b700 1 --2- 172.21.15.132:0/3458723272 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f5c9c02ff10 0x7f5c9c0323d0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f5c980096a0 tx=0x7f5c98006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:14:25.489 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:25.487+0000 7f5ca6ffd700 1 -- 172.21.15.132:0/3458723272 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5ca00095f0 con 0x7f5cb00f5410 2024-02-20T00:14:25.693 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:25.691+0000 7f5cb49f3700 1 -- 172.21.15.132:0/3458723272 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5cb0107110 con 0x7f5cb00f5410 2024-02-20T00:14:25.694 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:25.691+0000 7f5ca6ffd700 1 -- 172.21.15.132:0/3458723272 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5ca001fb70 con 0x7f5cb00f5410 2024-02-20T00:14:25.694 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:14:25.694 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:14:25.695 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:25.693+0000 7f5cb49f3700 1 -- 172.21.15.132:0/3458723272 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f5c9c02ff10 msgr2=0x7f5c9c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:14:25.696 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:25.693+0000 7f5cb49f3700 1 --2- 172.21.15.132:0/3458723272 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f5c9c02ff10 0x7f5c9c0323d0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f5c980096a0 tx=0x7f5c98006b40).stop 2024-02-20T00:14:25.696 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:25.693+0000 7f5cb49f3700 1 -- 172.21.15.132:0/3458723272 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5cb00f5410 msgr2=0x7f5cb01056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:14:25.696 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:25.693+0000 7f5cb49f3700 1 --2- 172.21.15.132:0/3458723272 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5cb00f5410 0x7f5cb01056e0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7f5ca0005010 tx=0x7f5ca000b040).stop 2024-02-20T00:14:25.696 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:25.693+0000 7f5cb49f3700 1 -- 172.21.15.132:0/3458723272 shutdown_connections 2024-02-20T00:14:25.696 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:25.693+0000 7f5cb49f3700 1 --2- 172.21.15.132:0/3458723272 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f5c9c02ff10 0x7f5c9c0323d0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:14:25.696 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:25.693+0000 7f5cb49f3700 1 --2- 172.21.15.132:0/3458723272 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5cb00f5410 0x7f5cb01056e0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:14:25.696 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:25.693+0000 7f5cb49f3700 1 -- 172.21.15.132:0/3458723272 >> 172.21.15.132:0/3458723272 conn(0x7f5cb00f0b70 msgr2=0x7f5cb00f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:14:25.696 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:25.693+0000 7f5cb49f3700 1 -- 172.21.15.132:0/3458723272 shutdown_connections 2024-02-20T00:14:25.696 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:25.693+0000 7f5cb49f3700 1 -- 172.21.15.132:0/3458723272 wait complete. 2024-02-20T00:14:25.697 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:14:26.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:14:25 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:14:24.877689+0000 mgr.smithi084.bmqrnf (mgr.14162) 135 : cluster [DBG] pgmap v128: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:14:26.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:14:25 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:14:25.693203+0000 mon.smithi084 (mon.0) 188 : audit [DBG] from='client.? 172.21.15.132:0/3458723272' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:14:27.387 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:14:27.388 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:14:28.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:14:27 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:14:26.877957+0000 mgr.smithi084.bmqrnf (mgr.14162) 136 : cluster [DBG] pgmap v129: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:14:29.304 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:29.300+0000 7f382c1d5700 1 -- 172.21.15.132:0/3643100462 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f38240f5410 msgr2=0x7f38240f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:14:29.304 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:29.300+0000 7f382c1d5700 1 --2- 172.21.15.132:0/3643100462 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f38240f5410 0x7f38240f5830 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 rx=0x7f3814009fb0 tx=0x7f3814009b30).stop 2024-02-20T00:14:29.304 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:29.300+0000 7f382c1d5700 1 -- 172.21.15.132:0/3643100462 shutdown_connections 2024-02-20T00:14:29.304 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:29.300+0000 7f382c1d5700 1 --2- 172.21.15.132:0/3643100462 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f38240f5410 0x7f38240f5830 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:14:29.304 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:29.300+0000 7f382c1d5700 1 -- 172.21.15.132:0/3643100462 >> 172.21.15.132:0/3643100462 conn(0x7f38240f0b70 msgr2=0x7f38240f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:14:29.305 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:29.300+0000 7f382c1d5700 1 -- 172.21.15.132:0/3643100462 shutdown_connections 2024-02-20T00:14:29.305 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:29.301+0000 7f382c1d5700 1 -- 172.21.15.132:0/3643100462 wait complete. 2024-02-20T00:14:29.305 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:29.301+0000 7f382c1d5700 1 Processor -- start 2024-02-20T00:14:29.305 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:29.301+0000 7f382c1d5700 1 -- start start 2024-02-20T00:14:29.305 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:29.301+0000 7f382c1d5700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f38240f5410 0x7f3824109500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:14:29.305 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:29.301+0000 7f382c1d5700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3824060220 con 0x7f38240f5410 2024-02-20T00:14:29.305 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:29.301+0000 7f3829f71700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f38240f5410 0x7f3824109500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:14:29.305 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:29.301+0000 7f3829f71700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f38240f5410 0x7f3824109500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:41926/0 (socket says 172.21.15.132:41926) 2024-02-20T00:14:29.305 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:29.301+0000 7f3829f71700 1 -- 172.21.15.132:0/2911908627 learned_addr learned my addr 172.21.15.132:0/2911908627 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:14:29.305 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:29.302+0000 7f3829f71700 1 -- 172.21.15.132:0/2911908627 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3814005040 con 0x7f38240f5410 2024-02-20T00:14:29.305 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:29.302+0000 7f3829f71700 1 --2- 172.21.15.132:0/2911908627 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f38240f5410 0x7f3824109500 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7f3814005010 tx=0x7f381400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:14:29.307 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:29.302+0000 7f381a7fc700 1 -- 172.21.15.132:0/2911908627 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3814009320 con 0x7f38240f5410 2024-02-20T00:14:29.307 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:29.302+0000 7f382c1d5700 1 -- 172.21.15.132:0/2911908627 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3824060420 con 0x7f38240f5410 2024-02-20T00:14:29.307 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:29.302+0000 7f382c1d5700 1 -- 172.21.15.132:0/2911908627 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3824109bb0 con 0x7f38240f5410 2024-02-20T00:14:29.307 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:29.302+0000 7f381a7fc700 1 -- 172.21.15.132:0/2911908627 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3814013380 con 0x7f38240f5410 2024-02-20T00:14:29.307 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:29.302+0000 7f381a7fc700 1 -- 172.21.15.132:0/2911908627 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3814014df0 con 0x7f38240f5410 2024-02-20T00:14:29.307 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:29.303+0000 7f381a7fc700 1 -- 172.21.15.132:0/2911908627 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f381401e420 con 0x7f38240f5410 2024-02-20T00:14:29.307 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:29.303+0000 7f381a7fc700 1 --2- 172.21.15.132:0/2911908627 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f381002ff60 0x7f3810032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:14:29.307 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:29.303+0000 7f382c1d5700 1 -- 172.21.15.132:0/2911908627 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3824059100 con 0x7f38240f5410 2024-02-20T00:14:29.307 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:29.303+0000 7f381a7fc700 1 -- 172.21.15.132:0/2911908627 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3814040f40 con 0x7f38240f5410 2024-02-20T00:14:29.307 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:29.303+0000 7f3829770700 1 --2- 172.21.15.132:0/2911908627 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f381002ff60 0x7f3810032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:14:29.307 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:29.304+0000 7f3829770700 1 --2- 172.21.15.132:0/2911908627 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f381002ff60 0x7f3810032420 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7f38200096a0 tx=0x7f3820006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:14:29.310 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:29.308+0000 7f381a7fc700 1 -- 172.21.15.132:0/2911908627 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f38140263a0 con 0x7f38240f5410 2024-02-20T00:14:29.517 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:29.515+0000 7f382c1d5700 1 -- 172.21.15.132:0/2911908627 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f38240fafa0 con 0x7f38240f5410 2024-02-20T00:14:29.518 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:29.515+0000 7f381a7fc700 1 -- 172.21.15.132:0/2911908627 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f381402b020 con 0x7f38240f5410 2024-02-20T00:14:29.518 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:14:29.518 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:14:29.519 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:29.517+0000 7f382c1d5700 1 -- 172.21.15.132:0/2911908627 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f381002ff60 msgr2=0x7f3810032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:14:29.520 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:29.517+0000 7f382c1d5700 1 --2- 172.21.15.132:0/2911908627 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f381002ff60 0x7f3810032420 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7f38200096a0 tx=0x7f3820006b40).stop 2024-02-20T00:14:29.520 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:29.517+0000 7f382c1d5700 1 -- 172.21.15.132:0/2911908627 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f38240f5410 msgr2=0x7f3824109500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:14:29.520 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:29.517+0000 7f382c1d5700 1 --2- 172.21.15.132:0/2911908627 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f38240f5410 0x7f3824109500 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7f3814005010 tx=0x7f381400b040).stop 2024-02-20T00:14:29.520 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:29.517+0000 7f382c1d5700 1 -- 172.21.15.132:0/2911908627 shutdown_connections 2024-02-20T00:14:29.520 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:29.517+0000 7f382c1d5700 1 --2- 172.21.15.132:0/2911908627 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f381002ff60 0x7f3810032420 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:14:29.520 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:29.517+0000 7f382c1d5700 1 --2- 172.21.15.132:0/2911908627 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f38240f5410 0x7f3824109500 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:14:29.520 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:29.517+0000 7f382c1d5700 1 -- 172.21.15.132:0/2911908627 >> 172.21.15.132:0/2911908627 conn(0x7f38240f0b70 msgr2=0x7f38240f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:14:29.520 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:29.517+0000 7f382c1d5700 1 -- 172.21.15.132:0/2911908627 shutdown_connections 2024-02-20T00:14:29.520 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:29.517+0000 7f382c1d5700 1 -- 172.21.15.132:0/2911908627 wait complete. 2024-02-20T00:14:29.520 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:14:30.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:14:29 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:14:28.878305+0000 mgr.smithi084.bmqrnf (mgr.14162) 137 : cluster [DBG] pgmap v130: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:14:30.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:14:29 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:14:29.517164+0000 mon.smithi084 (mon.0) 189 : audit [DBG] from='client.? 172.21.15.132:0/2911908627' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:14:31.219 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:14:31.220 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:14:32.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:14:31 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:14:30.878636+0000 mgr.smithi084.bmqrnf (mgr.14162) 138 : cluster [DBG] pgmap v131: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:14:33.089 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:33.085+0000 7f1015350700 1 -- 172.21.15.132:0/1304880441 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f10100f6510 msgr2=0x7f10100f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:14:33.089 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:33.085+0000 7f1015350700 1 --2- 172.21.15.132:0/1304880441 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f10100f6510 0x7f10100f6930 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 rx=0x7f1004009fb0 tx=0x7f1004009b30).stop 2024-02-20T00:14:33.089 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:33.085+0000 7f1015350700 1 -- 172.21.15.132:0/1304880441 shutdown_connections 2024-02-20T00:14:33.089 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:33.085+0000 7f1015350700 1 --2- 172.21.15.132:0/1304880441 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f10100f6510 0x7f10100f6930 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:14:33.090 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:33.085+0000 7f1015350700 1 -- 172.21.15.132:0/1304880441 >> 172.21.15.132:0/1304880441 conn(0x7f10100f1c70 msgr2=0x7f10100f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:14:33.090 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:33.085+0000 7f1015350700 1 -- 172.21.15.132:0/1304880441 shutdown_connections 2024-02-20T00:14:33.090 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:33.085+0000 7f1015350700 1 -- 172.21.15.132:0/1304880441 wait complete. 2024-02-20T00:14:33.090 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:33.086+0000 7f1015350700 1 Processor -- start 2024-02-20T00:14:33.090 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:33.086+0000 7f1015350700 1 -- start start 2024-02-20T00:14:33.090 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:33.086+0000 7f1015350700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f10100f6510 0x7f10101067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:14:33.090 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:33.086+0000 7f1015350700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1010106cd0 con 0x7f10100f6510 2024-02-20T00:14:33.090 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:33.086+0000 7f100effd700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f10100f6510 0x7f10101067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:14:33.090 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:33.086+0000 7f100effd700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f10100f6510 0x7f10101067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:41936/0 (socket says 172.21.15.132:41936) 2024-02-20T00:14:33.090 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:33.086+0000 7f100effd700 1 -- 172.21.15.132:0/528284736 learned_addr learned my addr 172.21.15.132:0/528284736 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:14:33.090 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:33.087+0000 7f100effd700 1 -- 172.21.15.132:0/528284736 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1004005040 con 0x7f10100f6510 2024-02-20T00:14:33.091 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:33.087+0000 7f100effd700 1 --2- 172.21.15.132:0/528284736 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f10100f6510 0x7f10101067e0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7f1004005010 tx=0x7f100400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:14:33.091 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:33.087+0000 7f0ff77fe700 1 -- 172.21.15.132:0/528284736 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1004009320 con 0x7f10100f6510 2024-02-20T00:14:33.091 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:33.087+0000 7f1015350700 1 -- 172.21.15.132:0/528284736 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1010106ed0 con 0x7f10100f6510 2024-02-20T00:14:33.091 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:33.087+0000 7f0ff77fe700 1 -- 172.21.15.132:0/528284736 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1004013380 con 0x7f10100f6510 2024-02-20T00:14:33.091 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:33.087+0000 7f0ff77fe700 1 -- 172.21.15.132:0/528284736 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1004014e50 con 0x7f10100f6510 2024-02-20T00:14:33.091 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:33.087+0000 7f1015350700 1 -- 172.21.15.132:0/528284736 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1010107360 con 0x7f10100f6510 2024-02-20T00:14:33.092 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:33.088+0000 7f0ff77fe700 1 -- 172.21.15.132:0/528284736 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f10040095f0 con 0x7f10100f6510 2024-02-20T00:14:33.092 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:33.088+0000 7f1015350700 1 -- 172.21.15.132:0/528284736 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1010045ad0 con 0x7f10100f6510 2024-02-20T00:14:33.092 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:33.088+0000 7f0ff77fe700 1 --2- 172.21.15.132:0/528284736 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f0ff802ff10 0x7f0ff80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:14:33.095 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:33.088+0000 7f0ff77fe700 1 -- 172.21.15.132:0/528284736 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1004040e90 con 0x7f10100f6510 2024-02-20T00:14:33.096 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:33.088+0000 7f100e7fc700 1 --2- 172.21.15.132:0/528284736 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f0ff802ff10 0x7f0ff80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:14:33.096 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:33.089+0000 7f100e7fc700 1 --2- 172.21.15.132:0/528284736 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f0ff802ff10 0x7f0ff80323d0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f10000096a0 tx=0x7f1000006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:14:33.097 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:33.093+0000 7f0ff77fe700 1 -- 172.21.15.132:0/528284736 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1004014420 con 0x7f10100f6510 2024-02-20T00:14:33.300 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:33.295+0000 7f1015350700 1 -- 172.21.15.132:0/528284736 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1010108210 con 0x7f10100f6510 2024-02-20T00:14:33.300 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:33.296+0000 7f0ff77fe700 1 -- 172.21.15.132:0/528284736 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1004010230 con 0x7f10100f6510 2024-02-20T00:14:33.300 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:14:33.300 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:14:33.301 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:33.297+0000 7f1015350700 1 -- 172.21.15.132:0/528284736 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f0ff802ff10 msgr2=0x7f0ff80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:14:33.301 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:33.297+0000 7f1015350700 1 --2- 172.21.15.132:0/528284736 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f0ff802ff10 0x7f0ff80323d0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f10000096a0 tx=0x7f1000006b40).stop 2024-02-20T00:14:33.301 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:33.298+0000 7f1015350700 1 -- 172.21.15.132:0/528284736 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f10100f6510 msgr2=0x7f10101067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:14:33.301 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:33.298+0000 7f1015350700 1 --2- 172.21.15.132:0/528284736 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f10100f6510 0x7f10101067e0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7f1004005010 tx=0x7f100400b040).stop 2024-02-20T00:14:33.301 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:33.298+0000 7f1015350700 1 -- 172.21.15.132:0/528284736 shutdown_connections 2024-02-20T00:14:33.301 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:33.298+0000 7f1015350700 1 --2- 172.21.15.132:0/528284736 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f0ff802ff10 0x7f0ff80323d0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:14:33.302 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:33.298+0000 7f1015350700 1 --2- 172.21.15.132:0/528284736 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f10100f6510 0x7f10101067e0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:14:33.302 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:33.298+0000 7f1015350700 1 -- 172.21.15.132:0/528284736 >> 172.21.15.132:0/528284736 conn(0x7f10100f1c70 msgr2=0x7f10100f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:14:33.302 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:33.298+0000 7f1015350700 1 -- 172.21.15.132:0/528284736 shutdown_connections 2024-02-20T00:14:33.302 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:33.298+0000 7f1015350700 1 -- 172.21.15.132:0/528284736 wait complete. 2024-02-20T00:14:33.302 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:14:34.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:14:33 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:14:32.878978+0000 mgr.smithi084.bmqrnf (mgr.14162) 139 : cluster [DBG] pgmap v132: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:14:34.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:14:33 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:14:33.297426+0000 mon.smithi084 (mon.0) 190 : audit [DBG] from='client.? 172.21.15.132:0/528284736' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:14:35.022 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:14:35.022 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:14:36.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:14:35 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:14:34.879359+0000 mgr.smithi084.bmqrnf (mgr.14162) 140 : cluster [DBG] pgmap v133: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:14:36.811 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:36.793+0000 7fe0839f5700 1 -- 172.21.15.132:0/4134011302 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe07c0f5410 msgr2=0x7fe07c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:14:36.811 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:36.793+0000 7fe0839f5700 1 --2- 172.21.15.132:0/4134011302 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe07c0f5410 0x7fe07c0f5830 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 rx=0x7fe06c009fb0 tx=0x7fe06c009b30).stop 2024-02-20T00:14:36.811 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:36.793+0000 7fe0839f5700 1 -- 172.21.15.132:0/4134011302 shutdown_connections 2024-02-20T00:14:36.812 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:36.793+0000 7fe0839f5700 1 --2- 172.21.15.132:0/4134011302 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe07c0f5410 0x7fe07c0f5830 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:14:36.812 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:36.793+0000 7fe0839f5700 1 -- 172.21.15.132:0/4134011302 >> 172.21.15.132:0/4134011302 conn(0x7fe07c0f0b70 msgr2=0x7fe07c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:14:36.812 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:36.793+0000 7fe0839f5700 1 -- 172.21.15.132:0/4134011302 shutdown_connections 2024-02-20T00:14:36.812 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:36.793+0000 7fe0839f5700 1 -- 172.21.15.132:0/4134011302 wait complete. 2024-02-20T00:14:36.812 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:36.794+0000 7fe0839f5700 1 Processor -- start 2024-02-20T00:14:36.812 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:36.794+0000 7fe0839f5700 1 -- start start 2024-02-20T00:14:36.812 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:36.794+0000 7fe0839f5700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe07c0f5410 0x7fe07c1094e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:14:36.812 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:36.794+0000 7fe0839f5700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe07c060240 con 0x7fe07c0f5410 2024-02-20T00:14:36.812 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:36.794+0000 7fe081791700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe07c0f5410 0x7fe07c1094e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:14:36.812 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:36.794+0000 7fe081791700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe07c0f5410 0x7fe07c1094e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:41946/0 (socket says 172.21.15.132:41946) 2024-02-20T00:14:36.812 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:36.794+0000 7fe081791700 1 -- 172.21.15.132:0/3829387526 learned_addr learned my addr 172.21.15.132:0/3829387526 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:14:36.812 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:36.794+0000 7fe081791700 1 -- 172.21.15.132:0/3829387526 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe06c005040 con 0x7fe07c0f5410 2024-02-20T00:14:36.813 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:36.794+0000 7fe081791700 1 --2- 172.21.15.132:0/3829387526 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe07c0f5410 0x7fe07c1094e0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7fe06c005010 tx=0x7fe06c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:14:36.813 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:36.795+0000 7fe071ffb700 1 -- 172.21.15.132:0/3829387526 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe06c009320 con 0x7fe07c0f5410 2024-02-20T00:14:36.813 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:36.795+0000 7fe0839f5700 1 -- 172.21.15.132:0/3829387526 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe07c060440 con 0x7fe07c0f5410 2024-02-20T00:14:36.813 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:36.795+0000 7fe071ffb700 1 -- 172.21.15.132:0/3829387526 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe06c013380 con 0x7fe07c0f5410 2024-02-20T00:14:36.813 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:36.795+0000 7fe071ffb700 1 -- 172.21.15.132:0/3829387526 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe06c014e50 con 0x7fe07c0f5410 2024-02-20T00:14:36.813 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:36.795+0000 7fe0839f5700 1 -- 172.21.15.132:0/3829387526 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe07c109b90 con 0x7fe07c0f5410 2024-02-20T00:14:36.813 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:36.796+0000 7fe071ffb700 1 -- 172.21.15.132:0/3829387526 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe06c014420 con 0x7fe07c0f5410 2024-02-20T00:14:36.813 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:36.796+0000 7fe071ffb700 1 --2- 172.21.15.132:0/3829387526 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fe06802ff10 0x7fe0680323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:14:36.813 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:36.796+0000 7fe0839f5700 1 -- 172.21.15.132:0/3829387526 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe07c045ad0 con 0x7fe07c0f5410 2024-02-20T00:14:36.813 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:36.796+0000 7fe071ffb700 1 -- 172.21.15.132:0/3829387526 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe06c040f70 con 0x7fe07c0f5410 2024-02-20T00:14:36.813 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:36.796+0000 7fe080f90700 1 --2- 172.21.15.132:0/3829387526 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fe06802ff10 0x7fe0680323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:14:36.813 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:36.796+0000 7fe080f90700 1 --2- 172.21.15.132:0/3829387526 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fe06802ff10 0x7fe0680323d0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7fe0780096a0 tx=0x7fe078006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:14:36.813 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:36.800+0000 7fe071ffb700 1 -- 172.21.15.132:0/3829387526 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe06c0095f0 con 0x7fe07c0f5410 2024-02-20T00:14:37.006 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:37.004+0000 7fe0839f5700 1 -- 172.21.15.132:0/3829387526 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe07c10a3b0 con 0x7fe07c0f5410 2024-02-20T00:14:37.007 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:37.005+0000 7fe071ffb700 1 -- 172.21.15.132:0/3829387526 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe06c01db70 con 0x7fe07c0f5410 2024-02-20T00:14:37.007 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:14:37.007 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:14:37.008 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:37.006+0000 7fe0839f5700 1 -- 172.21.15.132:0/3829387526 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fe06802ff10 msgr2=0x7fe0680323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:14:37.008 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:37.006+0000 7fe0839f5700 1 --2- 172.21.15.132:0/3829387526 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fe06802ff10 0x7fe0680323d0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7fe0780096a0 tx=0x7fe078006b40).stop 2024-02-20T00:14:37.009 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:37.006+0000 7fe0839f5700 1 -- 172.21.15.132:0/3829387526 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe07c0f5410 msgr2=0x7fe07c1094e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:14:37.009 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:37.006+0000 7fe0839f5700 1 --2- 172.21.15.132:0/3829387526 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe07c0f5410 0x7fe07c1094e0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7fe06c005010 tx=0x7fe06c00b040).stop 2024-02-20T00:14:37.009 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:37.006+0000 7fe0839f5700 1 -- 172.21.15.132:0/3829387526 shutdown_connections 2024-02-20T00:14:37.009 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:37.006+0000 7fe0839f5700 1 --2- 172.21.15.132:0/3829387526 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fe06802ff10 0x7fe0680323d0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:14:37.009 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:37.006+0000 7fe0839f5700 1 --2- 172.21.15.132:0/3829387526 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe07c0f5410 0x7fe07c1094e0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:14:37.009 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:37.006+0000 7fe0839f5700 1 -- 172.21.15.132:0/3829387526 >> 172.21.15.132:0/3829387526 conn(0x7fe07c0f0b70 msgr2=0x7fe07c0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:14:37.009 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:37.006+0000 7fe0839f5700 1 -- 172.21.15.132:0/3829387526 shutdown_connections 2024-02-20T00:14:37.009 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:37.006+0000 7fe0839f5700 1 -- 172.21.15.132:0/3829387526 wait complete. 2024-02-20T00:14:37.009 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:14:38.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:14:37 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:14:36.879748+0000 mgr.smithi084.bmqrnf (mgr.14162) 141 : cluster [DBG] pgmap v134: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:14:38.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:14:37 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:14:37.006105+0000 mon.smithi084 (mon.0) 191 : audit [DBG] from='client.? 172.21.15.132:0/3829387526' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:14:38.618 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:14:38.619 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:14:40.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:14:39 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:14:38.880087+0000 mgr.smithi084.bmqrnf (mgr.14162) 142 : cluster [DBG] pgmap v135: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:14:40.466 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:40.462+0000 7fea43144700 1 -- 172.21.15.132:0/1246733218 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fea3c0f6510 msgr2=0x7fea3c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:14:40.466 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:40.462+0000 7fea43144700 1 --2- 172.21.15.132:0/1246733218 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fea3c0f6510 0x7fea3c0f6930 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 rx=0x7fea30009fb0 tx=0x7fea30009b30).stop 2024-02-20T00:14:40.466 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:40.463+0000 7fea43144700 1 -- 172.21.15.132:0/1246733218 shutdown_connections 2024-02-20T00:14:40.466 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:40.463+0000 7fea43144700 1 --2- 172.21.15.132:0/1246733218 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fea3c0f6510 0x7fea3c0f6930 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:14:40.466 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:40.463+0000 7fea43144700 1 -- 172.21.15.132:0/1246733218 >> 172.21.15.132:0/1246733218 conn(0x7fea3c0f1c70 msgr2=0x7fea3c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:14:40.466 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:40.463+0000 7fea43144700 1 -- 172.21.15.132:0/1246733218 shutdown_connections 2024-02-20T00:14:40.466 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:40.463+0000 7fea43144700 1 -- 172.21.15.132:0/1246733218 wait complete. 2024-02-20T00:14:40.466 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:40.463+0000 7fea43144700 1 Processor -- start 2024-02-20T00:14:40.466 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:40.463+0000 7fea43144700 1 -- start start 2024-02-20T00:14:40.467 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:40.463+0000 7fea43144700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fea3c0f6510 0x7fea3c06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:14:40.467 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:40.463+0000 7fea43144700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea3c069e20 con 0x7fea3c0f6510 2024-02-20T00:14:40.467 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:40.464+0000 7fea40ee0700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fea3c0f6510 0x7fea3c06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:14:40.467 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:40.464+0000 7fea40ee0700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fea3c0f6510 0x7fea3c06b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:49838/0 (socket says 172.21.15.132:49838) 2024-02-20T00:14:40.467 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:40.464+0000 7fea40ee0700 1 -- 172.21.15.132:0/3460293309 learned_addr learned my addr 172.21.15.132:0/3460293309 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:14:40.467 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:40.464+0000 7fea40ee0700 1 -- 172.21.15.132:0/3460293309 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea30005040 con 0x7fea3c0f6510 2024-02-20T00:14:40.467 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:40.464+0000 7fea40ee0700 1 --2- 172.21.15.132:0/3460293309 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fea3c0f6510 0x7fea3c06b9a0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7fea30005010 tx=0x7fea3000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:14:40.468 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:40.464+0000 7fea397fa700 1 -- 172.21.15.132:0/3460293309 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fea30009320 con 0x7fea3c0f6510 2024-02-20T00:14:40.468 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:40.464+0000 7fea43144700 1 -- 172.21.15.132:0/3460293309 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea3c069fc0 con 0x7fea3c0f6510 2024-02-20T00:14:40.468 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:40.464+0000 7fea397fa700 1 -- 172.21.15.132:0/3460293309 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fea30013380 con 0x7fea3c0f6510 2024-02-20T00:14:40.468 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:40.464+0000 7fea397fa700 1 -- 172.21.15.132:0/3460293309 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fea30014e50 con 0x7fea3c0f6510 2024-02-20T00:14:40.468 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:40.464+0000 7fea43144700 1 -- 172.21.15.132:0/3460293309 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea3c06a450 con 0x7fea3c0f6510 2024-02-20T00:14:40.468 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:40.465+0000 7fea397fa700 1 -- 172.21.15.132:0/3460293309 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fea30014420 con 0x7fea3c0f6510 2024-02-20T00:14:40.468 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:40.465+0000 7fea43144700 1 -- 172.21.15.132:0/3460293309 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea3c059100 con 0x7fea3c0f6510 2024-02-20T00:14:40.468 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:40.465+0000 7fea397fa700 1 --2- 172.21.15.132:0/3460293309 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fea2c02ff10 0x7fea2c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:14:40.469 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:40.465+0000 7fea397fa700 1 -- 172.21.15.132:0/3460293309 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fea30040e60 con 0x7fea3c0f6510 2024-02-20T00:14:40.469 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:40.465+0000 7fea3bfff700 1 --2- 172.21.15.132:0/3460293309 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fea2c02ff10 0x7fea2c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:14:40.469 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:40.466+0000 7fea3bfff700 1 --2- 172.21.15.132:0/3460293309 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fea2c02ff10 0x7fea2c0323d0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7fea280096a0 tx=0x7fea28006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:14:40.472 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:40.471+0000 7fea397fa700 1 -- 172.21.15.132:0/3460293309 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fea3001db70 con 0x7fea3c0f6510 2024-02-20T00:14:40.677 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:40.674+0000 7fea43144700 1 -- 172.21.15.132:0/3460293309 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fea3c0fc0a0 con 0x7fea3c0f6510 2024-02-20T00:14:40.677 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:40.675+0000 7fea397fa700 1 -- 172.21.15.132:0/3460293309 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fea3001db70 con 0x7fea3c0f6510 2024-02-20T00:14:40.678 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:14:40.678 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:14:40.679 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:40.677+0000 7fea43144700 1 -- 172.21.15.132:0/3460293309 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fea2c02ff10 msgr2=0x7fea2c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:14:40.679 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:40.677+0000 7fea43144700 1 --2- 172.21.15.132:0/3460293309 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fea2c02ff10 0x7fea2c0323d0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7fea280096a0 tx=0x7fea28006b40).stop 2024-02-20T00:14:40.679 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:40.677+0000 7fea43144700 1 -- 172.21.15.132:0/3460293309 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fea3c0f6510 msgr2=0x7fea3c06b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:14:40.679 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:40.677+0000 7fea43144700 1 --2- 172.21.15.132:0/3460293309 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fea3c0f6510 0x7fea3c06b9a0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7fea30005010 tx=0x7fea3000b040).stop 2024-02-20T00:14:40.679 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:40.677+0000 7fea43144700 1 -- 172.21.15.132:0/3460293309 shutdown_connections 2024-02-20T00:14:40.680 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:40.677+0000 7fea43144700 1 --2- 172.21.15.132:0/3460293309 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fea2c02ff10 0x7fea2c0323d0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:14:40.680 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:40.677+0000 7fea43144700 1 --2- 172.21.15.132:0/3460293309 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fea3c0f6510 0x7fea3c06b9a0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:14:40.680 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:40.677+0000 7fea43144700 1 -- 172.21.15.132:0/3460293309 >> 172.21.15.132:0/3460293309 conn(0x7fea3c0f1c70 msgr2=0x7fea3c0f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:14:40.680 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:40.677+0000 7fea43144700 1 -- 172.21.15.132:0/3460293309 shutdown_connections 2024-02-20T00:14:40.683 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:40.677+0000 7fea43144700 1 -- 172.21.15.132:0/3460293309 wait complete. 2024-02-20T00:14:40.683 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:14:41.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:14:40 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:14:40.676715+0000 mon.smithi084 (mon.0) 192 : audit [DBG] from='client.? 172.21.15.132:0/3460293309' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:14:42.355 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:14:42.355 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:14:42.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:14:41 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:14:40.880413+0000 mgr.smithi084.bmqrnf (mgr.14162) 143 : cluster [DBG] pgmap v136: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:14:44.336 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:44.231+0000 7f0887245700 1 -- 172.21.15.132:0/1636039863 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f08800f5410 msgr2=0x7f08800f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:14:44.336 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:44.231+0000 7f0887245700 1 --2- 172.21.15.132:0/1636039863 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f08800f5410 0x7f08800f5830 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 rx=0x7f0874009fb0 tx=0x7f0874009b30).stop 2024-02-20T00:14:44.336 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:44.231+0000 7f0887245700 1 -- 172.21.15.132:0/1636039863 shutdown_connections 2024-02-20T00:14:44.336 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:44.231+0000 7f0887245700 1 --2- 172.21.15.132:0/1636039863 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f08800f5410 0x7f08800f5830 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:14:44.336 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:44.231+0000 7f0887245700 1 -- 172.21.15.132:0/1636039863 >> 172.21.15.132:0/1636039863 conn(0x7f08800f0b70 msgr2=0x7f08800f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:14:44.337 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:44.231+0000 7f0887245700 1 -- 172.21.15.132:0/1636039863 shutdown_connections 2024-02-20T00:14:44.337 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:44.231+0000 7f0887245700 1 -- 172.21.15.132:0/1636039863 wait complete. 2024-02-20T00:14:44.337 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:44.232+0000 7f0887245700 1 Processor -- start 2024-02-20T00:14:44.337 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:44.232+0000 7f0887245700 1 -- start start 2024-02-20T00:14:44.337 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:44.232+0000 7f0887245700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f08800f5410 0x7f0880109500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:14:44.337 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:44.232+0000 7f0887245700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0880060220 con 0x7f08800f5410 2024-02-20T00:14:44.337 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:44.233+0000 7f0884fe1700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f08800f5410 0x7f0880109500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:14:44.337 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:44.233+0000 7f0884fe1700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f08800f5410 0x7f0880109500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:49854/0 (socket says 172.21.15.132:49854) 2024-02-20T00:14:44.337 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:44.233+0000 7f0884fe1700 1 -- 172.21.15.132:0/536827534 learned_addr learned my addr 172.21.15.132:0/536827534 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:14:44.337 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:44.233+0000 7f0884fe1700 1 -- 172.21.15.132:0/536827534 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0874005040 con 0x7f08800f5410 2024-02-20T00:14:44.337 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:44.233+0000 7f0884fe1700 1 --2- 172.21.15.132:0/536827534 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f08800f5410 0x7f0880109500 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7f0874005010 tx=0x7f087400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:14:44.337 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:44.233+0000 7f087d7fa700 1 -- 172.21.15.132:0/536827534 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0874009320 con 0x7f08800f5410 2024-02-20T00:14:44.337 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:44.233+0000 7f0887245700 1 -- 172.21.15.132:0/536827534 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0880060420 con 0x7f08800f5410 2024-02-20T00:14:44.337 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:44.233+0000 7f087d7fa700 1 -- 172.21.15.132:0/536827534 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0874013380 con 0x7f08800f5410 2024-02-20T00:14:44.338 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:44.233+0000 7f087d7fa700 1 -- 172.21.15.132:0/536827534 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0874014e50 con 0x7f08800f5410 2024-02-20T00:14:44.338 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:44.233+0000 7f0887245700 1 -- 172.21.15.132:0/536827534 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0880109bb0 con 0x7f08800f5410 2024-02-20T00:14:44.338 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:44.234+0000 7f087d7fa700 1 -- 172.21.15.132:0/536827534 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f08740263a0 con 0x7f08800f5410 2024-02-20T00:14:44.338 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:44.234+0000 7f0887245700 1 -- 172.21.15.132:0/536827534 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0880059100 con 0x7f08800f5410 2024-02-20T00:14:44.338 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:44.234+0000 7f087d7fa700 1 --2- 172.21.15.132:0/536827534 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f087002ff10 0x7f08700323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:14:44.338 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:44.234+0000 7f087d7fa700 1 -- 172.21.15.132:0/536827534 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0874041140 con 0x7f08800f5410 2024-02-20T00:14:44.338 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:44.235+0000 7f087ffff700 1 --2- 172.21.15.132:0/536827534 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f087002ff10 0x7f08700323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:14:44.338 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:44.235+0000 7f087ffff700 1 --2- 172.21.15.132:0/536827534 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f087002ff10 0x7f08700323d0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7f086c0096a0 tx=0x7f086c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:14:44.338 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:44.239+0000 7f087d7fa700 1 -- 172.21.15.132:0/536827534 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f087401ccf0 con 0x7f08800f5410 2024-02-20T00:14:44.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:14:43 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:14:42.880708+0000 mgr.smithi084.bmqrnf (mgr.14162) 144 : cluster [DBG] pgmap v137: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:14:44.449 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:44.448+0000 7f0887245700 1 -- 172.21.15.132:0/536827534 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0880045ad0 con 0x7f08800f5410 2024-02-20T00:14:44.449 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:44.448+0000 7f087d7fa700 1 -- 172.21.15.132:0/536827534 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0874012330 con 0x7f08800f5410 2024-02-20T00:14:44.450 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:14:44.450 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:14:44.451 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:44.450+0000 7f0887245700 1 -- 172.21.15.132:0/536827534 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f087002ff10 msgr2=0x7f08700323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:14:44.451 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:44.450+0000 7f0887245700 1 --2- 172.21.15.132:0/536827534 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f087002ff10 0x7f08700323d0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7f086c0096a0 tx=0x7f086c006b40).stop 2024-02-20T00:14:44.452 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:44.450+0000 7f0887245700 1 -- 172.21.15.132:0/536827534 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f08800f5410 msgr2=0x7f0880109500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:14:44.452 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:44.450+0000 7f0887245700 1 --2- 172.21.15.132:0/536827534 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f08800f5410 0x7f0880109500 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7f0874005010 tx=0x7f087400b040).stop 2024-02-20T00:14:44.452 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:44.450+0000 7f0887245700 1 -- 172.21.15.132:0/536827534 shutdown_connections 2024-02-20T00:14:44.452 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:44.450+0000 7f0887245700 1 --2- 172.21.15.132:0/536827534 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f087002ff10 0x7f08700323d0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:14:44.452 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:44.450+0000 7f0887245700 1 --2- 172.21.15.132:0/536827534 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f08800f5410 0x7f0880109500 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:14:44.452 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:44.450+0000 7f0887245700 1 -- 172.21.15.132:0/536827534 >> 172.21.15.132:0/536827534 conn(0x7f08800f0b70 msgr2=0x7f08800f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:14:44.452 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:44.450+0000 7f0887245700 1 -- 172.21.15.132:0/536827534 shutdown_connections 2024-02-20T00:14:44.452 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:44.450+0000 7f0887245700 1 -- 172.21.15.132:0/536827534 wait complete. 2024-02-20T00:14:44.453 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:14:45.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:14:44 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:14:44.448855+0000 mon.smithi084 (mon.0) 193 : audit [DBG] from='client.? 172.21.15.132:0/536827534' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:14:46.040 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:14:46.041 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:14:46.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:14:45 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:14:44.880929+0000 mgr.smithi084.bmqrnf (mgr.14162) 145 : cluster [DBG] pgmap v138: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:14:47.992 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:47.989+0000 7f46c9381700 1 -- 172.21.15.132:0/1085636179 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f46c40f6510 msgr2=0x7f46c40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:14:47.992 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:47.989+0000 7f46c9381700 1 --2- 172.21.15.132:0/1085636179 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f46c40f6510 0x7f46c40f6930 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 rx=0x7f46ac009fb0 tx=0x7f46ac009b30).stop 2024-02-20T00:14:47.993 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:47.989+0000 7f46c9381700 1 -- 172.21.15.132:0/1085636179 shutdown_connections 2024-02-20T00:14:47.993 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:47.989+0000 7f46c9381700 1 --2- 172.21.15.132:0/1085636179 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f46c40f6510 0x7f46c40f6930 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:14:47.993 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:47.989+0000 7f46c9381700 1 -- 172.21.15.132:0/1085636179 >> 172.21.15.132:0/1085636179 conn(0x7f46c40f1c70 msgr2=0x7f46c40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:14:47.993 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:47.989+0000 7f46c9381700 1 -- 172.21.15.132:0/1085636179 shutdown_connections 2024-02-20T00:14:47.993 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:47.989+0000 7f46c9381700 1 -- 172.21.15.132:0/1085636179 wait complete. 2024-02-20T00:14:47.993 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:47.990+0000 7f46c9381700 1 Processor -- start 2024-02-20T00:14:47.993 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:47.990+0000 7f46c9381700 1 -- start start 2024-02-20T00:14:47.993 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:47.990+0000 7f46c9381700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f46c40f6510 0x7f46c41067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:14:47.993 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:47.990+0000 7f46c9381700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46c4106cd0 con 0x7f46c40f6510 2024-02-20T00:14:47.993 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:47.990+0000 7f46c2ffd700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f46c40f6510 0x7f46c41067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:14:47.993 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:47.991+0000 7f46c2ffd700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f46c40f6510 0x7f46c41067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:38062/0 (socket says 172.21.15.132:38062) 2024-02-20T00:14:47.993 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:47.991+0000 7f46c2ffd700 1 -- 172.21.15.132:0/1995065313 learned_addr learned my addr 172.21.15.132:0/1995065313 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:14:47.993 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:47.991+0000 7f46c2ffd700 1 -- 172.21.15.132:0/1995065313 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46ac005040 con 0x7f46c40f6510 2024-02-20T00:14:47.994 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:47.991+0000 7f46c2ffd700 1 --2- 172.21.15.132:0/1995065313 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f46c40f6510 0x7f46c41067e0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7f46ac0089f0 tx=0x7f46ac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:14:47.994 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:47.991+0000 7f46bb7fe700 1 -- 172.21.15.132:0/1995065313 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f46ac009320 con 0x7f46c40f6510 2024-02-20T00:14:47.994 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:47.991+0000 7f46c9381700 1 -- 172.21.15.132:0/1995065313 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46c4106ed0 con 0x7f46c40f6510 2024-02-20T00:14:47.994 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:47.991+0000 7f46bb7fe700 1 -- 172.21.15.132:0/1995065313 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f46ac013380 con 0x7f46c40f6510 2024-02-20T00:14:47.994 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:47.991+0000 7f46bb7fe700 1 -- 172.21.15.132:0/1995065313 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f46ac01ed80 con 0x7f46c40f6510 2024-02-20T00:14:47.994 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:47.991+0000 7f46c9381700 1 -- 172.21.15.132:0/1995065313 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46c4107360 con 0x7f46c40f6510 2024-02-20T00:14:47.995 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:47.992+0000 7f46bb7fe700 1 -- 172.21.15.132:0/1995065313 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f46ac01e420 con 0x7f46c40f6510 2024-02-20T00:14:47.995 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:47.992+0000 7f46bb7fe700 1 --2- 172.21.15.132:0/1995065313 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f46b002ff60 0x7f46b0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:14:47.995 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:47.992+0000 7f46bb7fe700 1 -- 172.21.15.132:0/1995065313 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f46ac040ed0 con 0x7f46c40f6510 2024-02-20T00:14:47.995 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:47.992+0000 7f46c9381700 1 -- 172.21.15.132:0/1995065313 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46c4045ad0 con 0x7f46c40f6510 2024-02-20T00:14:47.995 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:47.992+0000 7f46c27fc700 1 --2- 172.21.15.132:0/1995065313 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f46b002ff60 0x7f46b0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:14:47.995 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:47.993+0000 7f46c27fc700 1 --2- 172.21.15.132:0/1995065313 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f46b002ff60 0x7f46b0032420 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f46b40096a0 tx=0x7f46b4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:14:47.998 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:47.997+0000 7f46bb7fe700 1 -- 172.21.15.132:0/1995065313 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f46ac0095f0 con 0x7f46c40f6510 2024-02-20T00:14:48.201 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:48.200+0000 7f46c9381700 1 -- 172.21.15.132:0/1995065313 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f46c4108210 con 0x7f46c40f6510 2024-02-20T00:14:48.202 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:48.201+0000 7f46bb7fe700 1 -- 172.21.15.132:0/1995065313 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f46ac0095f0 con 0x7f46c40f6510 2024-02-20T00:14:48.202 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:14:48.202 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:14:48.204 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:48.202+0000 7f46c9381700 1 -- 172.21.15.132:0/1995065313 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f46b002ff60 msgr2=0x7f46b0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:14:48.204 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:48.202+0000 7f46c9381700 1 --2- 172.21.15.132:0/1995065313 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f46b002ff60 0x7f46b0032420 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f46b40096a0 tx=0x7f46b4006b40).stop 2024-02-20T00:14:48.204 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:48.202+0000 7f46c9381700 1 -- 172.21.15.132:0/1995065313 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f46c40f6510 msgr2=0x7f46c41067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:14:48.204 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:48.202+0000 7f46c9381700 1 --2- 172.21.15.132:0/1995065313 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f46c40f6510 0x7f46c41067e0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7f46ac0089f0 tx=0x7f46ac00b040).stop 2024-02-20T00:14:48.204 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:48.202+0000 7f46c9381700 1 -- 172.21.15.132:0/1995065313 shutdown_connections 2024-02-20T00:14:48.204 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:48.202+0000 7f46c9381700 1 --2- 172.21.15.132:0/1995065313 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f46b002ff60 0x7f46b0032420 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:14:48.204 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:48.202+0000 7f46c9381700 1 --2- 172.21.15.132:0/1995065313 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f46c40f6510 0x7f46c41067e0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:14:48.204 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:48.202+0000 7f46c9381700 1 -- 172.21.15.132:0/1995065313 >> 172.21.15.132:0/1995065313 conn(0x7f46c40f1c70 msgr2=0x7f46c40f26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:14:48.204 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:48.202+0000 7f46c9381700 1 -- 172.21.15.132:0/1995065313 shutdown_connections 2024-02-20T00:14:48.204 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:48.202+0000 7f46c9381700 1 -- 172.21.15.132:0/1995065313 wait complete. 2024-02-20T00:14:48.205 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:14:48.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:14:47 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:14:46.881247+0000 mgr.smithi084.bmqrnf (mgr.14162) 146 : cluster [DBG] pgmap v139: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:14:49.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:14:48 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:14:48.201233+0000 mon.smithi084 (mon.0) 194 : audit [DBG] from='client.? 172.21.15.132:0/1995065313' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:14:49.900 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:14:49.901 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:14:50.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:14:49 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:14:48.881634+0000 mgr.smithi084.bmqrnf (mgr.14162) 147 : cluster [DBG] pgmap v140: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:14:51.810 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:51.809+0000 7f04055b2700 1 -- 172.21.15.132:0/2193603437 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f04000f51a0 msgr2=0x7f04000f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:14:51.810 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:51.809+0000 7f04055b2700 1 --2- 172.21.15.132:0/2193603437 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f04000f51a0 0x7f04000f55c0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 rx=0x7f03f4009fb0 tx=0x7f03f4009b30).stop 2024-02-20T00:14:51.810 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:51.809+0000 7f04055b2700 1 -- 172.21.15.132:0/2193603437 shutdown_connections 2024-02-20T00:14:51.811 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:51.809+0000 7f04055b2700 1 --2- 172.21.15.132:0/2193603437 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f04000f51a0 0x7f04000f55c0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:14:51.811 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:51.809+0000 7f04055b2700 1 -- 172.21.15.132:0/2193603437 >> 172.21.15.132:0/2193603437 conn(0x7f04000f09a0 msgr2=0x7f04000f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:14:51.811 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:51.809+0000 7f04055b2700 1 -- 172.21.15.132:0/2193603437 shutdown_connections 2024-02-20T00:14:51.811 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:51.809+0000 7f04055b2700 1 -- 172.21.15.132:0/2193603437 wait complete. 2024-02-20T00:14:51.811 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:51.809+0000 7f04055b2700 1 Processor -- start 2024-02-20T00:14:51.811 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:51.809+0000 7f04055b2700 1 -- start start 2024-02-20T00:14:51.813 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:51.809+0000 7f04055b2700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f04000f51a0 0x7f04001054d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:14:51.813 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:51.809+0000 7f04055b2700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04001059c0 con 0x7f04000f51a0 2024-02-20T00:14:51.813 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:51.810+0000 7f03fffff700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f04000f51a0 0x7f04001054d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:14:51.813 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:51.810+0000 7f03fffff700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f04000f51a0 0x7f04001054d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:38082/0 (socket says 172.21.15.132:38082) 2024-02-20T00:14:51.813 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:51.810+0000 7f03fffff700 1 -- 172.21.15.132:0/1676448178 learned_addr learned my addr 172.21.15.132:0/1676448178 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:14:51.813 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:51.810+0000 7f03fffff700 1 -- 172.21.15.132:0/1676448178 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03f4005040 con 0x7f04000f51a0 2024-02-20T00:14:51.813 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:51.810+0000 7f03fffff700 1 --2- 172.21.15.132:0/1676448178 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f04000f51a0 0x7f04001054d0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7f03f40089f0 tx=0x7f03f400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:14:51.813 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:51.810+0000 7f03fcff9700 1 -- 172.21.15.132:0/1676448178 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f03f4009320 con 0x7f04000f51a0 2024-02-20T00:14:51.813 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:51.810+0000 7f04055b2700 1 -- 172.21.15.132:0/1676448178 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0400105bc0 con 0x7f04000f51a0 2024-02-20T00:14:51.814 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:51.810+0000 7f03fcff9700 1 -- 172.21.15.132:0/1676448178 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f03f4013380 con 0x7f04000f51a0 2024-02-20T00:14:51.814 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:51.810+0000 7f03fcff9700 1 -- 172.21.15.132:0/1676448178 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f03f4014e20 con 0x7f04000f51a0 2024-02-20T00:14:51.814 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:51.810+0000 7f04055b2700 1 -- 172.21.15.132:0/1676448178 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0400106050 con 0x7f04000f51a0 2024-02-20T00:14:51.814 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:51.811+0000 7f03fcff9700 1 -- 172.21.15.132:0/1676448178 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f03f4014420 con 0x7f04000f51a0 2024-02-20T00:14:51.814 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:51.811+0000 7f04055b2700 1 -- 172.21.15.132:0/1676448178 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0400045ad0 con 0x7f04000f51a0 2024-02-20T00:14:51.814 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:51.811+0000 7f03fcff9700 1 --2- 172.21.15.132:0/1676448178 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f03e802ff10 0x7f03e80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:14:51.814 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:51.811+0000 7f03fcff9700 1 -- 172.21.15.132:0/1676448178 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f03f4040fe0 con 0x7f04000f51a0 2024-02-20T00:14:51.814 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:51.811+0000 7f03ff7fe700 1 --2- 172.21.15.132:0/1676448178 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f03e802ff10 0x7f03e80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:14:51.814 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:51.812+0000 7f03ff7fe700 1 --2- 172.21.15.132:0/1676448178 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f03e802ff10 0x7f03e80323d0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f03f00096a0 tx=0x7f03f0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:14:51.817 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:51.816+0000 7f03fcff9700 1 -- 172.21.15.132:0/1676448178 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f03f406d2b0 con 0x7f04000f51a0 2024-02-20T00:14:52.021 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:52.019+0000 7f04055b2700 1 -- 172.21.15.132:0/1676448178 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0400106f00 con 0x7f04000f51a0 2024-02-20T00:14:52.021 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:52.020+0000 7f03fcff9700 1 -- 172.21.15.132:0/1676448178 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f03f40095f0 con 0x7f04000f51a0 2024-02-20T00:14:52.022 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:14:52.022 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:14:52.023 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:52.022+0000 7f04055b2700 1 -- 172.21.15.132:0/1676448178 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f03e802ff10 msgr2=0x7f03e80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:14:52.023 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:52.022+0000 7f04055b2700 1 --2- 172.21.15.132:0/1676448178 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f03e802ff10 0x7f03e80323d0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f03f00096a0 tx=0x7f03f0006b40).stop 2024-02-20T00:14:52.023 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:52.022+0000 7f04055b2700 1 -- 172.21.15.132:0/1676448178 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f04000f51a0 msgr2=0x7f04001054d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:14:52.023 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:52.022+0000 7f04055b2700 1 --2- 172.21.15.132:0/1676448178 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f04000f51a0 0x7f04001054d0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7f03f40089f0 tx=0x7f03f400b040).stop 2024-02-20T00:14:52.023 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:52.022+0000 7f04055b2700 1 -- 172.21.15.132:0/1676448178 shutdown_connections 2024-02-20T00:14:52.024 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:52.022+0000 7f04055b2700 1 --2- 172.21.15.132:0/1676448178 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f03e802ff10 0x7f03e80323d0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:14:52.024 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:52.022+0000 7f04055b2700 1 --2- 172.21.15.132:0/1676448178 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f04000f51a0 0x7f04001054d0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:14:52.024 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:52.022+0000 7f04055b2700 1 -- 172.21.15.132:0/1676448178 >> 172.21.15.132:0/1676448178 conn(0x7f04000f09a0 msgr2=0x7f04000f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:14:52.024 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:52.022+0000 7f04055b2700 1 -- 172.21.15.132:0/1676448178 shutdown_connections 2024-02-20T00:14:52.024 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:52.022+0000 7f04055b2700 1 -- 172.21.15.132:0/1676448178 wait complete. 2024-02-20T00:14:52.024 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:14:52.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:14:51 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:14:50.881953+0000 mgr.smithi084.bmqrnf (mgr.14162) 148 : cluster [DBG] pgmap v141: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:14:53.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:14:52 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:14:52.020789+0000 mon.smithi084 (mon.0) 195 : audit [DBG] from='client.? 172.21.15.132:0/1676448178' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:14:53.644 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:14:53.644 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:14:54.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:14:53 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:14:52.882324+0000 mgr.smithi084.bmqrnf (mgr.14162) 149 : cluster [DBG] pgmap v142: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:14:55.465 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:55.463+0000 7ff1abfff700 1 -- 172.21.15.132:0/3926810448 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff1ac0f62a0 msgr2=0x7ff1ac0f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:14:55.466 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:55.463+0000 7ff1abfff700 1 --2- 172.21.15.132:0/3926810448 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff1ac0f62a0 0x7ff1ac0f66c0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 rx=0x7ff19c009fb0 tx=0x7ff19c009b30).stop 2024-02-20T00:14:55.466 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:55.463+0000 7ff1abfff700 1 -- 172.21.15.132:0/3926810448 shutdown_connections 2024-02-20T00:14:55.466 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:55.463+0000 7ff1abfff700 1 --2- 172.21.15.132:0/3926810448 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff1ac0f62a0 0x7ff1ac0f66c0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:14:55.466 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:55.463+0000 7ff1abfff700 1 -- 172.21.15.132:0/3926810448 >> 172.21.15.132:0/3926810448 conn(0x7ff1ac0f1aa0 msgr2=0x7ff1ac0f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:14:55.466 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:55.463+0000 7ff1abfff700 1 -- 172.21.15.132:0/3926810448 shutdown_connections 2024-02-20T00:14:55.466 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:55.463+0000 7ff1abfff700 1 -- 172.21.15.132:0/3926810448 wait complete. 2024-02-20T00:14:55.466 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:55.463+0000 7ff1abfff700 1 Processor -- start 2024-02-20T00:14:55.466 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:55.463+0000 7ff1abfff700 1 -- start start 2024-02-20T00:14:55.466 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:55.463+0000 7ff1abfff700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff1ac0f62a0 0x7ff1ac06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:14:55.466 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:55.463+0000 7ff1abfff700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1ac069e20 con 0x7ff1ac0f62a0 2024-02-20T00:14:55.466 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:55.464+0000 7ff1aaffd700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff1ac0f62a0 0x7ff1ac06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:14:55.467 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:55.464+0000 7ff1aaffd700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff1ac0f62a0 0x7ff1ac06b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:38090/0 (socket says 172.21.15.132:38090) 2024-02-20T00:14:55.467 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:55.464+0000 7ff1aaffd700 1 -- 172.21.15.132:0/1535494177 learned_addr learned my addr 172.21.15.132:0/1535494177 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:14:55.467 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:55.464+0000 7ff1aaffd700 1 -- 172.21.15.132:0/1535494177 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff19c005040 con 0x7ff1ac0f62a0 2024-02-20T00:14:55.467 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:55.464+0000 7ff1aaffd700 1 --2- 172.21.15.132:0/1535494177 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff1ac0f62a0 0x7ff1ac06b9a0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7ff19c009830 tx=0x7ff19c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:14:55.467 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:55.464+0000 7ff1937fe700 1 -- 172.21.15.132:0/1535494177 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff19c009320 con 0x7ff1ac0f62a0 2024-02-20T00:14:55.467 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:55.464+0000 7ff1abfff700 1 -- 172.21.15.132:0/1535494177 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1ac069fc0 con 0x7ff1ac0f62a0 2024-02-20T00:14:55.467 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:55.464+0000 7ff1937fe700 1 -- 172.21.15.132:0/1535494177 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff19c013380 con 0x7ff1ac0f62a0 2024-02-20T00:14:55.467 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:55.464+0000 7ff1937fe700 1 -- 172.21.15.132:0/1535494177 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff19c014420 con 0x7ff1ac0f62a0 2024-02-20T00:14:55.467 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:55.464+0000 7ff1abfff700 1 -- 172.21.15.132:0/1535494177 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1ac06a450 con 0x7ff1ac0f62a0 2024-02-20T00:14:55.468 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:55.465+0000 7ff1937fe700 1 -- 172.21.15.132:0/1535494177 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff19c014580 con 0x7ff1ac0f62a0 2024-02-20T00:14:55.468 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:55.465+0000 7ff1abfff700 1 -- 172.21.15.132:0/1535494177 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1ac059100 con 0x7ff1ac0f62a0 2024-02-20T00:14:55.468 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:55.465+0000 7ff1937fe700 1 --2- 172.21.15.132:0/1535494177 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7ff19402ffb0 0x7ff194032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:14:55.468 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:55.465+0000 7ff1937fe700 1 -- 172.21.15.132:0/1535494177 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff19c0411e0 con 0x7ff1ac0f62a0 2024-02-20T00:14:55.469 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:55.465+0000 7ff1aa7fc700 1 --2- 172.21.15.132:0/1535494177 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7ff19402ffb0 0x7ff194032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:14:55.469 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:55.466+0000 7ff1aa7fc700 1 --2- 172.21.15.132:0/1535494177 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7ff19402ffb0 0x7ff194032470 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7ff1a00096a0 tx=0x7ff1a0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:14:55.471 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:55.470+0000 7ff1937fe700 1 -- 172.21.15.132:0/1535494177 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff19c027770 con 0x7ff1ac0f62a0 2024-02-20T00:14:55.674 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:55.673+0000 7ff1abfff700 1 -- 172.21.15.132:0/1535494177 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff1ac045ad0 con 0x7ff1ac0f62a0 2024-02-20T00:14:55.675 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:55.673+0000 7ff1937fe700 1 -- 172.21.15.132:0/1535494177 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff19c004490 con 0x7ff1ac0f62a0 2024-02-20T00:14:55.675 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:14:55.675 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:14:55.676 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:55.675+0000 7ff1abfff700 1 -- 172.21.15.132:0/1535494177 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7ff19402ffb0 msgr2=0x7ff194032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:14:55.677 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:55.675+0000 7ff1abfff700 1 --2- 172.21.15.132:0/1535494177 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7ff19402ffb0 0x7ff194032470 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7ff1a00096a0 tx=0x7ff1a0006b40).stop 2024-02-20T00:14:55.677 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:55.675+0000 7ff1abfff700 1 -- 172.21.15.132:0/1535494177 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff1ac0f62a0 msgr2=0x7ff1ac06b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:14:55.677 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:55.675+0000 7ff1abfff700 1 --2- 172.21.15.132:0/1535494177 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff1ac0f62a0 0x7ff1ac06b9a0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7ff19c009830 tx=0x7ff19c00b040).stop 2024-02-20T00:14:55.677 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:55.675+0000 7ff1abfff700 1 -- 172.21.15.132:0/1535494177 shutdown_connections 2024-02-20T00:14:55.677 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:55.675+0000 7ff1abfff700 1 --2- 172.21.15.132:0/1535494177 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7ff19402ffb0 0x7ff194032470 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:14:55.677 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:55.675+0000 7ff1abfff700 1 --2- 172.21.15.132:0/1535494177 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff1ac0f62a0 0x7ff1ac06b9a0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:14:55.677 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:55.675+0000 7ff1abfff700 1 -- 172.21.15.132:0/1535494177 >> 172.21.15.132:0/1535494177 conn(0x7ff1ac0f1aa0 msgr2=0x7ff1ac0f2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:14:55.677 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:55.675+0000 7ff1abfff700 1 -- 172.21.15.132:0/1535494177 shutdown_connections 2024-02-20T00:14:55.677 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:55.675+0000 7ff1abfff700 1 -- 172.21.15.132:0/1535494177 wait complete. 2024-02-20T00:14:55.677 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:14:56.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:14:55 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:14:54.882697+0000 mgr.smithi084.bmqrnf (mgr.14162) 150 : cluster [DBG] pgmap v143: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:14:56.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:14:55 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:14:55.674090+0000 mon.smithi084 (mon.0) 196 : audit [DBG] from='client.? 172.21.15.132:0/1535494177' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:14:57.361 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:14:57.361 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:14:58.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:14:57 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:14:56.882969+0000 mgr.smithi084.bmqrnf (mgr.14162) 151 : cluster [DBG] pgmap v144: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:14:58.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:14:57 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:14:56.974618+0000 mon.smithi084 (mon.0) 197 : audit [INF] from='mgr.14162 172.21.15.84:0/2152933216' entity='mgr.smithi084.bmqrnf' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi084.bmqrnf/mirror_snapshot_schedule"}]: dispatch 2024-02-20T00:14:58.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:14:57 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:14:56.980629+0000 mon.smithi084 (mon.0) 198 : audit [INF] from='mgr.14162 172.21.15.84:0/2152933216' entity='mgr.smithi084.bmqrnf' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi084.bmqrnf/trash_purge_schedule"}]: dispatch 2024-02-20T00:14:59.138 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:59.135+0000 7eff10dbc700 1 -- 172.21.15.132:0/1842990826 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7eff0c0f5410 msgr2=0x7eff0c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:14:59.138 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:59.135+0000 7eff10dbc700 1 --2- 172.21.15.132:0/1842990826 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7eff0c0f5410 0x7eff0c0f5830 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 rx=0x7efef4009fb0 tx=0x7efef4009b30).stop 2024-02-20T00:14:59.138 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:59.135+0000 7eff10dbc700 1 -- 172.21.15.132:0/1842990826 shutdown_connections 2024-02-20T00:14:59.138 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:59.135+0000 7eff10dbc700 1 --2- 172.21.15.132:0/1842990826 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7eff0c0f5410 0x7eff0c0f5830 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:14:59.139 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:59.135+0000 7eff10dbc700 1 -- 172.21.15.132:0/1842990826 >> 172.21.15.132:0/1842990826 conn(0x7eff0c0f0b70 msgr2=0x7eff0c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:14:59.139 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:59.135+0000 7eff10dbc700 1 -- 172.21.15.132:0/1842990826 shutdown_connections 2024-02-20T00:14:59.139 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:59.135+0000 7eff10dbc700 1 -- 172.21.15.132:0/1842990826 wait complete. 2024-02-20T00:14:59.139 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:59.136+0000 7eff10dbc700 1 Processor -- start 2024-02-20T00:14:59.139 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:59.136+0000 7eff10dbc700 1 -- start start 2024-02-20T00:14:59.139 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:59.136+0000 7eff10dbc700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7eff0c0f5410 0x7eff0c1094e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:14:59.139 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:59.136+0000 7eff10dbc700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff0c060240 con 0x7eff0c0f5410 2024-02-20T00:14:59.141 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:59.136+0000 7eff0a59c700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7eff0c0f5410 0x7eff0c1094e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:14:59.141 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:59.136+0000 7eff0a59c700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7eff0c0f5410 0x7eff0c1094e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:53086/0 (socket says 172.21.15.132:53086) 2024-02-20T00:14:59.141 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:59.136+0000 7eff0a59c700 1 -- 172.21.15.132:0/3469054651 learned_addr learned my addr 172.21.15.132:0/3469054651 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:14:59.141 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:59.137+0000 7eff0a59c700 1 -- 172.21.15.132:0/3469054651 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efef4005040 con 0x7eff0c0f5410 2024-02-20T00:14:59.141 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:59.137+0000 7eff0a59c700 1 --2- 172.21.15.132:0/3469054651 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7eff0c0f5410 0x7eff0c1094e0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7efef4005010 tx=0x7efef400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:14:59.141 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:59.137+0000 7eff02ffd700 1 -- 172.21.15.132:0/3469054651 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efef4009320 con 0x7eff0c0f5410 2024-02-20T00:14:59.141 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:59.137+0000 7eff10dbc700 1 -- 172.21.15.132:0/3469054651 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff0c060440 con 0x7eff0c0f5410 2024-02-20T00:14:59.141 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:59.137+0000 7eff02ffd700 1 -- 172.21.15.132:0/3469054651 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efef4013380 con 0x7eff0c0f5410 2024-02-20T00:14:59.141 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:59.137+0000 7eff02ffd700 1 -- 172.21.15.132:0/3469054651 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efef4014e50 con 0x7eff0c0f5410 2024-02-20T00:14:59.141 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:59.137+0000 7eff10dbc700 1 -- 172.21.15.132:0/3469054651 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff0c109b90 con 0x7eff0c0f5410 2024-02-20T00:14:59.142 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:59.138+0000 7eff02ffd700 1 -- 172.21.15.132:0/3469054651 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7efef4014420 con 0x7eff0c0f5410 2024-02-20T00:14:59.142 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:59.138+0000 7eff10dbc700 1 -- 172.21.15.132:0/3469054651 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff0c045ad0 con 0x7eff0c0f5410 2024-02-20T00:14:59.142 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:59.138+0000 7eff02ffd700 1 --2- 172.21.15.132:0/3469054651 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7efef802ff10 0x7efef80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:14:59.142 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:59.138+0000 7eff02ffd700 1 -- 172.21.15.132:0/3469054651 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7efef4040f70 con 0x7eff0c0f5410 2024-02-20T00:14:59.142 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:59.138+0000 7eff09d9b700 1 --2- 172.21.15.132:0/3469054651 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7efef802ff10 0x7efef80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:14:59.142 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:59.139+0000 7eff09d9b700 1 --2- 172.21.15.132:0/3469054651 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7efef802ff10 0x7efef80323d0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7efefc0096a0 tx=0x7efefc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:14:59.145 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:59.143+0000 7eff02ffd700 1 -- 172.21.15.132:0/3469054651 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7efef40095f0 con 0x7eff0c0f5410 2024-02-20T00:14:59.354 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:59.352+0000 7eff10dbc700 1 -- 172.21.15.132:0/3469054651 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7eff0c10a3a0 con 0x7eff0c0f5410 2024-02-20T00:14:59.355 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:59.352+0000 7eff02ffd700 1 -- 172.21.15.132:0/3469054651 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7efef401db70 con 0x7eff0c0f5410 2024-02-20T00:14:59.355 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:14:59.355 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:14:59.356 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:59.354+0000 7eff10dbc700 1 -- 172.21.15.132:0/3469054651 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7efef802ff10 msgr2=0x7efef80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:14:59.357 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:59.354+0000 7eff10dbc700 1 --2- 172.21.15.132:0/3469054651 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7efef802ff10 0x7efef80323d0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7efefc0096a0 tx=0x7efefc006b40).stop 2024-02-20T00:14:59.357 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:59.354+0000 7eff10dbc700 1 -- 172.21.15.132:0/3469054651 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7eff0c0f5410 msgr2=0x7eff0c1094e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:14:59.357 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:59.354+0000 7eff10dbc700 1 --2- 172.21.15.132:0/3469054651 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7eff0c0f5410 0x7eff0c1094e0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7efef4005010 tx=0x7efef400b040).stop 2024-02-20T00:14:59.357 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:59.354+0000 7eff10dbc700 1 -- 172.21.15.132:0/3469054651 shutdown_connections 2024-02-20T00:14:59.357 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:59.354+0000 7eff10dbc700 1 --2- 172.21.15.132:0/3469054651 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7efef802ff10 0x7efef80323d0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:14:59.357 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:59.354+0000 7eff10dbc700 1 --2- 172.21.15.132:0/3469054651 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7eff0c0f5410 0x7eff0c1094e0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:14:59.357 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:59.354+0000 7eff10dbc700 1 -- 172.21.15.132:0/3469054651 >> 172.21.15.132:0/3469054651 conn(0x7eff0c0f0b70 msgr2=0x7eff0c0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:14:59.357 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:59.354+0000 7eff10dbc700 1 -- 172.21.15.132:0/3469054651 shutdown_connections 2024-02-20T00:14:59.357 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:14:59.354+0000 7eff10dbc700 1 -- 172.21.15.132:0/3469054651 wait complete. 2024-02-20T00:14:59.358 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:15:00.360 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:14:59 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:14:58.883256+0000 mgr.smithi084.bmqrnf (mgr.14162) 152 : cluster [DBG] pgmap v145: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:15:00.360 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:14:59 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:14:59.354082+0000 mon.smithi084 (mon.0) 199 : audit [DBG] from='client.? 172.21.15.132:0/3469054651' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:15:01.052 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:15:01.052 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:15:02.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:15:01 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:15:00.883550+0000 mgr.smithi084.bmqrnf (mgr.14162) 153 : cluster [DBG] pgmap v146: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:15:02.864 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:02.860+0000 7fc8de726700 1 -- 172.21.15.132:0/3579982908 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc8d80f5410 msgr2=0x7fc8d80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:15:02.864 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:02.860+0000 7fc8de726700 1 --2- 172.21.15.132:0/3579982908 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc8d80f5410 0x7fc8d80f5830 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 rx=0x7fc8c8009fb0 tx=0x7fc8c8009b30).stop 2024-02-20T00:15:02.864 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:02.860+0000 7fc8de726700 1 -- 172.21.15.132:0/3579982908 shutdown_connections 2024-02-20T00:15:02.864 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:02.860+0000 7fc8de726700 1 --2- 172.21.15.132:0/3579982908 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc8d80f5410 0x7fc8d80f5830 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:15:02.864 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:02.860+0000 7fc8de726700 1 -- 172.21.15.132:0/3579982908 >> 172.21.15.132:0/3579982908 conn(0x7fc8d80f0b70 msgr2=0x7fc8d80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:15:02.865 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:02.860+0000 7fc8de726700 1 -- 172.21.15.132:0/3579982908 shutdown_connections 2024-02-20T00:15:02.865 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:02.860+0000 7fc8de726700 1 -- 172.21.15.132:0/3579982908 wait complete. 2024-02-20T00:15:02.865 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:02.861+0000 7fc8de726700 1 Processor -- start 2024-02-20T00:15:02.865 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:02.861+0000 7fc8de726700 1 -- start start 2024-02-20T00:15:02.865 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:02.862+0000 7fc8de726700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc8d80f5410 0x7fc8d81056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:15:02.865 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:02.862+0000 7fc8de726700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8d8105bd0 con 0x7fc8d80f5410 2024-02-20T00:15:02.865 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:02.862+0000 7fc8d7fff700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc8d80f5410 0x7fc8d81056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:15:02.865 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:02.862+0000 7fc8d7fff700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc8d80f5410 0x7fc8d81056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:53116/0 (socket says 172.21.15.132:53116) 2024-02-20T00:15:02.865 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:02.862+0000 7fc8d7fff700 1 -- 172.21.15.132:0/1207673608 learned_addr learned my addr 172.21.15.132:0/1207673608 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:15:02.865 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:02.862+0000 7fc8d7fff700 1 -- 172.21.15.132:0/1207673608 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8c8005040 con 0x7fc8d80f5410 2024-02-20T00:15:02.865 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:02.862+0000 7fc8d7fff700 1 --2- 172.21.15.132:0/1207673608 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc8d80f5410 0x7fc8d81056e0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7fc8c8005010 tx=0x7fc8c800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:15:02.867 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:02.862+0000 7fc8d4ff9700 1 -- 172.21.15.132:0/1207673608 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc8c8009320 con 0x7fc8d80f5410 2024-02-20T00:15:02.867 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:02.862+0000 7fc8d4ff9700 1 -- 172.21.15.132:0/1207673608 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc8c8013380 con 0x7fc8d80f5410 2024-02-20T00:15:02.867 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:02.862+0000 7fc8de726700 1 -- 172.21.15.132:0/1207673608 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8d8105dd0 con 0x7fc8d80f5410 2024-02-20T00:15:02.867 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:02.862+0000 7fc8d4ff9700 1 -- 172.21.15.132:0/1207673608 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc8c8014e50 con 0x7fc8d80f5410 2024-02-20T00:15:02.867 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:02.862+0000 7fc8de726700 1 -- 172.21.15.132:0/1207673608 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8d8106260 con 0x7fc8d80f5410 2024-02-20T00:15:02.867 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:02.863+0000 7fc8d4ff9700 1 -- 172.21.15.132:0/1207673608 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc8c8014420 con 0x7fc8d80f5410 2024-02-20T00:15:02.867 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:02.863+0000 7fc8de726700 1 -- 172.21.15.132:0/1207673608 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8d8045ad0 con 0x7fc8d80f5410 2024-02-20T00:15:02.867 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:02.863+0000 7fc8d4ff9700 1 --2- 172.21.15.132:0/1207673608 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fc8c002ff10 0x7fc8c00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:15:02.867 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:02.863+0000 7fc8d4ff9700 1 -- 172.21.15.132:0/1207673608 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc8c8040f40 con 0x7fc8d80f5410 2024-02-20T00:15:02.867 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:02.864+0000 7fc8d77fe700 1 --2- 172.21.15.132:0/1207673608 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fc8c002ff10 0x7fc8c00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:15:02.867 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:02.864+0000 7fc8d77fe700 1 --2- 172.21.15.132:0/1207673608 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fc8c002ff10 0x7fc8c00323d0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7fc8cc0096a0 tx=0x7fc8cc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:15:02.870 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:02.868+0000 7fc8d4ff9700 1 -- 172.21.15.132:0/1207673608 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc8c80095f0 con 0x7fc8d80f5410 2024-02-20T00:15:03.073 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:03.070+0000 7fc8de726700 1 -- 172.21.15.132:0/1207673608 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc8d8107110 con 0x7fc8d80f5410 2024-02-20T00:15:03.073 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:03.071+0000 7fc8d4ff9700 1 -- 172.21.15.132:0/1207673608 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc8c801db70 con 0x7fc8d80f5410 2024-02-20T00:15:03.073 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:15:03.074 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:15:03.075 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:03.073+0000 7fc8de726700 1 -- 172.21.15.132:0/1207673608 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fc8c002ff10 msgr2=0x7fc8c00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:15:03.075 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:03.073+0000 7fc8de726700 1 --2- 172.21.15.132:0/1207673608 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fc8c002ff10 0x7fc8c00323d0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7fc8cc0096a0 tx=0x7fc8cc006b40).stop 2024-02-20T00:15:03.075 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:03.073+0000 7fc8de726700 1 -- 172.21.15.132:0/1207673608 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc8d80f5410 msgr2=0x7fc8d81056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:15:03.075 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:03.073+0000 7fc8de726700 1 --2- 172.21.15.132:0/1207673608 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc8d80f5410 0x7fc8d81056e0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7fc8c8005010 tx=0x7fc8c800b040).stop 2024-02-20T00:15:03.075 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:03.073+0000 7fc8de726700 1 -- 172.21.15.132:0/1207673608 shutdown_connections 2024-02-20T00:15:03.075 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:03.073+0000 7fc8de726700 1 --2- 172.21.15.132:0/1207673608 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fc8c002ff10 0x7fc8c00323d0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:15:03.075 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:03.073+0000 7fc8de726700 1 --2- 172.21.15.132:0/1207673608 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc8d80f5410 0x7fc8d81056e0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:15:03.075 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:03.073+0000 7fc8de726700 1 -- 172.21.15.132:0/1207673608 >> 172.21.15.132:0/1207673608 conn(0x7fc8d80f0b70 msgr2=0x7fc8d80f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:15:03.076 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:03.073+0000 7fc8de726700 1 -- 172.21.15.132:0/1207673608 shutdown_connections 2024-02-20T00:15:03.076 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:03.073+0000 7fc8de726700 1 -- 172.21.15.132:0/1207673608 wait complete. 2024-02-20T00:15:03.076 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:15:04.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:15:03 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:15:02.883885+0000 mgr.smithi084.bmqrnf (mgr.14162) 154 : cluster [DBG] pgmap v147: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:15:04.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:15:03 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:15:03.072732+0000 mon.smithi084 (mon.0) 200 : audit [DBG] from='client.? 172.21.15.132:0/1207673608' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:15:04.726 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:15:04.726 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:15:06.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:15:05 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:15:04.884307+0000 mgr.smithi084.bmqrnf (mgr.14162) 155 : cluster [DBG] pgmap v148: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:15:06.559 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:06.555+0000 7f0953db7700 1 -- 172.21.15.132:0/184243606 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f094c0f6510 msgr2=0x7f094c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:15:06.559 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:06.555+0000 7f0953db7700 1 --2- 172.21.15.132:0/184243606 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f094c0f6510 0x7f094c0f6930 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 rx=0x7f0948009fb0 tx=0x7f0948009b30).stop 2024-02-20T00:15:06.559 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:06.555+0000 7f0953db7700 1 -- 172.21.15.132:0/184243606 shutdown_connections 2024-02-20T00:15:06.560 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:06.555+0000 7f0953db7700 1 --2- 172.21.15.132:0/184243606 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f094c0f6510 0x7f094c0f6930 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:15:06.560 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:06.555+0000 7f0953db7700 1 -- 172.21.15.132:0/184243606 >> 172.21.15.132:0/184243606 conn(0x7f094c0f1c70 msgr2=0x7f094c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:15:06.560 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:06.556+0000 7f0953db7700 1 -- 172.21.15.132:0/184243606 shutdown_connections 2024-02-20T00:15:06.560 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:06.556+0000 7f0953db7700 1 -- 172.21.15.132:0/184243606 wait complete. 2024-02-20T00:15:06.560 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:06.556+0000 7f0953db7700 1 Processor -- start 2024-02-20T00:15:06.560 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:06.556+0000 7f0953db7700 1 -- start start 2024-02-20T00:15:06.560 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:06.556+0000 7f0953db7700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f094c0f6510 0x7f094c06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:15:06.560 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:06.556+0000 7f0953db7700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f094c069e20 con 0x7f094c0f6510 2024-02-20T00:15:06.560 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:06.556+0000 7f0951b53700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f094c0f6510 0x7f094c06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:15:06.560 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:06.556+0000 7f0951b53700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f094c0f6510 0x7f094c06b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:53136/0 (socket says 172.21.15.132:53136) 2024-02-20T00:15:06.560 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:06.556+0000 7f0951b53700 1 -- 172.21.15.132:0/3785571541 learned_addr learned my addr 172.21.15.132:0/3785571541 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:15:06.560 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:06.557+0000 7f0951b53700 1 -- 172.21.15.132:0/3785571541 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0948005040 con 0x7f094c0f6510 2024-02-20T00:15:06.560 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:06.557+0000 7f0951b53700 1 --2- 172.21.15.132:0/3785571541 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f094c0f6510 0x7f094c06b9a0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7f0948005010 tx=0x7f094800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:15:06.560 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:06.557+0000 7f09427fc700 1 -- 172.21.15.132:0/3785571541 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0948009320 con 0x7f094c0f6510 2024-02-20T00:15:06.561 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:06.557+0000 7f0953db7700 1 -- 172.21.15.132:0/3785571541 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f094c069fc0 con 0x7f094c0f6510 2024-02-20T00:15:06.561 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:06.557+0000 7f09427fc700 1 -- 172.21.15.132:0/3785571541 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0948013380 con 0x7f094c0f6510 2024-02-20T00:15:06.561 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:06.557+0000 7f09427fc700 1 -- 172.21.15.132:0/3785571541 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0948014e50 con 0x7f094c0f6510 2024-02-20T00:15:06.561 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:06.557+0000 7f0953db7700 1 -- 172.21.15.132:0/3785571541 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f094c06a450 con 0x7f094c0f6510 2024-02-20T00:15:06.562 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:06.558+0000 7f09427fc700 1 -- 172.21.15.132:0/3785571541 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f09480263a0 con 0x7f094c0f6510 2024-02-20T00:15:06.562 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:06.558+0000 7f0953db7700 1 -- 172.21.15.132:0/3785571541 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f094c059100 con 0x7f094c0f6510 2024-02-20T00:15:06.562 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:06.558+0000 7f09427fc700 1 --2- 172.21.15.132:0/3785571541 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f093802ff10 0x7f09380323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:15:06.562 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:06.558+0000 7f09427fc700 1 -- 172.21.15.132:0/3785571541 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0948041270 con 0x7f094c0f6510 2024-02-20T00:15:06.562 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:06.558+0000 7f0951352700 1 --2- 172.21.15.132:0/3785571541 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f093802ff10 0x7f09380323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:15:06.562 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:06.559+0000 7f0951352700 1 --2- 172.21.15.132:0/3785571541 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f093802ff10 0x7f09380323d0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f093c0096a0 tx=0x7f093c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:15:06.565 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:06.563+0000 7f09427fc700 1 -- 172.21.15.132:0/3785571541 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f09480126f0 con 0x7f094c0f6510 2024-02-20T00:15:06.773 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:06.771+0000 7f0953db7700 1 -- 172.21.15.132:0/3785571541 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f094c0fc0a0 con 0x7f094c0f6510 2024-02-20T00:15:06.774 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:06.772+0000 7f09427fc700 1 -- 172.21.15.132:0/3785571541 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0948012330 con 0x7f094c0f6510 2024-02-20T00:15:06.774 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:15:06.774 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:15:06.775 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:06.773+0000 7f0953db7700 1 -- 172.21.15.132:0/3785571541 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f093802ff10 msgr2=0x7f09380323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:15:06.776 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:06.773+0000 7f0953db7700 1 --2- 172.21.15.132:0/3785571541 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f093802ff10 0x7f09380323d0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f093c0096a0 tx=0x7f093c006b40).stop 2024-02-20T00:15:06.776 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:06.773+0000 7f0953db7700 1 -- 172.21.15.132:0/3785571541 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f094c0f6510 msgr2=0x7f094c06b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:15:06.776 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:06.773+0000 7f0953db7700 1 --2- 172.21.15.132:0/3785571541 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f094c0f6510 0x7f094c06b9a0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7f0948005010 tx=0x7f094800b040).stop 2024-02-20T00:15:06.776 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:06.773+0000 7f0953db7700 1 -- 172.21.15.132:0/3785571541 shutdown_connections 2024-02-20T00:15:06.776 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:06.773+0000 7f0953db7700 1 --2- 172.21.15.132:0/3785571541 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f093802ff10 0x7f09380323d0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:15:06.776 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:06.773+0000 7f0953db7700 1 --2- 172.21.15.132:0/3785571541 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f094c0f6510 0x7f094c06b9a0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:15:06.776 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:06.773+0000 7f0953db7700 1 -- 172.21.15.132:0/3785571541 >> 172.21.15.132:0/3785571541 conn(0x7f094c0f1c70 msgr2=0x7f094c0f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:15:06.776 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:06.773+0000 7f0953db7700 1 -- 172.21.15.132:0/3785571541 shutdown_connections 2024-02-20T00:15:06.776 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:06.773+0000 7f0953db7700 1 -- 172.21.15.132:0/3785571541 wait complete. 2024-02-20T00:15:06.776 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:15:07.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:15:06 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:15:06.773464+0000 mon.smithi084 (mon.0) 201 : audit [DBG] from='client.? 172.21.15.132:0/3785571541' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:15:08.346 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:15:08.347 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:15:08.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:15:07 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:15:06.884623+0000 mgr.smithi084.bmqrnf (mgr.14162) 156 : cluster [DBG] pgmap v149: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:15:10.241 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:10.237+0000 7f885cbd0700 1 -- 172.21.15.132:0/2710158925 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8858060280 msgr2=0x7f88580f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:15:10.241 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:10.237+0000 7f885cbd0700 1 --2- 172.21.15.132:0/2710158925 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8858060280 0x7f88580f90f0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 rx=0x7f884c009fb0 tx=0x7f884c009b30).stop 2024-02-20T00:15:10.241 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:10.237+0000 7f885cbd0700 1 -- 172.21.15.132:0/2710158925 shutdown_connections 2024-02-20T00:15:10.241 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:10.237+0000 7f885cbd0700 1 --2- 172.21.15.132:0/2710158925 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8858060280 0x7f88580f90f0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:15:10.241 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:10.237+0000 7f885cbd0700 1 -- 172.21.15.132:0/2710158925 >> 172.21.15.132:0/2710158925 conn(0x7f88580f0b70 msgr2=0x7f88580f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:15:10.241 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:10.237+0000 7f885cbd0700 1 -- 172.21.15.132:0/2710158925 shutdown_connections 2024-02-20T00:15:10.241 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:10.237+0000 7f885cbd0700 1 -- 172.21.15.132:0/2710158925 wait complete. 2024-02-20T00:15:10.241 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:10.237+0000 7f885cbd0700 1 Processor -- start 2024-02-20T00:15:10.242 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:10.237+0000 7f885cbd0700 1 -- start start 2024-02-20T00:15:10.242 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:10.238+0000 7f885cbd0700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8858060280 0x7f88580f6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:15:10.242 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:10.238+0000 7f885cbd0700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88580f4ff0 con 0x7f8858060280 2024-02-20T00:15:10.242 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:10.238+0000 7f885659c700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8858060280 0x7f88580f6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:15:10.242 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:10.238+0000 7f885659c700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8858060280 0x7f88580f6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:51792/0 (socket says 172.21.15.132:51792) 2024-02-20T00:15:10.242 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:10.238+0000 7f885659c700 1 -- 172.21.15.132:0/313307957 learned_addr learned my addr 172.21.15.132:0/313307957 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:15:10.242 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:10.238+0000 7f885659c700 1 -- 172.21.15.132:0/313307957 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f884c005040 con 0x7f8858060280 2024-02-20T00:15:10.242 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:10.238+0000 7f885659c700 1 --2- 172.21.15.132:0/313307957 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8858060280 0x7f88580f6b70 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7f884c0089f0 tx=0x7f884c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:15:10.242 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:10.238+0000 7f8846ffd700 1 -- 172.21.15.132:0/313307957 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f884c009320 con 0x7f8858060280 2024-02-20T00:15:10.242 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:10.238+0000 7f885cbd0700 1 -- 172.21.15.132:0/313307957 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88580f5190 con 0x7f8858060280 2024-02-20T00:15:10.242 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:10.238+0000 7f8846ffd700 1 -- 172.21.15.132:0/313307957 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f884c013380 con 0x7f8858060280 2024-02-20T00:15:10.242 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:10.238+0000 7f8846ffd700 1 -- 172.21.15.132:0/313307957 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f884c014e20 con 0x7f8858060280 2024-02-20T00:15:10.242 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:10.238+0000 7f885cbd0700 1 -- 172.21.15.132:0/313307957 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88580f5620 con 0x7f8858060280 2024-02-20T00:15:10.243 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:10.239+0000 7f8846ffd700 1 -- 172.21.15.132:0/313307957 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f884c0263a0 con 0x7f8858060280 2024-02-20T00:15:10.243 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:10.239+0000 7f885cbd0700 1 -- 172.21.15.132:0/313307957 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8858059100 con 0x7f8858060280 2024-02-20T00:15:10.243 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:10.239+0000 7f8846ffd700 1 --2- 172.21.15.132:0/313307957 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f884002ff10 0x7f88400323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:15:10.243 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:10.239+0000 7f8846ffd700 1 -- 172.21.15.132:0/313307957 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f884c041210 con 0x7f8858060280 2024-02-20T00:15:10.243 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:10.239+0000 7f8855d9b700 1 --2- 172.21.15.132:0/313307957 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f884002ff10 0x7f88400323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:15:10.244 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:10.240+0000 7f8855d9b700 1 --2- 172.21.15.132:0/313307957 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f884002ff10 0x7f88400323d0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f88480096a0 tx=0x7f8848006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:15:10.246 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:10.244+0000 7f8846ffd700 1 -- 172.21.15.132:0/313307957 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f884c01ccf0 con 0x7f8858060280 2024-02-20T00:15:10.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:15:09 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:15:08.884835+0000 mgr.smithi084.bmqrnf (mgr.14162) 157 : cluster [DBG] pgmap v150: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:15:10.451 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:10.449+0000 7f885cbd0700 1 -- 172.21.15.132:0/313307957 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f88580fc090 con 0x7f8858060280 2024-02-20T00:15:10.452 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:10.449+0000 7f8846ffd700 1 -- 172.21.15.132:0/313307957 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f884c01ccf0 con 0x7f8858060280 2024-02-20T00:15:10.452 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:15:10.452 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:15:10.453 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:10.451+0000 7f885cbd0700 1 -- 172.21.15.132:0/313307957 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f884002ff10 msgr2=0x7f88400323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:15:10.453 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:10.451+0000 7f885cbd0700 1 --2- 172.21.15.132:0/313307957 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f884002ff10 0x7f88400323d0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f88480096a0 tx=0x7f8848006b40).stop 2024-02-20T00:15:10.454 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:10.451+0000 7f885cbd0700 1 -- 172.21.15.132:0/313307957 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8858060280 msgr2=0x7f88580f6b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:15:10.454 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:10.451+0000 7f885cbd0700 1 --2- 172.21.15.132:0/313307957 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8858060280 0x7f88580f6b70 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7f884c0089f0 tx=0x7f884c00b040).stop 2024-02-20T00:15:10.454 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:10.451+0000 7f885cbd0700 1 -- 172.21.15.132:0/313307957 shutdown_connections 2024-02-20T00:15:10.454 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:10.451+0000 7f885cbd0700 1 --2- 172.21.15.132:0/313307957 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f884002ff10 0x7f88400323d0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:15:10.454 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:10.451+0000 7f885cbd0700 1 --2- 172.21.15.132:0/313307957 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8858060280 0x7f88580f6b70 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:15:10.454 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:10.451+0000 7f885cbd0700 1 -- 172.21.15.132:0/313307957 >> 172.21.15.132:0/313307957 conn(0x7f88580f0b70 msgr2=0x7f88580f15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:15:10.454 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:10.451+0000 7f885cbd0700 1 -- 172.21.15.132:0/313307957 shutdown_connections 2024-02-20T00:15:10.454 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:10.451+0000 7f885cbd0700 1 -- 172.21.15.132:0/313307957 wait complete. 2024-02-20T00:15:10.454 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:15:11.216 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:15:10 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:15:10.451063+0000 mon.smithi084 (mon.0) 202 : audit [DBG] from='client.? 172.21.15.132:0/313307957' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:15:12.108 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:15:11 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:15:10.885133+0000 mgr.smithi084.bmqrnf (mgr.14162) 158 : cluster [DBG] pgmap v151: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:15:12.109 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:15:11 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cephadm 2024-02-20T00:15:11.215685+0000 mgr.smithi084.bmqrnf (mgr.14162) 159 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Deploy daemon alertmanager.smithi084 ... 2024-02-20T00:15:12.109 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:15:11 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi084 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/alertmanager 2024-02-20T00:15:12.109 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:15:11 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2024-02-20T00:15:12.109 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:15:11 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: stat: stderr time="2024-02-20T00:10:43Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-20T00:15:12.109 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:15:11 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: stat: stderr time="2024-02-20T00:11:21Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-20T00:15:12.109 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:15:11 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: stat: stderr time="2024-02-20T00:11:58Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-20T00:15:12.109 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:15:11 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-20T00:15:12.109 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:15:11 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi084 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/prometheus 2024-02-20T00:15:12.109 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:15:11 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2024-02-20T00:15:12.109 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:15:11 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: stat: stderr time="2024-02-20T00:13:12Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-20T00:15:12.109 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:15:11 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: stat: stderr time="2024-02-20T00:13:50Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-20T00:15:12.109 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:15:11 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: stat: stderr time="2024-02-20T00:14:27Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-20T00:15:12.109 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:15:11 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: stat: stderr Error: copying system image from manifest list: determining manifest MIME type for docker://prom/alertmanager:v0.20.0: reading manifest sha256:b9323917a2eda265bec69e59a457f001c529facbbc8166df277f4850cdac61a0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-20T00:15:12.109 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:15:11 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: Traceback (most recent call last): 2024-02-20T00:15:12.110 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:15:11 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: File "/var/lib/ceph/2a2e6c42-cf84-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-02-20T00:15:12.110 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:15:11 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: main() 2024-02-20T00:15:12.110 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:15:11 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: File "/var/lib/ceph/2a2e6c42-cf84-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-02-20T00:15:12.110 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:15:11 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: r = ctx.func(ctx) 2024-02-20T00:15:12.110 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:15:11 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: File "/var/lib/ceph/2a2e6c42-cf84-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-02-20T00:15:12.110 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:15:11 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: return func(ctx) 2024-02-20T00:15:12.110 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:15:11 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: File "/var/lib/ceph/2a2e6c42-cf84-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy 2024-02-20T00:15:12.110 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:15:11 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2024-02-20T00:15:12.110 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:15:11 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: File "/var/lib/ceph/2a2e6c42-cf84-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4191, in extract_uid_gid_monitoring 2024-02-20T00:15:12.110 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:15:11 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: uid, gid = extract_uid_gid(ctx, file_path=['/etc/alertmanager', '/etc/prometheus']) 2024-02-20T00:15:12.110 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:15:11 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: File "/var/lib/ceph/2a2e6c42-cf84-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid 2024-02-20T00:15:12.110 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:15:11 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: raise RuntimeError('uid/gid not found') 2024-02-20T00:15:12.110 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:15:11 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: RuntimeError: uid/gid not found 2024-02-20T00:15:12.111 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:15:11 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: Traceback (most recent call last): 2024-02-20T00:15:12.111 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:15:11 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1144, in _remote_connection 2024-02-20T00:15:12.111 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:15:11 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: yield (conn, connr) 2024-02-20T00:15:12.111 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:15:11 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1059, in _run_cephadm 2024-02-20T00:15:12.111 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:15:11 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: code, '\n'.join(err))) 2024-02-20T00:15:12.111 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:15:11 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Deploy daemon alertmanager.smithi084 ... 2024-02-20T00:15:12.111 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:15:11 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi084 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/alertmanager 2024-02-20T00:15:12.111 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:15:11 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2024-02-20T00:15:12.111 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:15:11 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: stat: stderr time="2024-02-20T00:10:43Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-20T00:15:12.111 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:15:11 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: stat: stderr time="2024-02-20T00:11:21Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-20T00:15:12.111 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:15:11 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: stat: stderr time="2024-02-20T00:11:58Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-20T00:15:12.111 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:15:11 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-20T00:15:12.111 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:15:11 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi084 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/prometheus 2024-02-20T00:15:12.112 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:15:11 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2024-02-20T00:15:12.112 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:15:11 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: stat: stderr time="2024-02-20T00:13:12Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-20T00:15:12.112 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:15:11 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: stat: stderr time="2024-02-20T00:13:50Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-20T00:15:12.112 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:15:11 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: stat: stderr time="2024-02-20T00:14:27Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-20T00:15:12.112 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:15:11 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: stat: stderr Error: copying system image from manifest list: determining manifest MIME type for docker://prom/alertmanager:v0.20.0: reading manifest sha256:b9323917a2eda265bec69e59a457f001c529facbbc8166df277f4850cdac61a0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-20T00:15:12.112 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:15:11 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: Traceback (most recent call last): 2024-02-20T00:15:12.112 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:15:11 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: File "/var/lib/ceph/2a2e6c42-cf84-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-02-20T00:15:12.112 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:15:11 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: main() 2024-02-20T00:15:12.112 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:15:11 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: File "/var/lib/ceph/2a2e6c42-cf84-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-02-20T00:15:12.112 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:15:11 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: r = ctx.func(ctx) 2024-02-20T00:15:12.112 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:15:11 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: File "/var/lib/ceph/2a2e6c42-cf84-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-02-20T00:15:12.112 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:15:11 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: return func(ctx) 2024-02-20T00:15:12.112 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:15:11 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: File "/var/lib/ceph/2a2e6c42-cf84-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy 2024-02-20T00:15:12.113 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:15:11 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2024-02-20T00:15:12.113 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:15:11 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: File "/var/lib/ceph/2a2e6c42-cf84-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4191, in extract_uid_gid_monitoring 2024-02-20T00:15:12.113 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:15:11 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: uid, gid = extract_uid_gid(ctx, file_path=['/etc/alertmanager', '/etc/prometheus']) 2024-02-20T00:15:12.113 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:15:11 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: File "/var/lib/ceph/2a2e6c42-cf84-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid 2024-02-20T00:15:12.113 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:15:11 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: raise RuntimeError('uid/gid not found') 2024-02-20T00:15:12.113 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:15:11 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: RuntimeError: uid/gid not found 2024-02-20T00:15:12.113 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:15:11 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:15:11.216900+0000 mon.smithi084 (mon.0) 203 : audit [INF] from='mgr.14162 172.21.15.84:0/2152933216' entity='mgr.smithi084.bmqrnf' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi084", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-20T00:15:12.113 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:15:11 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:15:11.221643+0000 mon.smithi084 (mon.0) 204 : audit [INF] from='mgr.14162 172.21.15.84:0/2152933216' entity='mgr.smithi084.bmqrnf' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi084", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-02-20T00:15:12.113 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:15:11 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:15:11.222408+0000 mon.smithi084 (mon.0) 205 : audit [DBG] from='mgr.14162 172.21.15.84:0/2152933216' entity='mgr.smithi084.bmqrnf' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T00:15:12.113 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:15:11 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cephadm 2024-02-20T00:15:11.223122+0000 mgr.smithi084.bmqrnf (mgr.14162) 160 : cephadm [INF] Deploying daemon crash.smithi084 on smithi084 2024-02-20T00:15:12.130 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:15:12.131 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:15:13.967 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:13.960+0000 7f2619f2d700 1 -- 172.21.15.132:0/821297950 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f26140f5410 msgr2=0x7f26140f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:15:13.967 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:13.960+0000 7f2619f2d700 1 --2- 172.21.15.132:0/821297950 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f26140f5410 0x7f26140f5830 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 rx=0x7f2608009fb0 tx=0x7f2608009b30).stop 2024-02-20T00:15:13.967 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:13.960+0000 7f2619f2d700 1 -- 172.21.15.132:0/821297950 shutdown_connections 2024-02-20T00:15:13.967 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:13.960+0000 7f2619f2d700 1 --2- 172.21.15.132:0/821297950 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f26140f5410 0x7f26140f5830 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:15:13.967 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:13.960+0000 7f2619f2d700 1 -- 172.21.15.132:0/821297950 >> 172.21.15.132:0/821297950 conn(0x7f26140f0b70 msgr2=0x7f26140f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:15:13.968 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:13.960+0000 7f2619f2d700 1 -- 172.21.15.132:0/821297950 shutdown_connections 2024-02-20T00:15:13.968 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:13.960+0000 7f2619f2d700 1 -- 172.21.15.132:0/821297950 wait complete. 2024-02-20T00:15:13.968 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:13.960+0000 7f2619f2d700 1 Processor -- start 2024-02-20T00:15:13.968 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:13.961+0000 7f2619f2d700 1 -- start start 2024-02-20T00:15:13.968 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:13.961+0000 7f2619f2d700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f26140f5410 0x7f26141067d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:15:13.968 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:13.961+0000 7f2619f2d700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2614106cc0 con 0x7f26140f5410 2024-02-20T00:15:13.968 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:13.961+0000 7f26137fe700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f26140f5410 0x7f26141067d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:15:13.968 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:13.961+0000 7f26137fe700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f26140f5410 0x7f26141067d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:51804/0 (socket says 172.21.15.132:51804) 2024-02-20T00:15:13.968 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:13.961+0000 7f26137fe700 1 -- 172.21.15.132:0/2850617881 learned_addr learned my addr 172.21.15.132:0/2850617881 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:15:13.968 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:13.961+0000 7f26137fe700 1 -- 172.21.15.132:0/2850617881 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2608005040 con 0x7f26140f5410 2024-02-20T00:15:13.968 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:13.962+0000 7f26137fe700 1 --2- 172.21.15.132:0/2850617881 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f26140f5410 0x7f26141067d0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7f2608005010 tx=0x7f260800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:15:13.968 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:13.962+0000 7f25fbfff700 1 -- 172.21.15.132:0/2850617881 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2608009320 con 0x7f26140f5410 2024-02-20T00:15:13.969 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:13.962+0000 7f2619f2d700 1 -- 172.21.15.132:0/2850617881 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2614106ec0 con 0x7f26140f5410 2024-02-20T00:15:13.969 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:13.962+0000 7f25fbfff700 1 -- 172.21.15.132:0/2850617881 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2608013380 con 0x7f26140f5410 2024-02-20T00:15:13.969 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:13.962+0000 7f25fbfff700 1 -- 172.21.15.132:0/2850617881 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2608014e50 con 0x7f26140f5410 2024-02-20T00:15:13.969 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:13.962+0000 7f2619f2d700 1 -- 172.21.15.132:0/2850617881 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2614107350 con 0x7f26140f5410 2024-02-20T00:15:13.969 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:13.963+0000 7f25fbfff700 1 -- 172.21.15.132:0/2850617881 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f26080263a0 con 0x7f26140f5410 2024-02-20T00:15:13.969 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:13.963+0000 7f2619f2d700 1 -- 172.21.15.132:0/2850617881 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2614045ad0 con 0x7f26140f5410 2024-02-20T00:15:13.969 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:13.963+0000 7f25fbfff700 1 --2- 172.21.15.132:0/2850617881 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f25fc02ff10 0x7f25fc0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:15:13.969 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:13.963+0000 7f25fbfff700 1 -- 172.21.15.132:0/2850617881 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f26080412b0 con 0x7f26140f5410 2024-02-20T00:15:13.969 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:13.963+0000 7f2612ffd700 1 --2- 172.21.15.132:0/2850617881 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f25fc02ff10 0x7f25fc0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:15:13.969 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:13.963+0000 7f2612ffd700 1 --2- 172.21.15.132:0/2850617881 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f25fc02ff10 0x7f25fc0323d0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f26040096a0 tx=0x7f2604006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:15:13.970 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:13.967+0000 7f25fbfff700 1 -- 172.21.15.132:0/2850617881 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2608014420 con 0x7f26140f5410 2024-02-20T00:15:14.176 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:14.173+0000 7f2619f2d700 1 -- 172.21.15.132:0/2850617881 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2614108200 con 0x7f26140f5410 2024-02-20T00:15:14.176 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:14.174+0000 7f25fbfff700 1 -- 172.21.15.132:0/2850617881 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2608014420 con 0x7f26140f5410 2024-02-20T00:15:14.176 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:15:14.177 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:15:14.178 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:14.176+0000 7f2619f2d700 1 -- 172.21.15.132:0/2850617881 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f25fc02ff10 msgr2=0x7f25fc0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:15:14.178 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:14.176+0000 7f2619f2d700 1 --2- 172.21.15.132:0/2850617881 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f25fc02ff10 0x7f25fc0323d0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f26040096a0 tx=0x7f2604006b40).stop 2024-02-20T00:15:14.178 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:14.176+0000 7f2619f2d700 1 -- 172.21.15.132:0/2850617881 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f26140f5410 msgr2=0x7f26141067d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:15:14.178 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:14.176+0000 7f2619f2d700 1 --2- 172.21.15.132:0/2850617881 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f26140f5410 0x7f26141067d0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7f2608005010 tx=0x7f260800b040).stop 2024-02-20T00:15:14.178 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:14.176+0000 7f2619f2d700 1 -- 172.21.15.132:0/2850617881 shutdown_connections 2024-02-20T00:15:14.179 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:14.176+0000 7f2619f2d700 1 --2- 172.21.15.132:0/2850617881 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f25fc02ff10 0x7f25fc0323d0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:15:14.179 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:14.176+0000 7f2619f2d700 1 --2- 172.21.15.132:0/2850617881 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f26140f5410 0x7f26141067d0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:15:14.179 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:14.176+0000 7f2619f2d700 1 -- 172.21.15.132:0/2850617881 >> 172.21.15.132:0/2850617881 conn(0x7f26140f0b70 msgr2=0x7f261405fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:15:14.179 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:14.176+0000 7f2619f2d700 1 -- 172.21.15.132:0/2850617881 shutdown_connections 2024-02-20T00:15:14.179 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:14.176+0000 7f2619f2d700 1 -- 172.21.15.132:0/2850617881 wait complete. 2024-02-20T00:15:14.179 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:15:14.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:15:13 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:15:12.885392+0000 mgr.smithi084.bmqrnf (mgr.14162) 161 : cluster [DBG] pgmap v152: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:15:14.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:15:13 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:15:13.379886+0000 mon.smithi084 (mon.0) 206 : audit [INF] from='mgr.14162 172.21.15.84:0/2152933216' entity='mgr.smithi084.bmqrnf' 2024-02-20T00:15:14.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:15:13 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:15:13.455910+0000 mon.smithi084 (mon.0) 207 : audit [INF] from='mgr.14162 172.21.15.84:0/2152933216' entity='mgr.smithi084.bmqrnf' 2024-02-20T00:15:14.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:15:13 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:15:13.458254+0000 mon.smithi084 (mon.0) 208 : audit [INF] from='mgr.14162 172.21.15.84:0/2152933216' entity='mgr.smithi084.bmqrnf' 2024-02-20T00:15:14.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:15:13 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:15:13.458677+0000 mon.smithi084 (mon.0) 209 : audit [INF] from='mgr.14162 172.21.15.84:0/2152933216' entity='mgr.smithi084.bmqrnf' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-20T00:15:14.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:15:13 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:15:13.461343+0000 mon.smithi084 (mon.0) 210 : audit [INF] from='mgr.14162 172.21.15.84:0/2152933216' entity='mgr.smithi084.bmqrnf' 2024-02-20T00:15:15.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:15:14 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:15:13.459110+0000 mgr.smithi084.bmqrnf (mgr.14162) 162 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-20T00:15:15.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:15:14 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cephadm 2024-02-20T00:15:13.463643+0000 mgr.smithi084.bmqrnf (mgr.14162) 163 : cephadm [INF] Deploying daemon grafana.smithi084 on smithi084 2024-02-20T00:15:15.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:15:14 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:15:14.175610+0000 mon.smithi084 (mon.0) 211 : audit [DBG] from='client.? 172.21.15.132:0/2850617881' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:15:15.808 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:15:15.809 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:15:16.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:15:15 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:15:14.885715+0000 mgr.smithi084.bmqrnf (mgr.14162) 164 : cluster [DBG] pgmap v153: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:15:17.621 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:17.618+0000 7f2f4a20a700 1 -- 172.21.15.132:0/3023472663 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2f440f51a0 msgr2=0x7f2f440f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:15:17.622 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:17.618+0000 7f2f4a20a700 1 --2- 172.21.15.132:0/3023472663 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2f440f51a0 0x7f2f440f55c0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 rx=0x7f2f40009fb0 tx=0x7f2f40009b30).stop 2024-02-20T00:15:17.622 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:17.618+0000 7f2f4a20a700 1 -- 172.21.15.132:0/3023472663 shutdown_connections 2024-02-20T00:15:17.622 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:17.618+0000 7f2f4a20a700 1 --2- 172.21.15.132:0/3023472663 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2f440f51a0 0x7f2f440f55c0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:15:17.622 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:17.618+0000 7f2f4a20a700 1 -- 172.21.15.132:0/3023472663 >> 172.21.15.132:0/3023472663 conn(0x7f2f440f09a0 msgr2=0x7f2f440f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:15:17.622 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:17.618+0000 7f2f4a20a700 1 -- 172.21.15.132:0/3023472663 shutdown_connections 2024-02-20T00:15:17.622 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:17.618+0000 7f2f4a20a700 1 -- 172.21.15.132:0/3023472663 wait complete. 2024-02-20T00:15:17.622 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:17.618+0000 7f2f4a20a700 1 Processor -- start 2024-02-20T00:15:17.622 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:17.618+0000 7f2f4a20a700 1 -- start start 2024-02-20T00:15:17.622 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:17.618+0000 7f2f4a20a700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2f440f51a0 0x7f2f441054d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:15:17.622 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:17.618+0000 7f2f4a20a700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f441059c0 con 0x7f2f440f51a0 2024-02-20T00:15:17.622 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:17.619+0000 7f2f49208700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2f440f51a0 0x7f2f441054d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:15:17.622 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:17.619+0000 7f2f49208700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2f440f51a0 0x7f2f441054d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:55556/0 (socket says 172.21.15.132:55556) 2024-02-20T00:15:17.622 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:17.619+0000 7f2f49208700 1 -- 172.21.15.132:0/3855992881 learned_addr learned my addr 172.21.15.132:0/3855992881 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:15:17.623 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:17.619+0000 7f2f49208700 1 -- 172.21.15.132:0/3855992881 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f40005040 con 0x7f2f440f51a0 2024-02-20T00:15:17.623 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:17.619+0000 7f2f49208700 1 --2- 172.21.15.132:0/3855992881 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2f440f51a0 0x7f2f441054d0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7f2f400089f0 tx=0x7f2f4000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:15:17.623 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:17.619+0000 7f2f39ffb700 1 -- 172.21.15.132:0/3855992881 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2f40009320 con 0x7f2f440f51a0 2024-02-20T00:15:17.623 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:17.619+0000 7f2f4a20a700 1 -- 172.21.15.132:0/3855992881 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f44105bc0 con 0x7f2f440f51a0 2024-02-20T00:15:17.623 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:17.619+0000 7f2f39ffb700 1 -- 172.21.15.132:0/3855992881 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2f40013380 con 0x7f2f440f51a0 2024-02-20T00:15:17.623 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:17.619+0000 7f2f39ffb700 1 -- 172.21.15.132:0/3855992881 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2f40014e20 con 0x7f2f440f51a0 2024-02-20T00:15:17.623 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:17.619+0000 7f2f4a20a700 1 -- 172.21.15.132:0/3855992881 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f44106050 con 0x7f2f440f51a0 2024-02-20T00:15:17.624 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:17.620+0000 7f2f39ffb700 1 -- 172.21.15.132:0/3855992881 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2f400095f0 con 0x7f2f440f51a0 2024-02-20T00:15:17.624 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:17.620+0000 7f2f4a20a700 1 -- 172.21.15.132:0/3855992881 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f44045ad0 con 0x7f2f440f51a0 2024-02-20T00:15:17.624 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:17.621+0000 7f2f39ffb700 1 --2- 172.21.15.132:0/3855992881 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f2f3002ff10 0x7f2f300323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:15:17.624 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:17.621+0000 7f2f39ffb700 1 -- 172.21.15.132:0/3855992881 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2f40040ec0 con 0x7f2f440f51a0 2024-02-20T00:15:17.624 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:17.621+0000 7f2f48a07700 1 --2- 172.21.15.132:0/3855992881 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f2f3002ff10 0x7f2f300323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:15:17.624 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:17.621+0000 7f2f48a07700 1 --2- 172.21.15.132:0/3855992881 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f2f3002ff10 0x7f2f300323d0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f2f340096a0 tx=0x7f2f34006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:15:17.627 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:17.626+0000 7f2f39ffb700 1 -- 172.21.15.132:0/3855992881 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2f40014420 con 0x7f2f440f51a0 2024-02-20T00:15:17.831 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:17.828+0000 7f2f4a20a700 1 -- 172.21.15.132:0/3855992881 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2f44106f00 con 0x7f2f440f51a0 2024-02-20T00:15:17.831 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:17.829+0000 7f2f39ffb700 1 -- 172.21.15.132:0/3855992881 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2f40010230 con 0x7f2f440f51a0 2024-02-20T00:15:17.832 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:15:17.832 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:15:17.833 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:17.831+0000 7f2f4a20a700 1 -- 172.21.15.132:0/3855992881 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f2f3002ff10 msgr2=0x7f2f300323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:15:17.834 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:17.831+0000 7f2f4a20a700 1 --2- 172.21.15.132:0/3855992881 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f2f3002ff10 0x7f2f300323d0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f2f340096a0 tx=0x7f2f34006b40).stop 2024-02-20T00:15:17.834 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:17.831+0000 7f2f4a20a700 1 -- 172.21.15.132:0/3855992881 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2f440f51a0 msgr2=0x7f2f441054d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:15:17.834 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:17.831+0000 7f2f4a20a700 1 --2- 172.21.15.132:0/3855992881 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2f440f51a0 0x7f2f441054d0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7f2f400089f0 tx=0x7f2f4000b040).stop 2024-02-20T00:15:17.834 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:17.831+0000 7f2f4a20a700 1 -- 172.21.15.132:0/3855992881 shutdown_connections 2024-02-20T00:15:17.834 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:17.831+0000 7f2f4a20a700 1 --2- 172.21.15.132:0/3855992881 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f2f3002ff10 0x7f2f300323d0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:15:17.834 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:17.831+0000 7f2f4a20a700 1 --2- 172.21.15.132:0/3855992881 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2f440f51a0 0x7f2f441054d0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:15:17.834 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:17.831+0000 7f2f4a20a700 1 -- 172.21.15.132:0/3855992881 >> 172.21.15.132:0/3855992881 conn(0x7f2f440f09a0 msgr2=0x7f2f440f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:15:17.834 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:17.831+0000 7f2f4a20a700 1 -- 172.21.15.132:0/3855992881 shutdown_connections 2024-02-20T00:15:17.834 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:17.831+0000 7f2f4a20a700 1 -- 172.21.15.132:0/3855992881 wait complete. 2024-02-20T00:15:17.834 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:15:18.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:15:17 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:15:16.885999+0000 mgr.smithi084.bmqrnf (mgr.14162) 165 : cluster [DBG] pgmap v154: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:15:18.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:15:17 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:15:17.830682+0000 mon.smithi084 (mon.0) 212 : audit [DBG] from='client.? 172.21.15.132:0/3855992881' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:15:19.490 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:15:19.490 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:15:20.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:15:19 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:15:18.886341+0000 mgr.smithi084.bmqrnf (mgr.14162) 166 : cluster [DBG] pgmap v155: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:15:21.397 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:21.395+0000 7fe292e37700 1 -- 172.21.15.132:0/292158475 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe28c0f5410 msgr2=0x7fe28c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:15:21.397 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:21.395+0000 7fe292e37700 1 --2- 172.21.15.132:0/292158475 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe28c0f5410 0x7fe28c0f5830 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 rx=0x7fe280009fb0 tx=0x7fe280009b30).stop 2024-02-20T00:15:21.397 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:21.395+0000 7fe292e37700 1 -- 172.21.15.132:0/292158475 shutdown_connections 2024-02-20T00:15:21.397 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:21.395+0000 7fe292e37700 1 --2- 172.21.15.132:0/292158475 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe28c0f5410 0x7fe28c0f5830 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:15:21.397 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:21.395+0000 7fe292e37700 1 -- 172.21.15.132:0/292158475 >> 172.21.15.132:0/292158475 conn(0x7fe28c0f0b70 msgr2=0x7fe28c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:15:21.397 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:21.395+0000 7fe292e37700 1 -- 172.21.15.132:0/292158475 shutdown_connections 2024-02-20T00:15:21.397 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:21.395+0000 7fe292e37700 1 -- 172.21.15.132:0/292158475 wait complete. 2024-02-20T00:15:21.399 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:21.396+0000 7fe292e37700 1 Processor -- start 2024-02-20T00:15:21.399 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:21.396+0000 7fe292e37700 1 -- start start 2024-02-20T00:15:21.399 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:21.396+0000 7fe292e37700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe28c0f5410 0x7fe28c109520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:15:21.399 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:21.396+0000 7fe292e37700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe28c060240 con 0x7fe28c0f5410 2024-02-20T00:15:21.400 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:21.397+0000 7fe290bd3700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe28c0f5410 0x7fe28c109520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:15:21.400 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:21.397+0000 7fe290bd3700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe28c0f5410 0x7fe28c109520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:55570/0 (socket says 172.21.15.132:55570) 2024-02-20T00:15:21.400 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:21.397+0000 7fe290bd3700 1 -- 172.21.15.132:0/1742715996 learned_addr learned my addr 172.21.15.132:0/1742715996 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:15:21.400 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:21.397+0000 7fe290bd3700 1 -- 172.21.15.132:0/1742715996 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe280005040 con 0x7fe28c0f5410 2024-02-20T00:15:21.400 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:21.397+0000 7fe290bd3700 1 --2- 172.21.15.132:0/1742715996 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe28c0f5410 0x7fe28c109520 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7fe2800089f0 tx=0x7fe28000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:15:21.400 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:21.397+0000 7fe2897fa700 1 -- 172.21.15.132:0/1742715996 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe280009320 con 0x7fe28c0f5410 2024-02-20T00:15:21.400 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:21.397+0000 7fe2897fa700 1 -- 172.21.15.132:0/1742715996 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe280013380 con 0x7fe28c0f5410 2024-02-20T00:15:21.400 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:21.397+0000 7fe292e37700 1 -- 172.21.15.132:0/1742715996 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe28c060440 con 0x7fe28c0f5410 2024-02-20T00:15:21.400 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:21.397+0000 7fe2897fa700 1 -- 172.21.15.132:0/1742715996 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe28001ed80 con 0x7fe28c0f5410 2024-02-20T00:15:21.400 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:21.397+0000 7fe292e37700 1 -- 172.21.15.132:0/1742715996 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe28c109bd0 con 0x7fe28c0f5410 2024-02-20T00:15:21.400 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:21.398+0000 7fe2897fa700 1 -- 172.21.15.132:0/1742715996 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe280014420 con 0x7fe28c0f5410 2024-02-20T00:15:21.400 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:21.398+0000 7fe2897fa700 1 --2- 172.21.15.132:0/1742715996 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fe27c02ffb0 0x7fe27c032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:15:21.401 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:21.398+0000 7fe2897fa700 1 -- 172.21.15.132:0/1742715996 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe280040f40 con 0x7fe28c0f5410 2024-02-20T00:15:21.401 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:21.398+0000 7fe292e37700 1 -- 172.21.15.132:0/1742715996 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe28c045ad0 con 0x7fe28c0f5410 2024-02-20T00:15:21.401 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:21.398+0000 7fe28bfff700 1 --2- 172.21.15.132:0/1742715996 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fe27c02ffb0 0x7fe27c032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:15:21.401 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:21.399+0000 7fe28bfff700 1 --2- 172.21.15.132:0/1742715996 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fe27c02ffb0 0x7fe27c032470 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7fe2780096a0 tx=0x7fe278006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:15:21.404 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:21.403+0000 7fe2897fa700 1 -- 172.21.15.132:0/1742715996 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe28001c910 con 0x7fe28c0f5410 2024-02-20T00:15:21.609 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:21.608+0000 7fe292e37700 1 -- 172.21.15.132:0/1742715996 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe28c10a280 con 0x7fe28c0f5410 2024-02-20T00:15:21.610 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:21.608+0000 7fe2897fa700 1 -- 172.21.15.132:0/1742715996 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe28001a230 con 0x7fe28c0f5410 2024-02-20T00:15:21.610 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:15:21.610 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:15:21.612 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:21.610+0000 7fe292e37700 1 -- 172.21.15.132:0/1742715996 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fe27c02ffb0 msgr2=0x7fe27c032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:15:21.612 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:21.610+0000 7fe292e37700 1 --2- 172.21.15.132:0/1742715996 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fe27c02ffb0 0x7fe27c032470 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7fe2780096a0 tx=0x7fe278006b40).stop 2024-02-20T00:15:21.612 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:21.610+0000 7fe292e37700 1 -- 172.21.15.132:0/1742715996 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe28c0f5410 msgr2=0x7fe28c109520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:15:21.612 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:21.610+0000 7fe292e37700 1 --2- 172.21.15.132:0/1742715996 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe28c0f5410 0x7fe28c109520 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7fe2800089f0 tx=0x7fe28000b040).stop 2024-02-20T00:15:21.612 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:21.610+0000 7fe292e37700 1 -- 172.21.15.132:0/1742715996 shutdown_connections 2024-02-20T00:15:21.612 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:21.610+0000 7fe292e37700 1 --2- 172.21.15.132:0/1742715996 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fe27c02ffb0 0x7fe27c032470 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:15:21.612 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:21.610+0000 7fe292e37700 1 --2- 172.21.15.132:0/1742715996 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe28c0f5410 0x7fe28c109520 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:15:21.612 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:21.610+0000 7fe292e37700 1 -- 172.21.15.132:0/1742715996 >> 172.21.15.132:0/1742715996 conn(0x7fe28c0f0b70 msgr2=0x7fe28c0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:15:21.612 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:21.610+0000 7fe292e37700 1 -- 172.21.15.132:0/1742715996 shutdown_connections 2024-02-20T00:15:21.612 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:21.610+0000 7fe292e37700 1 -- 172.21.15.132:0/1742715996 wait complete. 2024-02-20T00:15:21.613 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:15:22.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:15:21 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:15:20.886580+0000 mgr.smithi084.bmqrnf (mgr.14162) 167 : cluster [DBG] pgmap v156: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:15:22.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:15:21 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:15:21.609156+0000 mon.smithi084 (mon.0) 213 : audit [DBG] from='client.? 172.21.15.132:0/1742715996' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:15:23.206 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:15:23.207 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:15:24.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:15:23 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:15:22.886783+0000 mgr.smithi084.bmqrnf (mgr.14162) 168 : cluster [DBG] pgmap v157: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:15:24.989 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:24.961+0000 7f699df70700 1 -- 172.21.15.132:0/1274406746 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f69980f51a0 msgr2=0x7f69980f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:15:24.990 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:24.961+0000 7f699df70700 1 --2- 172.21.15.132:0/1274406746 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f69980f51a0 0x7f69980f55c0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 rx=0x7f698c009fb0 tx=0x7f698c009b30).stop 2024-02-20T00:15:24.990 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:24.961+0000 7f699df70700 1 -- 172.21.15.132:0/1274406746 shutdown_connections 2024-02-20T00:15:24.990 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:24.961+0000 7f699df70700 1 --2- 172.21.15.132:0/1274406746 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f69980f51a0 0x7f69980f55c0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:15:24.990 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:24.961+0000 7f699df70700 1 -- 172.21.15.132:0/1274406746 >> 172.21.15.132:0/1274406746 conn(0x7f69980f09a0 msgr2=0x7f69980f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:15:24.990 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:24.961+0000 7f699df70700 1 -- 172.21.15.132:0/1274406746 shutdown_connections 2024-02-20T00:15:24.991 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:24.961+0000 7f699df70700 1 -- 172.21.15.132:0/1274406746 wait complete. 2024-02-20T00:15:24.991 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:24.962+0000 7f699df70700 1 Processor -- start 2024-02-20T00:15:24.991 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:24.962+0000 7f699df70700 1 -- start start 2024-02-20T00:15:24.991 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:24.963+0000 7f699df70700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f69980f51a0 0x7f69981054d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:15:24.991 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:24.963+0000 7f699df70700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69981059c0 con 0x7f69980f51a0 2024-02-20T00:15:24.991 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:24.963+0000 7f699cf6e700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f69980f51a0 0x7f69981054d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:15:24.992 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:24.963+0000 7f699cf6e700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f69980f51a0 0x7f69981054d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:55588/0 (socket says 172.21.15.132:55588) 2024-02-20T00:15:24.992 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:24.963+0000 7f699cf6e700 1 -- 172.21.15.132:0/206125545 learned_addr learned my addr 172.21.15.132:0/206125545 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:15:24.992 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:24.963+0000 7f699cf6e700 1 -- 172.21.15.132:0/206125545 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f698c005040 con 0x7f69980f51a0 2024-02-20T00:15:24.992 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:24.963+0000 7f699cf6e700 1 --2- 172.21.15.132:0/206125545 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f69980f51a0 0x7f69981054d0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7f698c0089f0 tx=0x7f698c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:15:24.992 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:24.963+0000 7f69957fa700 1 -- 172.21.15.132:0/206125545 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f698c009320 con 0x7f69980f51a0 2024-02-20T00:15:24.992 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:24.963+0000 7f69957fa700 1 -- 172.21.15.132:0/206125545 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f698c013380 con 0x7f69980f51a0 2024-02-20T00:15:24.993 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:24.963+0000 7f69957fa700 1 -- 172.21.15.132:0/206125545 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f698c014e20 con 0x7f69980f51a0 2024-02-20T00:15:24.993 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:24.963+0000 7f699df70700 1 -- 172.21.15.132:0/206125545 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6998105bc0 con 0x7f69980f51a0 2024-02-20T00:15:24.993 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:24.963+0000 7f699df70700 1 -- 172.21.15.132:0/206125545 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6998106050 con 0x7f69980f51a0 2024-02-20T00:15:24.993 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:24.964+0000 7f69957fa700 1 -- 172.21.15.132:0/206125545 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f698c014420 con 0x7f69980f51a0 2024-02-20T00:15:24.993 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:24.964+0000 7f699df70700 1 -- 172.21.15.132:0/206125545 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6998045ad0 con 0x7f69980f51a0 2024-02-20T00:15:24.993 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:24.964+0000 7f69957fa700 1 --2- 172.21.15.132:0/206125545 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f698802ff10 0x7f69880323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:15:24.994 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:24.964+0000 7f69957fa700 1 -- 172.21.15.132:0/206125545 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f698c040fc0 con 0x7f69980f51a0 2024-02-20T00:15:24.994 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:24.964+0000 7f6997fff700 1 --2- 172.21.15.132:0/206125545 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f698802ff10 0x7f69880323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:15:24.994 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:24.965+0000 7f6997fff700 1 --2- 172.21.15.132:0/206125545 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f698802ff10 0x7f69880323d0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f69840096a0 tx=0x7f6984006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:15:24.994 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:24.968+0000 7f69957fa700 1 -- 172.21.15.132:0/206125545 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f698c026bf0 con 0x7f69980f51a0 2024-02-20T00:15:25.174 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:25.173+0000 7f699df70700 1 -- 172.21.15.132:0/206125545 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6998106f00 con 0x7f69980f51a0 2024-02-20T00:15:25.175 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:25.174+0000 7f69957fa700 1 -- 172.21.15.132:0/206125545 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f698c01db70 con 0x7f69980f51a0 2024-02-20T00:15:25.175 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:15:25.175 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:15:25.176 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:25.175+0000 7f699df70700 1 -- 172.21.15.132:0/206125545 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f698802ff10 msgr2=0x7f69880323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:15:25.177 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:25.175+0000 7f699df70700 1 --2- 172.21.15.132:0/206125545 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f698802ff10 0x7f69880323d0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f69840096a0 tx=0x7f6984006b40).stop 2024-02-20T00:15:25.177 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:25.175+0000 7f699df70700 1 -- 172.21.15.132:0/206125545 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f69980f51a0 msgr2=0x7f69981054d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:15:25.177 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:25.175+0000 7f699df70700 1 --2- 172.21.15.132:0/206125545 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f69980f51a0 0x7f69981054d0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7f698c0089f0 tx=0x7f698c00b040).stop 2024-02-20T00:15:25.177 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:25.175+0000 7f699df70700 1 -- 172.21.15.132:0/206125545 shutdown_connections 2024-02-20T00:15:25.177 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:25.175+0000 7f699df70700 1 --2- 172.21.15.132:0/206125545 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f698802ff10 0x7f69880323d0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:15:25.177 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:25.175+0000 7f699df70700 1 --2- 172.21.15.132:0/206125545 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f69980f51a0 0x7f69981054d0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:15:25.177 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:25.175+0000 7f699df70700 1 -- 172.21.15.132:0/206125545 >> 172.21.15.132:0/206125545 conn(0x7f69980f09a0 msgr2=0x7f69980f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:15:25.177 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:25.175+0000 7f699df70700 1 -- 172.21.15.132:0/206125545 shutdown_connections 2024-02-20T00:15:25.177 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:25.175+0000 7f699df70700 1 -- 172.21.15.132:0/206125545 wait complete. 2024-02-20T00:15:25.177 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:15:26.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:15:25 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:15:24.887022+0000 mgr.smithi084.bmqrnf (mgr.14162) 169 : cluster [DBG] pgmap v158: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:15:26.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:15:25 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:15:25.174197+0000 mon.smithi084 (mon.0) 214 : audit [DBG] from='client.? 172.21.15.132:0/206125545' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:15:26.890 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:15:26.891 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:15:28.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:15:27 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:15:26.887223+0000 mgr.smithi084.bmqrnf (mgr.14162) 170 : cluster [DBG] pgmap v159: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:15:28.696 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:28.695+0000 7fb8929cb700 1 -- 172.21.15.132:0/1257502306 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb88c0f5410 msgr2=0x7fb88c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:15:28.697 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:28.695+0000 7fb8929cb700 1 --2- 172.21.15.132:0/1257502306 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb88c0f5410 0x7fb88c0f5830 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 rx=0x7fb87c009fb0 tx=0x7fb87c009b30).stop 2024-02-20T00:15:28.697 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:28.695+0000 7fb8929cb700 1 -- 172.21.15.132:0/1257502306 shutdown_connections 2024-02-20T00:15:28.697 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:28.695+0000 7fb8929cb700 1 --2- 172.21.15.132:0/1257502306 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb88c0f5410 0x7fb88c0f5830 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:15:28.697 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:28.695+0000 7fb8929cb700 1 -- 172.21.15.132:0/1257502306 >> 172.21.15.132:0/1257502306 conn(0x7fb88c0f0b70 msgr2=0x7fb88c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:15:28.697 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:28.695+0000 7fb8929cb700 1 -- 172.21.15.132:0/1257502306 shutdown_connections 2024-02-20T00:15:28.697 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:28.695+0000 7fb8929cb700 1 -- 172.21.15.132:0/1257502306 wait complete. 2024-02-20T00:15:28.697 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:28.695+0000 7fb8929cb700 1 Processor -- start 2024-02-20T00:15:28.699 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:28.696+0000 7fb8929cb700 1 -- start start 2024-02-20T00:15:28.700 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:28.696+0000 7fb8929cb700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb88c0f5410 0x7fb88c1067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:15:28.700 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:28.696+0000 7fb8929cb700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb88c106cd0 con 0x7fb88c0f5410 2024-02-20T00:15:28.700 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:28.696+0000 7fb88bfff700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb88c0f5410 0x7fb88c1067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:15:28.700 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:28.696+0000 7fb88bfff700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb88c0f5410 0x7fb88c1067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:37702/0 (socket says 172.21.15.132:37702) 2024-02-20T00:15:28.700 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:28.696+0000 7fb88bfff700 1 -- 172.21.15.132:0/3237659650 learned_addr learned my addr 172.21.15.132:0/3237659650 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:15:28.700 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:28.696+0000 7fb88bfff700 1 -- 172.21.15.132:0/3237659650 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb87c005040 con 0x7fb88c0f5410 2024-02-20T00:15:28.700 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:28.697+0000 7fb88bfff700 1 --2- 172.21.15.132:0/3237659650 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb88c0f5410 0x7fb88c1067e0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7fb87c005010 tx=0x7fb87c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:15:28.700 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:28.697+0000 7fb888ff9700 1 -- 172.21.15.132:0/3237659650 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb87c009320 con 0x7fb88c0f5410 2024-02-20T00:15:28.700 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:28.697+0000 7fb888ff9700 1 -- 172.21.15.132:0/3237659650 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb87c013380 con 0x7fb88c0f5410 2024-02-20T00:15:28.700 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:28.697+0000 7fb8929cb700 1 -- 172.21.15.132:0/3237659650 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb88c106ed0 con 0x7fb88c0f5410 2024-02-20T00:15:28.700 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:28.697+0000 7fb888ff9700 1 -- 172.21.15.132:0/3237659650 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb87c014e50 con 0x7fb88c0f5410 2024-02-20T00:15:28.700 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:28.697+0000 7fb8929cb700 1 -- 172.21.15.132:0/3237659650 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb88c107360 con 0x7fb88c0f5410 2024-02-20T00:15:28.701 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:28.697+0000 7fb888ff9700 1 -- 172.21.15.132:0/3237659650 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb87c0263a0 con 0x7fb88c0f5410 2024-02-20T00:15:28.701 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:28.698+0000 7fb888ff9700 1 --2- 172.21.15.132:0/3237659650 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fb87402ff10 0x7fb8740323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:15:28.701 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:28.698+0000 7fb8929cb700 1 -- 172.21.15.132:0/3237659650 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb88c045ad0 con 0x7fb88c0f5410 2024-02-20T00:15:28.701 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:28.698+0000 7fb888ff9700 1 -- 172.21.15.132:0/3237659650 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb87c0412b0 con 0x7fb88c0f5410 2024-02-20T00:15:28.701 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:28.698+0000 7fb88b7fe700 1 --2- 172.21.15.132:0/3237659650 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fb87402ff10 0x7fb8740323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:15:28.701 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:28.698+0000 7fb88b7fe700 1 --2- 172.21.15.132:0/3237659650 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fb87402ff10 0x7fb8740323d0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7fb8800096a0 tx=0x7fb880006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:15:28.704 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:28.703+0000 7fb888ff9700 1 -- 172.21.15.132:0/3237659650 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb87c012920 con 0x7fb88c0f5410 2024-02-20T00:15:28.912 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:28.911+0000 7fb8929cb700 1 -- 172.21.15.132:0/3237659650 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb88c108210 con 0x7fb88c0f5410 2024-02-20T00:15:28.912 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:28.911+0000 7fb888ff9700 1 -- 172.21.15.132:0/3237659650 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb87c00f340 con 0x7fb88c0f5410 2024-02-20T00:15:28.913 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:15:28.913 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:15:28.914 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:28.913+0000 7fb8929cb700 1 -- 172.21.15.132:0/3237659650 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fb87402ff10 msgr2=0x7fb8740323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:15:28.914 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:28.913+0000 7fb8929cb700 1 --2- 172.21.15.132:0/3237659650 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fb87402ff10 0x7fb8740323d0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7fb8800096a0 tx=0x7fb880006b40).stop 2024-02-20T00:15:28.914 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:28.913+0000 7fb8929cb700 1 -- 172.21.15.132:0/3237659650 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb88c0f5410 msgr2=0x7fb88c1067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:15:28.914 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:28.913+0000 7fb8929cb700 1 --2- 172.21.15.132:0/3237659650 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb88c0f5410 0x7fb88c1067e0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7fb87c005010 tx=0x7fb87c00b040).stop 2024-02-20T00:15:28.914 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:28.913+0000 7fb8929cb700 1 -- 172.21.15.132:0/3237659650 shutdown_connections 2024-02-20T00:15:28.914 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:28.913+0000 7fb8929cb700 1 --2- 172.21.15.132:0/3237659650 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fb87402ff10 0x7fb8740323d0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:15:28.914 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:28.913+0000 7fb8929cb700 1 --2- 172.21.15.132:0/3237659650 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb88c0f5410 0x7fb88c1067e0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:15:28.914 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:28.913+0000 7fb8929cb700 1 -- 172.21.15.132:0/3237659650 >> 172.21.15.132:0/3237659650 conn(0x7fb88c0f0b70 msgr2=0x7fb88c0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:15:28.914 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:28.913+0000 7fb8929cb700 1 -- 172.21.15.132:0/3237659650 shutdown_connections 2024-02-20T00:15:28.915 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:28.913+0000 7fb8929cb700 1 -- 172.21.15.132:0/3237659650 wait complete. 2024-02-20T00:15:28.915 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:15:29.228 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:15:28 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:15:28.911824+0000 mon.smithi084 (mon.0) 215 : audit [DBG] from='client.? 172.21.15.132:0/3237659650' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:15:30.108 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:15:29 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:15:28.887583+0000 mgr.smithi084.bmqrnf (mgr.14162) 171 : cluster [DBG] pgmap v160: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:15:30.627 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:15:30.628 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:15:32.130 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:15:31 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:15:30.887815+0000 mgr.smithi084.bmqrnf (mgr.14162) 172 : cluster [DBG] pgmap v161: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:15:32.624 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:32.621+0000 7f2de1b9f700 1 -- 172.21.15.132:0/230353370 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2ddc0f51a0 msgr2=0x7f2ddc0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:15:32.624 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:32.621+0000 7f2de1b9f700 1 --2- 172.21.15.132:0/230353370 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2ddc0f51a0 0x7f2ddc0f55c0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 rx=0x7f2dd0009fb0 tx=0x7f2dd0009b30).stop 2024-02-20T00:15:32.624 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:32.621+0000 7f2de1b9f700 1 -- 172.21.15.132:0/230353370 shutdown_connections 2024-02-20T00:15:32.624 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:32.621+0000 7f2de1b9f700 1 --2- 172.21.15.132:0/230353370 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2ddc0f51a0 0x7f2ddc0f55c0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:15:32.624 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:32.621+0000 7f2de1b9f700 1 -- 172.21.15.132:0/230353370 >> 172.21.15.132:0/230353370 conn(0x7f2ddc0f09a0 msgr2=0x7f2ddc0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:15:32.624 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:32.621+0000 7f2de1b9f700 1 -- 172.21.15.132:0/230353370 shutdown_connections 2024-02-20T00:15:32.625 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:32.621+0000 7f2de1b9f700 1 -- 172.21.15.132:0/230353370 wait complete. 2024-02-20T00:15:32.625 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:32.622+0000 7f2de1b9f700 1 Processor -- start 2024-02-20T00:15:32.625 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:32.622+0000 7f2de1b9f700 1 -- start start 2024-02-20T00:15:32.625 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:32.622+0000 7f2de1b9f700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2ddc0f51a0 0x7f2ddc1092f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:15:32.625 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:32.622+0000 7f2de1b9f700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ddc060220 con 0x7f2ddc0f51a0 2024-02-20T00:15:32.625 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:32.622+0000 7f2de0b9d700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2ddc0f51a0 0x7f2ddc1092f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:15:32.625 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:32.622+0000 7f2de0b9d700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2ddc0f51a0 0x7f2ddc1092f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:37708/0 (socket says 172.21.15.132:37708) 2024-02-20T00:15:32.625 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:32.622+0000 7f2de0b9d700 1 -- 172.21.15.132:0/1292691483 learned_addr learned my addr 172.21.15.132:0/1292691483 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:15:32.625 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:32.622+0000 7f2de0b9d700 1 -- 172.21.15.132:0/1292691483 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dd0005040 con 0x7f2ddc0f51a0 2024-02-20T00:15:32.625 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:32.623+0000 7f2de0b9d700 1 --2- 172.21.15.132:0/1292691483 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2ddc0f51a0 0x7f2ddc1092f0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7f2dd0009830 tx=0x7f2dd000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:15:32.625 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:32.623+0000 7f2dd97fa700 1 -- 172.21.15.132:0/1292691483 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2dd0009320 con 0x7f2ddc0f51a0 2024-02-20T00:15:32.626 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:32.623+0000 7f2de1b9f700 1 -- 172.21.15.132:0/1292691483 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ddc060420 con 0x7f2ddc0f51a0 2024-02-20T00:15:32.626 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:32.623+0000 7f2dd97fa700 1 -- 172.21.15.132:0/1292691483 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2dd0013380 con 0x7f2ddc0f51a0 2024-02-20T00:15:32.626 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:32.623+0000 7f2dd97fa700 1 -- 172.21.15.132:0/1292691483 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2dd0014420 con 0x7f2ddc0f51a0 2024-02-20T00:15:32.626 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:32.623+0000 7f2de1b9f700 1 -- 172.21.15.132:0/1292691483 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ddc1099a0 con 0x7f2ddc0f51a0 2024-02-20T00:15:32.628 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:32.623+0000 7f2dd97fa700 1 -- 172.21.15.132:0/1292691483 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2dd001e420 con 0x7f2ddc0f51a0 2024-02-20T00:15:32.628 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:32.624+0000 7f2dd97fa700 1 --2- 172.21.15.132:0/1292691483 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f2dcc02ff60 0x7f2dcc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:15:32.628 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:32.624+0000 7f2dd97fa700 1 -- 172.21.15.132:0/1292691483 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2dd0040ee0 con 0x7f2ddc0f51a0 2024-02-20T00:15:32.628 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:32.624+0000 7f2de1b9f700 1 -- 172.21.15.132:0/1292691483 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ddc059100 con 0x7f2ddc0f51a0 2024-02-20T00:15:32.628 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:32.624+0000 7f2ddbfff700 1 --2- 172.21.15.132:0/1292691483 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f2dcc02ff60 0x7f2dcc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:15:32.628 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:32.624+0000 7f2ddbfff700 1 --2- 172.21.15.132:0/1292691483 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f2dcc02ff60 0x7f2dcc032420 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f2dc80096a0 tx=0x7f2dc8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:15:32.629 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:32.628+0000 7f2dd97fa700 1 -- 172.21.15.132:0/1292691483 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2dd001c970 con 0x7f2ddc0f51a0 2024-02-20T00:15:32.835 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:32.834+0000 7f2de1b9f700 1 -- 172.21.15.132:0/1292691483 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2ddc045ad0 con 0x7f2ddc0f51a0 2024-02-20T00:15:32.836 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:32.834+0000 7f2dd97fa700 1 -- 172.21.15.132:0/1292691483 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2dd0004490 con 0x7f2ddc0f51a0 2024-02-20T00:15:32.836 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:15:32.836 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:15:32.837 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:32.836+0000 7f2de1b9f700 1 -- 172.21.15.132:0/1292691483 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f2dcc02ff60 msgr2=0x7f2dcc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:15:32.838 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:32.836+0000 7f2de1b9f700 1 --2- 172.21.15.132:0/1292691483 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f2dcc02ff60 0x7f2dcc032420 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f2dc80096a0 tx=0x7f2dc8006b40).stop 2024-02-20T00:15:32.838 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:32.836+0000 7f2de1b9f700 1 -- 172.21.15.132:0/1292691483 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2ddc0f51a0 msgr2=0x7f2ddc1092f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:15:32.838 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:32.836+0000 7f2de1b9f700 1 --2- 172.21.15.132:0/1292691483 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2ddc0f51a0 0x7f2ddc1092f0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7f2dd0009830 tx=0x7f2dd000b040).stop 2024-02-20T00:15:32.838 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:32.836+0000 7f2de1b9f700 1 -- 172.21.15.132:0/1292691483 shutdown_connections 2024-02-20T00:15:32.838 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:32.836+0000 7f2de1b9f700 1 --2- 172.21.15.132:0/1292691483 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f2dcc02ff60 0x7f2dcc032420 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:15:32.838 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:32.836+0000 7f2de1b9f700 1 --2- 172.21.15.132:0/1292691483 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2ddc0f51a0 0x7f2ddc1092f0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:15:32.838 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:32.836+0000 7f2de1b9f700 1 -- 172.21.15.132:0/1292691483 >> 172.21.15.132:0/1292691483 conn(0x7f2ddc0f09a0 msgr2=0x7f2ddc0f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:15:32.838 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:32.836+0000 7f2de1b9f700 1 -- 172.21.15.132:0/1292691483 shutdown_connections 2024-02-20T00:15:32.838 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:32.836+0000 7f2de1b9f700 1 -- 172.21.15.132:0/1292691483 wait complete. 2024-02-20T00:15:32.838 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:15:33.608 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:15:33 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:15:32.185982+0000 mon.smithi084 (mon.0) 216 : audit [INF] from='mgr.14162 172.21.15.84:0/2152933216' entity='mgr.smithi084.bmqrnf' 2024-02-20T00:15:33.608 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:15:33 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cephadm 2024-02-20T00:15:32.187281+0000 mgr.smithi084.bmqrnf (mgr.14162) 173 : cephadm [INF] Deploying daemon node-exporter.smithi084 on smithi084 2024-02-20T00:15:33.609 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:15:33 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:15:32.835083+0000 mon.smithi084 (mon.0) 217 : audit [DBG] from='client.? 172.21.15.132:0/1292691483' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:15:34.449 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:15:34.449 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:15:34.607 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:15:34 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:15:32.888151+0000 mgr.smithi084.bmqrnf (mgr.14162) 174 : cluster [DBG] pgmap v162: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:15:36.277 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:36.224+0000 7fbf67152700 1 -- 172.21.15.132:0/1397646928 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbf600f5410 msgr2=0x7fbf600f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:15:36.277 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:36.224+0000 7fbf67152700 1 --2- 172.21.15.132:0/1397646928 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbf600f5410 0x7fbf600f5830 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 rx=0x7fbf54009fb0 tx=0x7fbf54009b30).stop 2024-02-20T00:15:36.277 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:36.224+0000 7fbf67152700 1 -- 172.21.15.132:0/1397646928 shutdown_connections 2024-02-20T00:15:36.277 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:36.224+0000 7fbf67152700 1 --2- 172.21.15.132:0/1397646928 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbf600f5410 0x7fbf600f5830 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:15:36.277 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:36.224+0000 7fbf67152700 1 -- 172.21.15.132:0/1397646928 >> 172.21.15.132:0/1397646928 conn(0x7fbf600f0b70 msgr2=0x7fbf600f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:15:36.277 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:36.224+0000 7fbf67152700 1 -- 172.21.15.132:0/1397646928 shutdown_connections 2024-02-20T00:15:36.277 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:36.224+0000 7fbf67152700 1 -- 172.21.15.132:0/1397646928 wait complete. 2024-02-20T00:15:36.278 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:36.224+0000 7fbf67152700 1 Processor -- start 2024-02-20T00:15:36.278 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:36.224+0000 7fbf67152700 1 -- start start 2024-02-20T00:15:36.278 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:36.224+0000 7fbf67152700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbf600f5410 0x7fbf601094e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:15:36.278 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:36.224+0000 7fbf67152700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf60060240 con 0x7fbf600f5410 2024-02-20T00:15:36.278 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:36.225+0000 7fbf64eee700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbf600f5410 0x7fbf601094e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:15:36.278 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:36.225+0000 7fbf64eee700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbf600f5410 0x7fbf601094e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:37722/0 (socket says 172.21.15.132:37722) 2024-02-20T00:15:36.278 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:36.225+0000 7fbf64eee700 1 -- 172.21.15.132:0/2318256802 learned_addr learned my addr 172.21.15.132:0/2318256802 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:15:36.278 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:36.225+0000 7fbf64eee700 1 -- 172.21.15.132:0/2318256802 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf54005040 con 0x7fbf600f5410 2024-02-20T00:15:36.278 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:36.225+0000 7fbf64eee700 1 --2- 172.21.15.132:0/2318256802 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbf600f5410 0x7fbf601094e0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7fbf54005010 tx=0x7fbf5400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:15:36.278 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:36.225+0000 7fbf5d7fa700 1 -- 172.21.15.132:0/2318256802 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbf54009320 con 0x7fbf600f5410 2024-02-20T00:15:36.278 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:36.225+0000 7fbf67152700 1 -- 172.21.15.132:0/2318256802 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf60060440 con 0x7fbf600f5410 2024-02-20T00:15:36.278 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:36.225+0000 7fbf5d7fa700 1 -- 172.21.15.132:0/2318256802 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbf54013380 con 0x7fbf600f5410 2024-02-20T00:15:36.278 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:36.225+0000 7fbf5d7fa700 1 -- 172.21.15.132:0/2318256802 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbf54014e50 con 0x7fbf600f5410 2024-02-20T00:15:36.278 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:36.225+0000 7fbf67152700 1 -- 172.21.15.132:0/2318256802 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf60109b90 con 0x7fbf600f5410 2024-02-20T00:15:36.279 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:36.226+0000 7fbf5d7fa700 1 -- 172.21.15.132:0/2318256802 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbf54014420 con 0x7fbf600f5410 2024-02-20T00:15:36.279 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:36.226+0000 7fbf67152700 1 -- 172.21.15.132:0/2318256802 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf60045ad0 con 0x7fbf600f5410 2024-02-20T00:15:36.279 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:36.226+0000 7fbf5d7fa700 1 --2- 172.21.15.132:0/2318256802 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fbf5002ff10 0x7fbf500323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:15:36.279 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:36.226+0000 7fbf5d7fa700 1 -- 172.21.15.132:0/2318256802 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbf54040f40 con 0x7fbf600f5410 2024-02-20T00:15:36.279 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:36.227+0000 7fbf5ffff700 1 --2- 172.21.15.132:0/2318256802 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fbf5002ff10 0x7fbf500323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:15:36.279 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:36.227+0000 7fbf5ffff700 1 --2- 172.21.15.132:0/2318256802 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fbf5002ff10 0x7fbf500323d0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7fbf4c0096a0 tx=0x7fbf4c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:15:36.279 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:36.231+0000 7fbf5d7fa700 1 -- 172.21.15.132:0/2318256802 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbf540095f0 con 0x7fbf600f5410 2024-02-20T00:15:36.437 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:36.436+0000 7fbf67152700 1 -- 172.21.15.132:0/2318256802 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbf6010a3b0 con 0x7fbf600f5410 2024-02-20T00:15:36.438 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:36.437+0000 7fbf5d7fa700 1 -- 172.21.15.132:0/2318256802 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fbf5401db70 con 0x7fbf600f5410 2024-02-20T00:15:36.438 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:15:36.438 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:15:36.440 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:36.438+0000 7fbf67152700 1 -- 172.21.15.132:0/2318256802 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fbf5002ff10 msgr2=0x7fbf500323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:15:36.440 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:36.438+0000 7fbf67152700 1 --2- 172.21.15.132:0/2318256802 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fbf5002ff10 0x7fbf500323d0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7fbf4c0096a0 tx=0x7fbf4c006b40).stop 2024-02-20T00:15:36.440 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:36.438+0000 7fbf67152700 1 -- 172.21.15.132:0/2318256802 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbf600f5410 msgr2=0x7fbf601094e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:15:36.440 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:36.438+0000 7fbf67152700 1 --2- 172.21.15.132:0/2318256802 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbf600f5410 0x7fbf601094e0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7fbf54005010 tx=0x7fbf5400b040).stop 2024-02-20T00:15:36.440 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:36.438+0000 7fbf67152700 1 -- 172.21.15.132:0/2318256802 shutdown_connections 2024-02-20T00:15:36.440 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:36.438+0000 7fbf67152700 1 --2- 172.21.15.132:0/2318256802 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fbf5002ff10 0x7fbf500323d0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:15:36.440 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:36.438+0000 7fbf67152700 1 --2- 172.21.15.132:0/2318256802 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbf600f5410 0x7fbf601094e0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:15:36.440 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:36.438+0000 7fbf67152700 1 -- 172.21.15.132:0/2318256802 >> 172.21.15.132:0/2318256802 conn(0x7fbf600f0b70 msgr2=0x7fbf600f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:15:36.440 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:36.439+0000 7fbf67152700 1 -- 172.21.15.132:0/2318256802 shutdown_connections 2024-02-20T00:15:36.440 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:36.439+0000 7fbf67152700 1 -- 172.21.15.132:0/2318256802 wait complete. 2024-02-20T00:15:36.441 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:15:36.607 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:15:36 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:15:34.888406+0000 mgr.smithi084.bmqrnf (mgr.14162) 175 : cluster [DBG] pgmap v163: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:15:37.607 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:15:37 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:15:36.437556+0000 mon.smithi084 (mon.0) 218 : audit [DBG] from='client.? 172.21.15.132:0/2318256802' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:15:38.091 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:15:38.091 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:15:38.607 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:15:38 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:15:36.888692+0000 mgr.smithi084.bmqrnf (mgr.14162) 176 : cluster [DBG] pgmap v164: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:15:40.039 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:40.035+0000 7f181492d700 1 -- 172.21.15.132:0/2231051184 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f18100f62a0 msgr2=0x7f18100f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:15:40.039 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:40.035+0000 7f181492d700 1 --2- 172.21.15.132:0/2231051184 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f18100f62a0 0x7f18100f66c0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 rx=0x7f1800009fb0 tx=0x7f1800009b30).stop 2024-02-20T00:15:40.039 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:40.035+0000 7f181492d700 1 -- 172.21.15.132:0/2231051184 shutdown_connections 2024-02-20T00:15:40.039 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:40.035+0000 7f181492d700 1 --2- 172.21.15.132:0/2231051184 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f18100f62a0 0x7f18100f66c0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:15:40.039 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:40.035+0000 7f181492d700 1 -- 172.21.15.132:0/2231051184 >> 172.21.15.132:0/2231051184 conn(0x7f18100f1aa0 msgr2=0x7f18100f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:15:40.039 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:40.035+0000 7f181492d700 1 -- 172.21.15.132:0/2231051184 shutdown_connections 2024-02-20T00:15:40.040 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:40.035+0000 7f181492d700 1 -- 172.21.15.132:0/2231051184 wait complete. 2024-02-20T00:15:40.040 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:40.035+0000 7f181492d700 1 Processor -- start 2024-02-20T00:15:40.040 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:40.036+0000 7f181492d700 1 -- start start 2024-02-20T00:15:40.040 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:40.036+0000 7f181492d700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f18100f62a0 0x7f181006b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:15:40.040 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:40.036+0000 7f181492d700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1810069e20 con 0x7f18100f62a0 2024-02-20T00:15:40.040 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:40.036+0000 7f180f7fe700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f18100f62a0 0x7f181006b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:15:40.040 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:40.036+0000 7f180f7fe700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f18100f62a0 0x7f181006b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:32798/0 (socket says 172.21.15.132:32798) 2024-02-20T00:15:40.040 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:40.036+0000 7f180f7fe700 1 -- 172.21.15.132:0/924906406 learned_addr learned my addr 172.21.15.132:0/924906406 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:15:40.040 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:40.036+0000 7f180f7fe700 1 -- 172.21.15.132:0/924906406 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1800005040 con 0x7f18100f62a0 2024-02-20T00:15:40.040 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:40.036+0000 7f180f7fe700 1 --2- 172.21.15.132:0/924906406 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f18100f62a0 0x7f181006b9a0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7f18000089f0 tx=0x7f180000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:15:40.040 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:40.037+0000 7f17f7fff700 1 -- 172.21.15.132:0/924906406 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1800009320 con 0x7f18100f62a0 2024-02-20T00:15:40.041 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:40.037+0000 7f181492d700 1 -- 172.21.15.132:0/924906406 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1810069fc0 con 0x7f18100f62a0 2024-02-20T00:15:40.041 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:40.037+0000 7f17f7fff700 1 -- 172.21.15.132:0/924906406 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1800013380 con 0x7f18100f62a0 2024-02-20T00:15:40.041 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:40.037+0000 7f17f7fff700 1 -- 172.21.15.132:0/924906406 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1800014e20 con 0x7f18100f62a0 2024-02-20T00:15:40.041 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:40.037+0000 7f181492d700 1 -- 172.21.15.132:0/924906406 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f181006a450 con 0x7f18100f62a0 2024-02-20T00:15:40.042 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:40.037+0000 7f17f7fff700 1 -- 172.21.15.132:0/924906406 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f18000095f0 con 0x7f18100f62a0 2024-02-20T00:15:40.042 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:40.037+0000 7f181492d700 1 -- 172.21.15.132:0/924906406 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1810059100 con 0x7f18100f62a0 2024-02-20T00:15:40.042 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:40.038+0000 7f17f7fff700 1 --2- 172.21.15.132:0/924906406 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f17f802ff10 0x7f17f80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:15:40.042 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:40.038+0000 7f17f7fff700 1 -- 172.21.15.132:0/924906406 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1800040f50 con 0x7f18100f62a0 2024-02-20T00:15:40.042 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:40.038+0000 7f180effd700 1 --2- 172.21.15.132:0/924906406 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f17f802ff10 0x7f17f80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:15:40.042 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:40.038+0000 7f180effd700 1 --2- 172.21.15.132:0/924906406 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f17f802ff10 0x7f17f80323d0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f18040096a0 tx=0x7f1804006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:15:40.044 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:40.042+0000 7f17f7fff700 1 -- 172.21.15.132:0/924906406 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1800014420 con 0x7f18100f62a0 2024-02-20T00:15:40.247 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:40.244+0000 7f181492d700 1 -- 172.21.15.132:0/924906406 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f18100fbe30 con 0x7f18100f62a0 2024-02-20T00:15:40.247 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:40.245+0000 7f17f7fff700 1 -- 172.21.15.132:0/924906406 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f180001d980 con 0x7f18100f62a0 2024-02-20T00:15:40.247 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:15:40.247 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:15:40.249 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:40.246+0000 7f181492d700 1 -- 172.21.15.132:0/924906406 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f17f802ff10 msgr2=0x7f17f80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:15:40.249 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:40.246+0000 7f181492d700 1 --2- 172.21.15.132:0/924906406 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f17f802ff10 0x7f17f80323d0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f18040096a0 tx=0x7f1804006b40).stop 2024-02-20T00:15:40.249 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:40.246+0000 7f181492d700 1 -- 172.21.15.132:0/924906406 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f18100f62a0 msgr2=0x7f181006b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:15:40.249 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:40.246+0000 7f181492d700 1 --2- 172.21.15.132:0/924906406 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f18100f62a0 0x7f181006b9a0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7f18000089f0 tx=0x7f180000b040).stop 2024-02-20T00:15:40.249 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:40.246+0000 7f181492d700 1 -- 172.21.15.132:0/924906406 shutdown_connections 2024-02-20T00:15:40.249 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:40.246+0000 7f181492d700 1 --2- 172.21.15.132:0/924906406 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f17f802ff10 0x7f17f80323d0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:15:40.249 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:40.246+0000 7f181492d700 1 --2- 172.21.15.132:0/924906406 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f18100f62a0 0x7f181006b9a0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:15:40.249 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:40.246+0000 7f181492d700 1 -- 172.21.15.132:0/924906406 >> 172.21.15.132:0/924906406 conn(0x7f18100f1aa0 msgr2=0x7f18100f2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:15:40.249 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:40.247+0000 7f181492d700 1 -- 172.21.15.132:0/924906406 shutdown_connections 2024-02-20T00:15:40.250 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:40.247+0000 7f181492d700 1 -- 172.21.15.132:0/924906406 wait complete. 2024-02-20T00:15:40.250 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:15:40.608 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:15:40 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:15:38.888927+0000 mgr.smithi084.bmqrnf (mgr.14162) 177 : cluster [DBG] pgmap v165: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:15:41.607 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:15:41 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:15:40.246424+0000 mon.smithi084 (mon.0) 219 : audit [DBG] from='client.? 172.21.15.132:0/924906406' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:15:41.915 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:15:41.915 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:15:42.607 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:15:42 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:15:40.889270+0000 mgr.smithi084.bmqrnf (mgr.14162) 178 : cluster [DBG] pgmap v166: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:15:43.735 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:43.731+0000 7faa6ab9d700 1 -- 172.21.15.132:0/4085703936 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7faa640f62a0 msgr2=0x7faa640f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:15:43.736 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:43.731+0000 7faa6ab9d700 1 --2- 172.21.15.132:0/4085703936 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7faa640f62a0 0x7faa640f66c0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 rx=0x7faa54009fb0 tx=0x7faa54009b30).stop 2024-02-20T00:15:43.736 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:43.731+0000 7faa6ab9d700 1 -- 172.21.15.132:0/4085703936 shutdown_connections 2024-02-20T00:15:43.736 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:43.731+0000 7faa6ab9d700 1 --2- 172.21.15.132:0/4085703936 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7faa640f62a0 0x7faa640f66c0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:15:43.736 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:43.731+0000 7faa6ab9d700 1 -- 172.21.15.132:0/4085703936 >> 172.21.15.132:0/4085703936 conn(0x7faa640f1aa0 msgr2=0x7faa640f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:15:43.736 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:43.732+0000 7faa6ab9d700 1 -- 172.21.15.132:0/4085703936 shutdown_connections 2024-02-20T00:15:43.736 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:43.732+0000 7faa6ab9d700 1 -- 172.21.15.132:0/4085703936 wait complete. 2024-02-20T00:15:43.736 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:43.732+0000 7faa6ab9d700 1 Processor -- start 2024-02-20T00:15:43.736 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:43.732+0000 7faa6ab9d700 1 -- start start 2024-02-20T00:15:43.736 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:43.732+0000 7faa6ab9d700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7faa640f62a0 0x7faa641065d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:15:43.736 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:43.732+0000 7faa6ab9d700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa64106ac0 con 0x7faa640f62a0 2024-02-20T00:15:43.736 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:43.732+0000 7faa69b9b700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7faa640f62a0 0x7faa641065d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:15:43.737 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:43.732+0000 7faa69b9b700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7faa640f62a0 0x7faa641065d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:32806/0 (socket says 172.21.15.132:32806) 2024-02-20T00:15:43.737 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:43.732+0000 7faa69b9b700 1 -- 172.21.15.132:0/808142991 learned_addr learned my addr 172.21.15.132:0/808142991 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:15:43.737 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:43.733+0000 7faa69b9b700 1 -- 172.21.15.132:0/808142991 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa54005040 con 0x7faa640f62a0 2024-02-20T00:15:43.737 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:43.733+0000 7faa69b9b700 1 --2- 172.21.15.132:0/808142991 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7faa640f62a0 0x7faa641065d0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7faa540089f0 tx=0x7faa5400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:15:43.737 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:43.733+0000 7faa5a7fc700 1 -- 172.21.15.132:0/808142991 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faa54009320 con 0x7faa640f62a0 2024-02-20T00:15:43.737 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:43.733+0000 7faa6ab9d700 1 -- 172.21.15.132:0/808142991 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa64106cc0 con 0x7faa640f62a0 2024-02-20T00:15:43.737 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:43.733+0000 7faa5a7fc700 1 -- 172.21.15.132:0/808142991 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7faa54013380 con 0x7faa640f62a0 2024-02-20T00:15:43.737 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:43.733+0000 7faa5a7fc700 1 -- 172.21.15.132:0/808142991 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faa54014e20 con 0x7faa640f62a0 2024-02-20T00:15:43.737 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:43.733+0000 7faa6ab9d700 1 -- 172.21.15.132:0/808142991 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa64107150 con 0x7faa640f62a0 2024-02-20T00:15:43.738 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:43.734+0000 7faa5a7fc700 1 -- 172.21.15.132:0/808142991 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7faa540095f0 con 0x7faa640f62a0 2024-02-20T00:15:43.738 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:43.734+0000 7faa6ab9d700 1 -- 172.21.15.132:0/808142991 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa64045ad0 con 0x7faa640f62a0 2024-02-20T00:15:43.738 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:43.734+0000 7faa5a7fc700 1 --2- 172.21.15.132:0/808142991 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7faa5002ff10 0x7faa500323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:15:43.738 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:43.734+0000 7faa5a7fc700 1 -- 172.21.15.132:0/808142991 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7faa54040e90 con 0x7faa640f62a0 2024-02-20T00:15:43.738 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:43.734+0000 7faa6939a700 1 --2- 172.21.15.132:0/808142991 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7faa5002ff10 0x7faa500323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:15:43.739 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:43.735+0000 7faa6939a700 1 --2- 172.21.15.132:0/808142991 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7faa5002ff10 0x7faa500323d0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7faa600096a0 tx=0x7faa60006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:15:43.740 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:43.738+0000 7faa5a7fc700 1 -- 172.21.15.132:0/808142991 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7faa54014420 con 0x7faa640f62a0 2024-02-20T00:15:43.943 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:43.941+0000 7faa6ab9d700 1 -- 172.21.15.132:0/808142991 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7faa64108000 con 0x7faa640f62a0 2024-02-20T00:15:43.944 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:43.941+0000 7faa5a7fc700 1 -- 172.21.15.132:0/808142991 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7faa5401d980 con 0x7faa640f62a0 2024-02-20T00:15:43.944 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:15:43.944 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:15:43.946 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:43.943+0000 7faa6ab9d700 1 -- 172.21.15.132:0/808142991 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7faa5002ff10 msgr2=0x7faa500323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:15:43.946 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:43.943+0000 7faa6ab9d700 1 --2- 172.21.15.132:0/808142991 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7faa5002ff10 0x7faa500323d0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7faa600096a0 tx=0x7faa60006b40).stop 2024-02-20T00:15:43.946 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:43.943+0000 7faa6ab9d700 1 -- 172.21.15.132:0/808142991 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7faa640f62a0 msgr2=0x7faa641065d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:15:43.946 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:43.943+0000 7faa6ab9d700 1 --2- 172.21.15.132:0/808142991 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7faa640f62a0 0x7faa641065d0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7faa540089f0 tx=0x7faa5400b040).stop 2024-02-20T00:15:43.946 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:43.943+0000 7faa6ab9d700 1 -- 172.21.15.132:0/808142991 shutdown_connections 2024-02-20T00:15:43.946 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:43.943+0000 7faa6ab9d700 1 --2- 172.21.15.132:0/808142991 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7faa5002ff10 0x7faa500323d0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:15:43.946 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:43.943+0000 7faa6ab9d700 1 --2- 172.21.15.132:0/808142991 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7faa640f62a0 0x7faa641065d0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:15:43.946 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:43.943+0000 7faa6ab9d700 1 -- 172.21.15.132:0/808142991 >> 172.21.15.132:0/808142991 conn(0x7faa640f1aa0 msgr2=0x7faa640f2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:15:43.946 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:43.943+0000 7faa6ab9d700 1 -- 172.21.15.132:0/808142991 shutdown_connections 2024-02-20T00:15:43.946 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:43.943+0000 7faa6ab9d700 1 -- 172.21.15.132:0/808142991 wait complete. 2024-02-20T00:15:43.946 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:15:44.607 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:15:44 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:15:42.889629+0000 mgr.smithi084.bmqrnf (mgr.14162) 179 : cluster [DBG] pgmap v167: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:15:44.608 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:15:44 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:15:43.943145+0000 mon.smithi084 (mon.0) 220 : audit [DBG] from='client.? 172.21.15.132:0/808142991' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:15:45.702 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:15:45.703 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:15:46.608 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:15:46 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:15:44.889904+0000 mgr.smithi084.bmqrnf (mgr.14162) 180 : cluster [DBG] pgmap v168: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:15:47.618 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:47.616+0000 7f4ccffff700 1 -- 172.21.15.132:0/697218839 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4cd00f51a0 msgr2=0x7f4cd00f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:15:47.619 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:47.616+0000 7f4ccffff700 1 --2- 172.21.15.132:0/697218839 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4cd00f51a0 0x7f4cd00f55c0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 rx=0x7f4cc0009fb0 tx=0x7f4cc0009b30).stop 2024-02-20T00:15:47.619 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:47.616+0000 7f4ccffff700 1 -- 172.21.15.132:0/697218839 shutdown_connections 2024-02-20T00:15:47.619 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:47.616+0000 7f4ccffff700 1 --2- 172.21.15.132:0/697218839 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4cd00f51a0 0x7f4cd00f55c0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:15:47.619 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:47.616+0000 7f4ccffff700 1 -- 172.21.15.132:0/697218839 >> 172.21.15.132:0/697218839 conn(0x7f4cd00f09a0 msgr2=0x7f4cd00f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:15:47.619 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:47.616+0000 7f4ccffff700 1 -- 172.21.15.132:0/697218839 shutdown_connections 2024-02-20T00:15:47.619 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:47.616+0000 7f4ccffff700 1 -- 172.21.15.132:0/697218839 wait complete. 2024-02-20T00:15:47.620 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:47.617+0000 7f4ccffff700 1 Processor -- start 2024-02-20T00:15:47.620 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:47.617+0000 7f4ccffff700 1 -- start start 2024-02-20T00:15:47.620 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:47.617+0000 7f4ccffff700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4cd00f51a0 0x7f4cd00f2f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:15:47.621 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:47.617+0000 7f4ccffff700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cd00f13a0 con 0x7f4cd00f51a0 2024-02-20T00:15:47.621 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:47.617+0000 7f4cceffd700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4cd00f51a0 0x7f4cd00f2f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:15:47.621 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:47.617+0000 7f4cceffd700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4cd00f51a0 0x7f4cd00f2f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:33022/0 (socket says 172.21.15.132:33022) 2024-02-20T00:15:47.621 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:47.617+0000 7f4cceffd700 1 -- 172.21.15.132:0/2038842604 learned_addr learned my addr 172.21.15.132:0/2038842604 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:15:47.621 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:47.618+0000 7f4cceffd700 1 -- 172.21.15.132:0/2038842604 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cc0005040 con 0x7f4cd00f51a0 2024-02-20T00:15:47.621 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:47.618+0000 7f4cceffd700 1 --2- 172.21.15.132:0/2038842604 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4cd00f51a0 0x7f4cd00f2f20 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7f4cc0009830 tx=0x7f4cc000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:15:47.621 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:47.618+0000 7f4cb77fe700 1 -- 172.21.15.132:0/2038842604 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4cc0009320 con 0x7f4cd00f51a0 2024-02-20T00:15:47.621 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:47.618+0000 7f4ccffff700 1 -- 172.21.15.132:0/2038842604 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cd00f1540 con 0x7f4cd00f51a0 2024-02-20T00:15:47.621 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:47.618+0000 7f4cb77fe700 1 -- 172.21.15.132:0/2038842604 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4cc0013380 con 0x7f4cd00f51a0 2024-02-20T00:15:47.621 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:47.618+0000 7f4cb77fe700 1 -- 172.21.15.132:0/2038842604 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4cc0014420 con 0x7f4cd00f51a0 2024-02-20T00:15:47.621 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:47.618+0000 7f4ccffff700 1 -- 172.21.15.132:0/2038842604 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cd00f19d0 con 0x7f4cd00f51a0 2024-02-20T00:15:47.622 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:47.619+0000 7f4cb77fe700 1 -- 172.21.15.132:0/2038842604 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4cc001e420 con 0x7f4cd00f51a0 2024-02-20T00:15:47.622 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:47.619+0000 7f4ccffff700 1 -- 172.21.15.132:0/2038842604 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4cd0059100 con 0x7f4cd00f51a0 2024-02-20T00:15:47.622 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:47.619+0000 7f4cb77fe700 1 --2- 172.21.15.132:0/2038842604 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f4cb802ff60 0x7f4cb8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:15:47.622 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:47.619+0000 7f4cb77fe700 1 -- 172.21.15.132:0/2038842604 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4cc0040e30 con 0x7f4cd00f51a0 2024-02-20T00:15:47.622 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:47.619+0000 7f4cce7fc700 1 --2- 172.21.15.132:0/2038842604 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f4cb802ff60 0x7f4cb8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:15:47.623 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:47.619+0000 7f4cce7fc700 1 --2- 172.21.15.132:0/2038842604 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f4cb802ff60 0x7f4cb8032420 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f4cc40096a0 tx=0x7f4cc4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:15:47.625 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:47.623+0000 7f4cb77fe700 1 -- 172.21.15.132:0/2038842604 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4cc001cb10 con 0x7f4cd00f51a0 2024-02-20T00:15:47.829 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:47.827+0000 7f4ccffff700 1 -- 172.21.15.132:0/2038842604 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4cd0045ad0 con 0x7f4cd00f51a0 2024-02-20T00:15:47.830 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:47.828+0000 7f4cb77fe700 1 -- 172.21.15.132:0/2038842604 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4cc0004490 con 0x7f4cd00f51a0 2024-02-20T00:15:47.830 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:15:47.830 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:15:47.831 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:47.829+0000 7f4ccffff700 1 -- 172.21.15.132:0/2038842604 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f4cb802ff60 msgr2=0x7f4cb8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:15:47.832 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:47.829+0000 7f4ccffff700 1 --2- 172.21.15.132:0/2038842604 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f4cb802ff60 0x7f4cb8032420 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f4cc40096a0 tx=0x7f4cc4006b40).stop 2024-02-20T00:15:47.832 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:47.829+0000 7f4ccffff700 1 -- 172.21.15.132:0/2038842604 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4cd00f51a0 msgr2=0x7f4cd00f2f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:15:47.832 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:47.829+0000 7f4ccffff700 1 --2- 172.21.15.132:0/2038842604 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4cd00f51a0 0x7f4cd00f2f20 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7f4cc0009830 tx=0x7f4cc000b040).stop 2024-02-20T00:15:47.832 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:47.829+0000 7f4ccffff700 1 -- 172.21.15.132:0/2038842604 shutdown_connections 2024-02-20T00:15:47.832 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:47.829+0000 7f4ccffff700 1 --2- 172.21.15.132:0/2038842604 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f4cb802ff60 0x7f4cb8032420 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:15:47.832 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:47.829+0000 7f4ccffff700 1 --2- 172.21.15.132:0/2038842604 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4cd00f51a0 0x7f4cd00f2f20 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:15:47.832 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:47.829+0000 7f4ccffff700 1 -- 172.21.15.132:0/2038842604 >> 172.21.15.132:0/2038842604 conn(0x7f4cd00f09a0 msgr2=0x7f4cd005fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:15:47.832 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:47.829+0000 7f4ccffff700 1 -- 172.21.15.132:0/2038842604 shutdown_connections 2024-02-20T00:15:47.832 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:47.829+0000 7f4ccffff700 1 -- 172.21.15.132:0/2038842604 wait complete. 2024-02-20T00:15:47.833 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:15:48.607 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:15:48 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:15:46.890253+0000 mgr.smithi084.bmqrnf (mgr.14162) 181 : cluster [DBG] pgmap v169: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:15:48.608 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:15:48 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:15:47.829122+0000 mon.smithi084 (mon.0) 221 : audit [DBG] from='client.? 172.21.15.132:0/2038842604' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:15:49.476 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:15:49.477 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:15:50.608 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:15:50 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:15:48.890557+0000 mgr.smithi084.bmqrnf (mgr.14162) 182 : cluster [DBG] pgmap v170: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:15:51.667 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:51.664+0000 7fe322473700 1 -- 172.21.15.132:0/1646437647 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe31c0f5410 msgr2=0x7fe31c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:15:51.667 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:51.664+0000 7fe322473700 1 --2- 172.21.15.132:0/1646437647 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe31c0f5410 0x7fe31c0f5830 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 rx=0x7fe30c009fb0 tx=0x7fe30c009b30).stop 2024-02-20T00:15:51.668 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:51.664+0000 7fe322473700 1 -- 172.21.15.132:0/1646437647 shutdown_connections 2024-02-20T00:15:51.668 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:51.664+0000 7fe322473700 1 --2- 172.21.15.132:0/1646437647 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe31c0f5410 0x7fe31c0f5830 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:15:51.668 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:51.664+0000 7fe322473700 1 -- 172.21.15.132:0/1646437647 >> 172.21.15.132:0/1646437647 conn(0x7fe31c0f0b70 msgr2=0x7fe31c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:15:51.668 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:51.664+0000 7fe322473700 1 -- 172.21.15.132:0/1646437647 shutdown_connections 2024-02-20T00:15:51.668 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:51.664+0000 7fe322473700 1 -- 172.21.15.132:0/1646437647 wait complete. 2024-02-20T00:15:51.668 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:51.664+0000 7fe322473700 1 Processor -- start 2024-02-20T00:15:51.668 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:51.665+0000 7fe322473700 1 -- start start 2024-02-20T00:15:51.668 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:51.665+0000 7fe322473700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe31c0f5410 0x7fe31c1056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:15:51.668 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:51.665+0000 7fe322473700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe31c105bd0 con 0x7fe31c0f5410 2024-02-20T00:15:51.668 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:51.665+0000 7fe31bfff700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe31c0f5410 0x7fe31c1056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:15:51.668 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:51.665+0000 7fe31bfff700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe31c0f5410 0x7fe31c1056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:33044/0 (socket says 172.21.15.132:33044) 2024-02-20T00:15:51.668 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:51.665+0000 7fe31bfff700 1 -- 172.21.15.132:0/337139636 learned_addr learned my addr 172.21.15.132:0/337139636 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:15:51.670 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:51.666+0000 7fe31bfff700 1 -- 172.21.15.132:0/337139636 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe30c005040 con 0x7fe31c0f5410 2024-02-20T00:15:51.670 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:51.666+0000 7fe31bfff700 1 --2- 172.21.15.132:0/337139636 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe31c0f5410 0x7fe31c1056e0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7fe30c0089f0 tx=0x7fe30c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:15:51.670 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:51.666+0000 7fe318ff9700 1 -- 172.21.15.132:0/337139636 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe30c009320 con 0x7fe31c0f5410 2024-02-20T00:15:51.670 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:51.666+0000 7fe322473700 1 -- 172.21.15.132:0/337139636 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe31c105dd0 con 0x7fe31c0f5410 2024-02-20T00:15:51.670 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:51.666+0000 7fe318ff9700 1 -- 172.21.15.132:0/337139636 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe30c013380 con 0x7fe31c0f5410 2024-02-20T00:15:51.671 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:51.666+0000 7fe318ff9700 1 -- 172.21.15.132:0/337139636 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe30c01ed80 con 0x7fe31c0f5410 2024-02-20T00:15:51.671 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:51.666+0000 7fe322473700 1 -- 172.21.15.132:0/337139636 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe31c106260 con 0x7fe31c0f5410 2024-02-20T00:15:51.671 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:51.667+0000 7fe318ff9700 1 -- 172.21.15.132:0/337139636 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe30c01e420 con 0x7fe31c0f5410 2024-02-20T00:15:51.671 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:51.667+0000 7fe322473700 1 -- 172.21.15.132:0/337139636 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe31c045ad0 con 0x7fe31c0f5410 2024-02-20T00:15:51.671 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:51.667+0000 7fe318ff9700 1 --2- 172.21.15.132:0/337139636 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fe30402ff60 0x7fe304032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:15:51.671 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:51.667+0000 7fe318ff9700 1 -- 172.21.15.132:0/337139636 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe30c040fe0 con 0x7fe31c0f5410 2024-02-20T00:15:51.671 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:51.667+0000 7fe31b7fe700 1 --2- 172.21.15.132:0/337139636 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fe30402ff60 0x7fe304032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:15:51.671 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:51.668+0000 7fe31b7fe700 1 --2- 172.21.15.132:0/337139636 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fe30402ff60 0x7fe304032420 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7fe3100096a0 tx=0x7fe310006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:15:51.673 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:51.672+0000 7fe318ff9700 1 -- 172.21.15.132:0/337139636 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe30c0095f0 con 0x7fe31c0f5410 2024-02-20T00:15:51.877 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:51.875+0000 7fe322473700 1 -- 172.21.15.132:0/337139636 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe31c1070b0 con 0x7fe31c0f5410 2024-02-20T00:15:51.878 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:51.876+0000 7fe318ff9700 1 -- 172.21.15.132:0/337139636 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe30c01cb10 con 0x7fe31c0f5410 2024-02-20T00:15:51.879 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:15:51.879 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:15:51.880 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:51.878+0000 7fe322473700 1 -- 172.21.15.132:0/337139636 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fe30402ff60 msgr2=0x7fe304032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:15:51.880 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:51.878+0000 7fe322473700 1 --2- 172.21.15.132:0/337139636 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fe30402ff60 0x7fe304032420 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7fe3100096a0 tx=0x7fe310006b40).stop 2024-02-20T00:15:51.880 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:51.878+0000 7fe322473700 1 -- 172.21.15.132:0/337139636 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe31c0f5410 msgr2=0x7fe31c1056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:15:51.880 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:51.878+0000 7fe322473700 1 --2- 172.21.15.132:0/337139636 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe31c0f5410 0x7fe31c1056e0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7fe30c0089f0 tx=0x7fe30c00b040).stop 2024-02-20T00:15:51.880 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:51.878+0000 7fe322473700 1 -- 172.21.15.132:0/337139636 shutdown_connections 2024-02-20T00:15:51.880 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:51.878+0000 7fe322473700 1 --2- 172.21.15.132:0/337139636 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fe30402ff60 0x7fe304032420 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:15:51.880 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:51.878+0000 7fe322473700 1 --2- 172.21.15.132:0/337139636 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe31c0f5410 0x7fe31c1056e0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:15:51.881 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:51.878+0000 7fe322473700 1 -- 172.21.15.132:0/337139636 >> 172.21.15.132:0/337139636 conn(0x7fe31c0f0b70 msgr2=0x7fe31c0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:15:51.881 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:51.878+0000 7fe322473700 1 -- 172.21.15.132:0/337139636 shutdown_connections 2024-02-20T00:15:51.881 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:51.878+0000 7fe322473700 1 -- 172.21.15.132:0/337139636 wait complete. 2024-02-20T00:15:51.881 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:15:52.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:15:51 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:15:50.890902+0000 mgr.smithi084.bmqrnf (mgr.14162) 183 : cluster [DBG] pgmap v171: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:15:52.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:15:51 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:15:51.877584+0000 mon.smithi084 (mon.0) 222 : audit [DBG] from='client.? 172.21.15.132:0/337139636' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:15:53.599 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:15:53.599 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:15:54.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:15:53 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:15:52.891250+0000 mgr.smithi084.bmqrnf (mgr.14162) 184 : cluster [DBG] pgmap v172: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:15:55.559 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:55.555+0000 7faf3e00e700 1 -- 172.21.15.132:0/857778308 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7faf380f5410 msgr2=0x7faf380f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:15:55.559 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:55.555+0000 7faf3e00e700 1 --2- 172.21.15.132:0/857778308 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7faf380f5410 0x7faf380f5830 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 rx=0x7faf20009fb0 tx=0x7faf20009b30).stop 2024-02-20T00:15:55.559 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:55.555+0000 7faf3e00e700 1 -- 172.21.15.132:0/857778308 shutdown_connections 2024-02-20T00:15:55.559 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:55.555+0000 7faf3e00e700 1 --2- 172.21.15.132:0/857778308 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7faf380f5410 0x7faf380f5830 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:15:55.559 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:55.555+0000 7faf3e00e700 1 -- 172.21.15.132:0/857778308 >> 172.21.15.132:0/857778308 conn(0x7faf380f0b70 msgr2=0x7faf380f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:15:55.559 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:55.556+0000 7faf3e00e700 1 -- 172.21.15.132:0/857778308 shutdown_connections 2024-02-20T00:15:55.559 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:55.556+0000 7faf3e00e700 1 -- 172.21.15.132:0/857778308 wait complete. 2024-02-20T00:15:55.559 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:55.556+0000 7faf3e00e700 1 Processor -- start 2024-02-20T00:15:55.559 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:55.556+0000 7faf3e00e700 1 -- start start 2024-02-20T00:15:55.559 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:55.557+0000 7faf3e00e700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7faf380f5410 0x7faf381056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:15:55.559 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:55.557+0000 7faf3e00e700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf38105bd0 con 0x7faf380f5410 2024-02-20T00:15:55.561 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:55.557+0000 7faf377fe700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7faf380f5410 0x7faf381056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:15:55.561 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:55.557+0000 7faf377fe700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7faf380f5410 0x7faf381056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:33054/0 (socket says 172.21.15.132:33054) 2024-02-20T00:15:55.561 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:55.557+0000 7faf377fe700 1 -- 172.21.15.132:0/2742422104 learned_addr learned my addr 172.21.15.132:0/2742422104 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:15:55.562 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:55.557+0000 7faf377fe700 1 -- 172.21.15.132:0/2742422104 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf20005040 con 0x7faf380f5410 2024-02-20T00:15:55.562 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:55.557+0000 7faf377fe700 1 --2- 172.21.15.132:0/2742422104 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7faf380f5410 0x7faf381056e0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 rx=0x7faf200089f0 tx=0x7faf2000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:15:55.562 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:55.558+0000 7faf2ffff700 1 -- 172.21.15.132:0/2742422104 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faf20009320 con 0x7faf380f5410 2024-02-20T00:15:55.562 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:55.558+0000 7faf3e00e700 1 -- 172.21.15.132:0/2742422104 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf38105dd0 con 0x7faf380f5410 2024-02-20T00:15:55.562 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:55.558+0000 7faf2ffff700 1 -- 172.21.15.132:0/2742422104 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7faf20013380 con 0x7faf380f5410 2024-02-20T00:15:55.562 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:55.558+0000 7faf2ffff700 1 -- 172.21.15.132:0/2742422104 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faf2001ed80 con 0x7faf380f5410 2024-02-20T00:15:55.562 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:55.558+0000 7faf3e00e700 1 -- 172.21.15.132:0/2742422104 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf38106260 con 0x7faf380f5410 2024-02-20T00:15:55.562 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:55.558+0000 7faf2ffff700 1 -- 172.21.15.132:0/2742422104 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7faf20014420 con 0x7faf380f5410 2024-02-20T00:15:55.562 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:55.558+0000 7faf3e00e700 1 -- 172.21.15.132:0/2742422104 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf38045ad0 con 0x7faf380f5410 2024-02-20T00:15:55.562 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:55.559+0000 7faf2ffff700 1 --2- 172.21.15.132:0/2742422104 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7faf2402ff60 0x7faf24032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:15:55.562 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:55.559+0000 7faf2ffff700 1 -- 172.21.15.132:0/2742422104 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7faf20040e40 con 0x7faf380f5410 2024-02-20T00:15:55.562 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:55.559+0000 7faf36ffd700 1 --2- 172.21.15.132:0/2742422104 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7faf2402ff60 0x7faf24032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:15:55.562 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:55.559+0000 7faf36ffd700 1 --2- 172.21.15.132:0/2742422104 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7faf2402ff60 0x7faf24032420 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7faf280096a0 tx=0x7faf28006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:15:55.565 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:55.563+0000 7faf2ffff700 1 -- 172.21.15.132:0/2742422104 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7faf2001c6e0 con 0x7faf380f5410 2024-02-20T00:15:55.768 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:55.766+0000 7faf3e00e700 1 -- 172.21.15.132:0/2742422104 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7faf38107110 con 0x7faf380f5410 2024-02-20T00:15:55.769 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:55.766+0000 7faf2ffff700 1 -- 172.21.15.132:0/2742422104 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7faf2001c320 con 0x7faf380f5410 2024-02-20T00:15:55.769 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:15:55.769 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:15:55.770 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:55.768+0000 7faf3e00e700 1 -- 172.21.15.132:0/2742422104 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7faf2402ff60 msgr2=0x7faf24032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:15:55.771 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:55.768+0000 7faf3e00e700 1 --2- 172.21.15.132:0/2742422104 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7faf2402ff60 0x7faf24032420 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7faf280096a0 tx=0x7faf28006b40).stop 2024-02-20T00:15:55.771 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:55.768+0000 7faf3e00e700 1 -- 172.21.15.132:0/2742422104 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7faf380f5410 msgr2=0x7faf381056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:15:55.771 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:55.768+0000 7faf3e00e700 1 --2- 172.21.15.132:0/2742422104 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7faf380f5410 0x7faf381056e0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 rx=0x7faf200089f0 tx=0x7faf2000b040).stop 2024-02-20T00:15:55.771 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:55.768+0000 7faf3e00e700 1 -- 172.21.15.132:0/2742422104 shutdown_connections 2024-02-20T00:15:55.771 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:55.768+0000 7faf3e00e700 1 --2- 172.21.15.132:0/2742422104 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7faf2402ff60 0x7faf24032420 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:15:55.771 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:55.768+0000 7faf3e00e700 1 --2- 172.21.15.132:0/2742422104 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7faf380f5410 0x7faf381056e0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:15:55.771 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:55.768+0000 7faf3e00e700 1 -- 172.21.15.132:0/2742422104 >> 172.21.15.132:0/2742422104 conn(0x7faf380f0b70 msgr2=0x7faf380f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:15:55.771 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:55.768+0000 7faf3e00e700 1 -- 172.21.15.132:0/2742422104 shutdown_connections 2024-02-20T00:15:55.771 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:55.768+0000 7faf3e00e700 1 -- 172.21.15.132:0/2742422104 wait complete. 2024-02-20T00:15:55.771 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:15:56.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:15:55 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:15:54.891652+0000 mgr.smithi084.bmqrnf (mgr.14162) 185 : cluster [DBG] pgmap v173: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:15:56.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:15:55 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:15:55.768072+0000 mon.smithi084 (mon.0) 223 : audit [DBG] from='client.? 172.21.15.132:0/2742422104' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:15:57.397 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:15:57.398 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:15:58.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:15:57 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:15:56.891914+0000 mgr.smithi084.bmqrnf (mgr.14162) 186 : cluster [DBG] pgmap v174: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:15:58.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:15:57 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:15:56.975744+0000 mon.smithi084 (mon.0) 224 : audit [INF] from='mgr.14162 172.21.15.84:0/2152933216' entity='mgr.smithi084.bmqrnf' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi084.bmqrnf/mirror_snapshot_schedule"}]: dispatch 2024-02-20T00:15:58.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:15:57 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:15:56.981486+0000 mon.smithi084 (mon.0) 225 : audit [INF] from='mgr.14162 172.21.15.84:0/2152933216' entity='mgr.smithi084.bmqrnf' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi084.bmqrnf/trash_purge_schedule"}]: dispatch 2024-02-20T00:15:59.177 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:59.159+0000 7f74f1382700 1 -- 172.21.15.132:0/685601530 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f74ec0f5410 msgr2=0x7f74ec0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:15:59.177 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:59.159+0000 7f74f1382700 1 --2- 172.21.15.132:0/685601530 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f74ec0f5410 0x7f74ec0f5830 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 rx=0x7f74dc009fb0 tx=0x7f74dc009b30).stop 2024-02-20T00:15:59.177 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:59.159+0000 7f74f1382700 1 -- 172.21.15.132:0/685601530 shutdown_connections 2024-02-20T00:15:59.177 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:59.159+0000 7f74f1382700 1 --2- 172.21.15.132:0/685601530 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f74ec0f5410 0x7f74ec0f5830 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:15:59.177 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:59.159+0000 7f74f1382700 1 -- 172.21.15.132:0/685601530 >> 172.21.15.132:0/685601530 conn(0x7f74ec0f0b70 msgr2=0x7f74ec0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:15:59.178 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:59.159+0000 7f74f1382700 1 -- 172.21.15.132:0/685601530 shutdown_connections 2024-02-20T00:15:59.178 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:59.159+0000 7f74f1382700 1 -- 172.21.15.132:0/685601530 wait complete. 2024-02-20T00:15:59.178 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:59.160+0000 7f74f1382700 1 Processor -- start 2024-02-20T00:15:59.178 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:59.160+0000 7f74f1382700 1 -- start start 2024-02-20T00:15:59.178 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:59.160+0000 7f74f1382700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f74ec0f5410 0x7f74ec0fd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:15:59.178 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:59.160+0000 7f74f1382700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74ec0fb960 con 0x7f74ec0f5410 2024-02-20T00:15:59.178 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:59.160+0000 7f74eaffd700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f74ec0f5410 0x7f74ec0fd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:15:59.178 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:59.160+0000 7f74eaffd700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f74ec0f5410 0x7f74ec0fd4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:60216/0 (socket says 172.21.15.132:60216) 2024-02-20T00:15:59.178 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:59.160+0000 7f74eaffd700 1 -- 172.21.15.132:0/3615976956 learned_addr learned my addr 172.21.15.132:0/3615976956 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:15:59.178 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:59.161+0000 7f74eaffd700 1 -- 172.21.15.132:0/3615976956 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74dc005040 con 0x7f74ec0f5410 2024-02-20T00:15:59.178 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:59.161+0000 7f74eaffd700 1 --2- 172.21.15.132:0/3615976956 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f74ec0f5410 0x7f74ec0fd4e0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7f74dc005010 tx=0x7f74dc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:15:59.178 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:59.161+0000 7f74e37fe700 1 -- 172.21.15.132:0/3615976956 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f74dc009320 con 0x7f74ec0f5410 2024-02-20T00:15:59.178 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:59.161+0000 7f74f1382700 1 -- 172.21.15.132:0/3615976956 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74ec0fbb00 con 0x7f74ec0f5410 2024-02-20T00:15:59.179 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:59.161+0000 7f74e37fe700 1 -- 172.21.15.132:0/3615976956 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f74dc013380 con 0x7f74ec0f5410 2024-02-20T00:15:59.179 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:59.161+0000 7f74e37fe700 1 -- 172.21.15.132:0/3615976956 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f74dc014e50 con 0x7f74ec0f5410 2024-02-20T00:15:59.179 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:59.161+0000 7f74f1382700 1 -- 172.21.15.132:0/3615976956 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74ec0fbf90 con 0x7f74ec0f5410 2024-02-20T00:15:59.179 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:59.162+0000 7f74f1382700 1 -- 172.21.15.132:0/3615976956 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74ec059100 con 0x7f74ec0f5410 2024-02-20T00:15:59.179 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:59.162+0000 7f74e37fe700 1 -- 172.21.15.132:0/3615976956 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f74dc0095f0 con 0x7f74ec0f5410 2024-02-20T00:15:59.179 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:59.162+0000 7f74e37fe700 1 --2- 172.21.15.132:0/3615976956 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f74d802ff10 0x7f74d80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:15:59.179 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:59.162+0000 7f74e37fe700 1 -- 172.21.15.132:0/3615976956 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f74dc040f70 con 0x7f74ec0f5410 2024-02-20T00:15:59.179 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:59.162+0000 7f74ea7fc700 1 --2- 172.21.15.132:0/3615976956 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f74d802ff10 0x7f74d80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:15:59.179 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:59.162+0000 7f74ea7fc700 1 --2- 172.21.15.132:0/3615976956 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f74d802ff10 0x7f74d80323d0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f74d40096a0 tx=0x7f74d4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:15:59.179 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:59.167+0000 7f74e37fe700 1 -- 172.21.15.132:0/3615976956 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f74dc014420 con 0x7f74ec0f5410 2024-02-20T00:15:59.376 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:59.374+0000 7f74f1382700 1 -- 172.21.15.132:0/3615976956 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f74ec045ad0 con 0x7f74ec0f5410 2024-02-20T00:15:59.376 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:59.374+0000 7f74e37fe700 1 -- 172.21.15.132:0/3615976956 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f74dc010230 con 0x7f74ec0f5410 2024-02-20T00:15:59.376 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:15:59.376 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:15:59.377 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:59.376+0000 7f74f1382700 1 -- 172.21.15.132:0/3615976956 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f74d802ff10 msgr2=0x7f74d80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:15:59.377 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:59.376+0000 7f74f1382700 1 --2- 172.21.15.132:0/3615976956 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f74d802ff10 0x7f74d80323d0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f74d40096a0 tx=0x7f74d4006b40).stop 2024-02-20T00:15:59.377 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:59.376+0000 7f74f1382700 1 -- 172.21.15.132:0/3615976956 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f74ec0f5410 msgr2=0x7f74ec0fd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:15:59.377 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:59.376+0000 7f74f1382700 1 --2- 172.21.15.132:0/3615976956 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f74ec0f5410 0x7f74ec0fd4e0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7f74dc005010 tx=0x7f74dc00b040).stop 2024-02-20T00:15:59.378 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:59.376+0000 7f74f1382700 1 -- 172.21.15.132:0/3615976956 shutdown_connections 2024-02-20T00:15:59.378 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:59.376+0000 7f74f1382700 1 --2- 172.21.15.132:0/3615976956 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f74d802ff10 0x7f74d80323d0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:15:59.378 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:59.376+0000 7f74f1382700 1 --2- 172.21.15.132:0/3615976956 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f74ec0f5410 0x7f74ec0fd4e0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:15:59.378 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:59.376+0000 7f74f1382700 1 -- 172.21.15.132:0/3615976956 >> 172.21.15.132:0/3615976956 conn(0x7f74ec0f0b70 msgr2=0x7f74ec05fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:15:59.378 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:59.376+0000 7f74f1382700 1 -- 172.21.15.132:0/3615976956 shutdown_connections 2024-02-20T00:15:59.378 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:15:59.376+0000 7f74f1382700 1 -- 172.21.15.132:0/3615976956 wait complete. 2024-02-20T00:15:59.379 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:16:00.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:15:59 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:15:58.892223+0000 mgr.smithi084.bmqrnf (mgr.14162) 187 : cluster [DBG] pgmap v175: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:16:00.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:15:59 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:15:59.374892+0000 mon.smithi084 (mon.0) 226 : audit [DBG] from='client.? 172.21.15.132:0/3615976956' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:16:01.006 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:16:01.006 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:16:02.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:16:01 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:16:00.892486+0000 mgr.smithi084.bmqrnf (mgr.14162) 188 : cluster [DBG] pgmap v176: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:16:02.910 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:02.908+0000 7f0f35d95700 1 -- 172.21.15.132:0/2435284768 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0f300f51a0 msgr2=0x7f0f300f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:16:02.911 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:02.908+0000 7f0f35d95700 1 --2- 172.21.15.132:0/2435284768 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0f300f51a0 0x7f0f300f55c0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 rx=0x7f0f24009fb0 tx=0x7f0f24009b30).stop 2024-02-20T00:16:02.911 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:02.908+0000 7f0f35d95700 1 -- 172.21.15.132:0/2435284768 shutdown_connections 2024-02-20T00:16:02.911 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:02.908+0000 7f0f35d95700 1 --2- 172.21.15.132:0/2435284768 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0f300f51a0 0x7f0f300f55c0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:16:02.911 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:02.908+0000 7f0f35d95700 1 -- 172.21.15.132:0/2435284768 >> 172.21.15.132:0/2435284768 conn(0x7f0f300f09a0 msgr2=0x7f0f300f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:16:02.911 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:02.908+0000 7f0f35d95700 1 -- 172.21.15.132:0/2435284768 shutdown_connections 2024-02-20T00:16:02.911 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:02.908+0000 7f0f35d95700 1 -- 172.21.15.132:0/2435284768 wait complete. 2024-02-20T00:16:02.911 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:02.908+0000 7f0f35d95700 1 Processor -- start 2024-02-20T00:16:02.911 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:02.908+0000 7f0f35d95700 1 -- start start 2024-02-20T00:16:02.911 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:02.909+0000 7f0f35d95700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0f300f51a0 0x7f0f301092f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:16:02.911 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:02.909+0000 7f0f35d95700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f30060220 con 0x7f0f300f51a0 2024-02-20T00:16:02.911 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:02.909+0000 7f0f34d93700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0f300f51a0 0x7f0f301092f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:16:02.911 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:02.909+0000 7f0f34d93700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0f300f51a0 0x7f0f301092f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:60236/0 (socket says 172.21.15.132:60236) 2024-02-20T00:16:02.911 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:02.909+0000 7f0f34d93700 1 -- 172.21.15.132:0/2789271663 learned_addr learned my addr 172.21.15.132:0/2789271663 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:16:02.911 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:02.909+0000 7f0f34d93700 1 -- 172.21.15.132:0/2789271663 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f24005040 con 0x7f0f300f51a0 2024-02-20T00:16:02.912 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:02.909+0000 7f0f34d93700 1 --2- 172.21.15.132:0/2789271663 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0f300f51a0 0x7f0f301092f0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 rx=0x7f0f240089f0 tx=0x7f0f2400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:16:02.913 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:02.910+0000 7f0f2d7fa700 1 -- 172.21.15.132:0/2789271663 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0f24009320 con 0x7f0f300f51a0 2024-02-20T00:16:02.913 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:02.910+0000 7f0f35d95700 1 -- 172.21.15.132:0/2789271663 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f30060420 con 0x7f0f300f51a0 2024-02-20T00:16:02.913 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:02.910+0000 7f0f2d7fa700 1 -- 172.21.15.132:0/2789271663 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0f24013380 con 0x7f0f300f51a0 2024-02-20T00:16:02.913 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:02.910+0000 7f0f2d7fa700 1 -- 172.21.15.132:0/2789271663 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0f24014e20 con 0x7f0f300f51a0 2024-02-20T00:16:02.913 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:02.910+0000 7f0f35d95700 1 -- 172.21.15.132:0/2789271663 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f301099a0 con 0x7f0f300f51a0 2024-02-20T00:16:02.913 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:02.911+0000 7f0f2d7fa700 1 -- 172.21.15.132:0/2789271663 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0f240095f0 con 0x7f0f300f51a0 2024-02-20T00:16:02.913 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:02.911+0000 7f0f35d95700 1 -- 172.21.15.132:0/2789271663 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f30059100 con 0x7f0f300f51a0 2024-02-20T00:16:02.913 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:02.911+0000 7f0f2d7fa700 1 --2- 172.21.15.132:0/2789271663 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f0f2002ff10 0x7f0f200323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:16:02.913 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:02.911+0000 7f0f2d7fa700 1 -- 172.21.15.132:0/2789271663 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0f24040e90 con 0x7f0f300f51a0 2024-02-20T00:16:02.913 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:02.911+0000 7f0f2ffff700 1 --2- 172.21.15.132:0/2789271663 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f0f2002ff10 0x7f0f200323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:16:02.914 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:02.911+0000 7f0f2ffff700 1 --2- 172.21.15.132:0/2789271663 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f0f2002ff10 0x7f0f200323d0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f0f1c0096a0 tx=0x7f0f1c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:16:02.916 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:02.915+0000 7f0f2d7fa700 1 -- 172.21.15.132:0/2789271663 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0f24014420 con 0x7f0f300f51a0 2024-02-20T00:16:03.121 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:03.120+0000 7f0f35d95700 1 -- 172.21.15.132:0/2789271663 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0f30045ad0 con 0x7f0f300f51a0 2024-02-20T00:16:03.122 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:03.121+0000 7f0f2d7fa700 1 -- 172.21.15.132:0/2789271663 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0f2401d980 con 0x7f0f300f51a0 2024-02-20T00:16:03.122 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:16:03.122 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:16:03.124 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:03.122+0000 7f0f35d95700 1 -- 172.21.15.132:0/2789271663 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f0f2002ff10 msgr2=0x7f0f200323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:16:03.124 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:03.122+0000 7f0f35d95700 1 --2- 172.21.15.132:0/2789271663 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f0f2002ff10 0x7f0f200323d0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f0f1c0096a0 tx=0x7f0f1c006b40).stop 2024-02-20T00:16:03.124 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:03.122+0000 7f0f35d95700 1 -- 172.21.15.132:0/2789271663 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0f300f51a0 msgr2=0x7f0f301092f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:16:03.124 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:03.122+0000 7f0f35d95700 1 --2- 172.21.15.132:0/2789271663 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0f300f51a0 0x7f0f301092f0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 rx=0x7f0f240089f0 tx=0x7f0f2400b040).stop 2024-02-20T00:16:03.124 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:03.122+0000 7f0f35d95700 1 -- 172.21.15.132:0/2789271663 shutdown_connections 2024-02-20T00:16:03.124 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:03.122+0000 7f0f35d95700 1 --2- 172.21.15.132:0/2789271663 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f0f2002ff10 0x7f0f200323d0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:16:03.124 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:03.122+0000 7f0f35d95700 1 --2- 172.21.15.132:0/2789271663 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0f300f51a0 0x7f0f301092f0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:16:03.124 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:03.122+0000 7f0f35d95700 1 -- 172.21.15.132:0/2789271663 >> 172.21.15.132:0/2789271663 conn(0x7f0f300f09a0 msgr2=0x7f0f300f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:16:03.124 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:03.122+0000 7f0f35d95700 1 -- 172.21.15.132:0/2789271663 shutdown_connections 2024-02-20T00:16:03.124 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:03.122+0000 7f0f35d95700 1 -- 172.21.15.132:0/2789271663 wait complete. 2024-02-20T00:16:03.125 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:16:04.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:16:03 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:16:02.892875+0000 mgr.smithi084.bmqrnf (mgr.14162) 189 : cluster [DBG] pgmap v177: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:16:04.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:16:03 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:16:03.121041+0000 mon.smithi084 (mon.0) 227 : audit [DBG] from='client.? 172.21.15.132:0/2789271663' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:16:04.941 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:16:04.941 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:16:06.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:16:05 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:16:04.893243+0000 mgr.smithi084.bmqrnf (mgr.14162) 190 : cluster [DBG] pgmap v178: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:16:06.910 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:06.900+0000 7f214c502700 1 -- 172.21.15.132:0/3935205136 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f21440f5410 msgr2=0x7f21440f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:16:06.910 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:06.900+0000 7f214c502700 1 --2- 172.21.15.132:0/3935205136 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f21440f5410 0x7f21440f5830 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 rx=0x7f2140009fb0 tx=0x7f2140009b30).stop 2024-02-20T00:16:06.910 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:06.900+0000 7f214c502700 1 -- 172.21.15.132:0/3935205136 shutdown_connections 2024-02-20T00:16:06.910 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:06.900+0000 7f214c502700 1 --2- 172.21.15.132:0/3935205136 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f21440f5410 0x7f21440f5830 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:16:06.910 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:06.900+0000 7f214c502700 1 -- 172.21.15.132:0/3935205136 >> 172.21.15.132:0/3935205136 conn(0x7f21440f0b70 msgr2=0x7f21440f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:16:06.911 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:06.900+0000 7f214c502700 1 -- 172.21.15.132:0/3935205136 shutdown_connections 2024-02-20T00:16:06.911 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:06.900+0000 7f214c502700 1 -- 172.21.15.132:0/3935205136 wait complete. 2024-02-20T00:16:06.911 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:06.901+0000 7f214c502700 1 Processor -- start 2024-02-20T00:16:06.911 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:06.901+0000 7f214c502700 1 -- start start 2024-02-20T00:16:06.911 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:06.901+0000 7f214c502700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f21440f5410 0x7f21441094e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:16:06.911 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:06.901+0000 7f214c502700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2144060240 con 0x7f21440f5410 2024-02-20T00:16:06.911 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:06.901+0000 7f214a29e700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f21440f5410 0x7f21441094e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:16:06.911 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:06.901+0000 7f214a29e700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f21440f5410 0x7f21441094e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:60252/0 (socket says 172.21.15.132:60252) 2024-02-20T00:16:06.911 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:06.901+0000 7f214a29e700 1 -- 172.21.15.132:0/1773457969 learned_addr learned my addr 172.21.15.132:0/1773457969 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:16:06.911 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:06.902+0000 7f214a29e700 1 -- 172.21.15.132:0/1773457969 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2140005040 con 0x7f21440f5410 2024-02-20T00:16:06.911 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:06.902+0000 7f214a29e700 1 --2- 172.21.15.132:0/1773457969 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f21440f5410 0x7f21441094e0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7f2140005010 tx=0x7f214000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:16:06.911 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:06.902+0000 7f213affd700 1 -- 172.21.15.132:0/1773457969 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2140009320 con 0x7f21440f5410 2024-02-20T00:16:06.911 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:06.902+0000 7f214c502700 1 -- 172.21.15.132:0/1773457969 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2144060440 con 0x7f21440f5410 2024-02-20T00:16:06.911 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:06.902+0000 7f213affd700 1 -- 172.21.15.132:0/1773457969 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2140013380 con 0x7f21440f5410 2024-02-20T00:16:06.912 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:06.902+0000 7f213affd700 1 -- 172.21.15.132:0/1773457969 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2140014e50 con 0x7f21440f5410 2024-02-20T00:16:06.912 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:06.902+0000 7f214c502700 1 -- 172.21.15.132:0/1773457969 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2144109b90 con 0x7f21440f5410 2024-02-20T00:16:06.912 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:06.903+0000 7f213affd700 1 -- 172.21.15.132:0/1773457969 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f21400095f0 con 0x7f21440f5410 2024-02-20T00:16:06.912 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:06.903+0000 7f214c502700 1 -- 172.21.15.132:0/1773457969 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2144045ad0 con 0x7f21440f5410 2024-02-20T00:16:06.912 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:06.903+0000 7f213affd700 1 --2- 172.21.15.132:0/1773457969 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f213002ff10 0x7f21300323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:16:06.912 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:06.903+0000 7f213affd700 1 -- 172.21.15.132:0/1773457969 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2140040fa0 con 0x7f21440f5410 2024-02-20T00:16:06.912 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:06.903+0000 7f2149a9d700 1 --2- 172.21.15.132:0/1773457969 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f213002ff10 0x7f21300323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:16:06.912 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:06.904+0000 7f2149a9d700 1 --2- 172.21.15.132:0/1773457969 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f213002ff10 0x7f21300323d0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f21340096a0 tx=0x7f2134006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:16:06.912 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:06.908+0000 7f213affd700 1 -- 172.21.15.132:0/1773457969 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2140014420 con 0x7f21440f5410 2024-02-20T00:16:07.116 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:07.115+0000 7f214c502700 1 -- 172.21.15.132:0/1773457969 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f214410a3b0 con 0x7f21440f5410 2024-02-20T00:16:07.117 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:07.116+0000 7f213affd700 1 -- 172.21.15.132:0/1773457969 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2140010230 con 0x7f21440f5410 2024-02-20T00:16:07.117 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:16:07.117 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:16:07.119 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:07.117+0000 7f214c502700 1 -- 172.21.15.132:0/1773457969 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f213002ff10 msgr2=0x7f21300323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:16:07.119 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:07.118+0000 7f214c502700 1 --2- 172.21.15.132:0/1773457969 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f213002ff10 0x7f21300323d0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f21340096a0 tx=0x7f2134006b40).stop 2024-02-20T00:16:07.119 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:07.118+0000 7f214c502700 1 -- 172.21.15.132:0/1773457969 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f21440f5410 msgr2=0x7f21441094e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:16:07.119 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:07.118+0000 7f214c502700 1 --2- 172.21.15.132:0/1773457969 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f21440f5410 0x7f21441094e0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7f2140005010 tx=0x7f214000b040).stop 2024-02-20T00:16:07.119 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:07.118+0000 7f214c502700 1 -- 172.21.15.132:0/1773457969 shutdown_connections 2024-02-20T00:16:07.119 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:07.118+0000 7f214c502700 1 --2- 172.21.15.132:0/1773457969 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f213002ff10 0x7f21300323d0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:16:07.119 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:07.118+0000 7f214c502700 1 --2- 172.21.15.132:0/1773457969 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f21440f5410 0x7f21441094e0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:16:07.119 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:07.118+0000 7f214c502700 1 -- 172.21.15.132:0/1773457969 >> 172.21.15.132:0/1773457969 conn(0x7f21440f0b70 msgr2=0x7f21440f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:16:07.120 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:07.118+0000 7f214c502700 1 -- 172.21.15.132:0/1773457969 shutdown_connections 2024-02-20T00:16:07.120 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:07.118+0000 7f214c502700 1 -- 172.21.15.132:0/1773457969 wait complete. 2024-02-20T00:16:07.120 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:16:08.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:16:07 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:16:06.893611+0000 mgr.smithi084.bmqrnf (mgr.14162) 191 : cluster [DBG] pgmap v179: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:16:08.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:16:07 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:16:07.116599+0000 mon.smithi084 (mon.0) 228 : audit [DBG] from='client.? 172.21.15.132:0/1773457969' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:16:08.786 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:16:08.787 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:16:10.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:16:09 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:16:08.893912+0000 mgr.smithi084.bmqrnf (mgr.14162) 192 : cluster [DBG] pgmap v180: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:16:10.636 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:10.634+0000 7f69cffff700 1 -- 172.21.15.132:0/31251635 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f69d00f51a0 msgr2=0x7f69d00f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:16:10.636 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:10.634+0000 7f69cffff700 1 --2- 172.21.15.132:0/31251635 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f69d00f51a0 0x7f69d00f55c0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 rx=0x7f69c0009fb0 tx=0x7f69c0009b30).stop 2024-02-20T00:16:10.636 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:10.634+0000 7f69cffff700 1 -- 172.21.15.132:0/31251635 shutdown_connections 2024-02-20T00:16:10.637 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:10.634+0000 7f69cffff700 1 --2- 172.21.15.132:0/31251635 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f69d00f51a0 0x7f69d00f55c0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:16:10.637 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:10.634+0000 7f69cffff700 1 -- 172.21.15.132:0/31251635 >> 172.21.15.132:0/31251635 conn(0x7f69d00f09a0 msgr2=0x7f69d00f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:16:10.637 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:10.634+0000 7f69cffff700 1 -- 172.21.15.132:0/31251635 shutdown_connections 2024-02-20T00:16:10.637 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:10.634+0000 7f69cffff700 1 -- 172.21.15.132:0/31251635 wait complete. 2024-02-20T00:16:10.637 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:10.635+0000 7f69cffff700 1 Processor -- start 2024-02-20T00:16:10.637 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:10.635+0000 7f69cffff700 1 -- start start 2024-02-20T00:16:10.637 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:10.635+0000 7f69cffff700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f69d00f51a0 0x7f69d00f2f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:16:10.637 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:10.635+0000 7f69cffff700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69d00f13e0 con 0x7f69d00f51a0 2024-02-20T00:16:10.639 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:10.635+0000 7f69ceffd700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f69d00f51a0 0x7f69d00f2f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:16:10.639 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:10.635+0000 7f69ceffd700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f69d00f51a0 0x7f69d00f2f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:50266/0 (socket says 172.21.15.132:50266) 2024-02-20T00:16:10.639 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:10.635+0000 7f69ceffd700 1 -- 172.21.15.132:0/2528110868 learned_addr learned my addr 172.21.15.132:0/2528110868 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:16:10.639 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:10.636+0000 7f69ceffd700 1 -- 172.21.15.132:0/2528110868 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69c0005040 con 0x7f69d00f51a0 2024-02-20T00:16:10.639 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:10.636+0000 7f69ceffd700 1 --2- 172.21.15.132:0/2528110868 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f69d00f51a0 0x7f69d00f2f60 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7f69c00089f0 tx=0x7f69c000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:16:10.639 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:10.636+0000 7f69b77fe700 1 -- 172.21.15.132:0/2528110868 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f69c0009320 con 0x7f69d00f51a0 2024-02-20T00:16:10.639 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:10.636+0000 7f69cffff700 1 -- 172.21.15.132:0/2528110868 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69d00f1580 con 0x7f69d00f51a0 2024-02-20T00:16:10.639 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:10.636+0000 7f69b77fe700 1 -- 172.21.15.132:0/2528110868 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f69c0013380 con 0x7f69d00f51a0 2024-02-20T00:16:10.639 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:10.636+0000 7f69b77fe700 1 -- 172.21.15.132:0/2528110868 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f69c0014e20 con 0x7f69d00f51a0 2024-02-20T00:16:10.639 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:10.636+0000 7f69cffff700 1 -- 172.21.15.132:0/2528110868 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69d00f1a10 con 0x7f69d00f51a0 2024-02-20T00:16:10.639 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:10.637+0000 7f69b77fe700 1 -- 172.21.15.132:0/2528110868 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f69c0014420 con 0x7f69d00f51a0 2024-02-20T00:16:10.640 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:10.637+0000 7f69cffff700 1 -- 172.21.15.132:0/2528110868 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69d0059100 con 0x7f69d00f51a0 2024-02-20T00:16:10.640 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:10.637+0000 7f69b77fe700 1 --2- 172.21.15.132:0/2528110868 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f69b802ff10 0x7f69b80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:16:10.640 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:10.637+0000 7f69b77fe700 1 -- 172.21.15.132:0/2528110868 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f69c0040f00 con 0x7f69d00f51a0 2024-02-20T00:16:10.640 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:10.637+0000 7f69ce7fc700 1 --2- 172.21.15.132:0/2528110868 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f69b802ff10 0x7f69b80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:16:10.640 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:10.638+0000 7f69ce7fc700 1 --2- 172.21.15.132:0/2528110868 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f69b802ff10 0x7f69b80323d0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f69c40096a0 tx=0x7f69c4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:16:10.643 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:10.642+0000 7f69b77fe700 1 -- 172.21.15.132:0/2528110868 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f69c00095f0 con 0x7f69d00f51a0 2024-02-20T00:16:10.846 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:10.845+0000 7f69cffff700 1 -- 172.21.15.132:0/2528110868 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f69d0045ad0 con 0x7f69d00f51a0 2024-02-20T00:16:10.847 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:10.845+0000 7f69b77fe700 1 -- 172.21.15.132:0/2528110868 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f69c001db70 con 0x7f69d00f51a0 2024-02-20T00:16:10.847 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:16:10.847 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:16:10.848 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:10.847+0000 7f69cffff700 1 -- 172.21.15.132:0/2528110868 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f69b802ff10 msgr2=0x7f69b80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:16:10.848 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:10.847+0000 7f69cffff700 1 --2- 172.21.15.132:0/2528110868 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f69b802ff10 0x7f69b80323d0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f69c40096a0 tx=0x7f69c4006b40).stop 2024-02-20T00:16:10.848 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:10.847+0000 7f69cffff700 1 -- 172.21.15.132:0/2528110868 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f69d00f51a0 msgr2=0x7f69d00f2f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:16:10.849 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:10.847+0000 7f69cffff700 1 --2- 172.21.15.132:0/2528110868 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f69d00f51a0 0x7f69d00f2f60 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7f69c00089f0 tx=0x7f69c000b040).stop 2024-02-20T00:16:10.849 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:10.847+0000 7f69cffff700 1 -- 172.21.15.132:0/2528110868 shutdown_connections 2024-02-20T00:16:10.849 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:10.847+0000 7f69cffff700 1 --2- 172.21.15.132:0/2528110868 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f69b802ff10 0x7f69b80323d0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:16:10.849 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:10.847+0000 7f69cffff700 1 --2- 172.21.15.132:0/2528110868 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f69d00f51a0 0x7f69d00f2f60 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:16:10.849 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:10.847+0000 7f69cffff700 1 -- 172.21.15.132:0/2528110868 >> 172.21.15.132:0/2528110868 conn(0x7f69d00f09a0 msgr2=0x7f69d005fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:16:10.849 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:10.847+0000 7f69cffff700 1 -- 172.21.15.132:0/2528110868 shutdown_connections 2024-02-20T00:16:10.849 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:10.847+0000 7f69cffff700 1 -- 172.21.15.132:0/2528110868 wait complete. 2024-02-20T00:16:10.849 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:16:11.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:16:10 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:16:10.846003+0000 mon.smithi084 (mon.0) 229 : audit [DBG] from='client.? 172.21.15.132:0/2528110868' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:16:12.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:16:11 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:16:10.894252+0000 mgr.smithi084.bmqrnf (mgr.14162) 193 : cluster [DBG] pgmap v181: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:16:12.593 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:16:12.593 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:16:14.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:16:13 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:16:12.894629+0000 mgr.smithi084.bmqrnf (mgr.14162) 194 : cluster [DBG] pgmap v182: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:16:14.552 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:14.531+0000 7fc7530ce700 1 -- 172.21.15.132:0/551806780 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc74c0f62a0 msgr2=0x7fc74c0f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:16:14.552 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:14.531+0000 7fc7530ce700 1 --2- 172.21.15.132:0/551806780 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc74c0f62a0 0x7fc74c0f66c0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 rx=0x7fc748009fb0 tx=0x7fc748009b30).stop 2024-02-20T00:16:14.552 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:14.531+0000 7fc7530ce700 1 -- 172.21.15.132:0/551806780 shutdown_connections 2024-02-20T00:16:14.552 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:14.531+0000 7fc7530ce700 1 --2- 172.21.15.132:0/551806780 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc74c0f62a0 0x7fc74c0f66c0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:16:14.552 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:14.531+0000 7fc7530ce700 1 -- 172.21.15.132:0/551806780 >> 172.21.15.132:0/551806780 conn(0x7fc74c0f1aa0 msgr2=0x7fc74c0f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:16:14.552 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:14.531+0000 7fc7530ce700 1 -- 172.21.15.132:0/551806780 shutdown_connections 2024-02-20T00:16:14.552 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:14.531+0000 7fc7530ce700 1 -- 172.21.15.132:0/551806780 wait complete. 2024-02-20T00:16:14.552 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:14.531+0000 7fc7530ce700 1 Processor -- start 2024-02-20T00:16:14.553 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:14.531+0000 7fc7530ce700 1 -- start start 2024-02-20T00:16:14.553 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:14.531+0000 7fc7530ce700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc74c0f62a0 0x7fc74c06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:16:14.553 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:14.531+0000 7fc7530ce700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc74c069e20 con 0x7fc74c0f62a0 2024-02-20T00:16:14.553 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:14.532+0000 7fc7520cc700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc74c0f62a0 0x7fc74c06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:16:14.553 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:14.532+0000 7fc7520cc700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc74c0f62a0 0x7fc74c06b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:50296/0 (socket says 172.21.15.132:50296) 2024-02-20T00:16:14.553 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:14.532+0000 7fc7520cc700 1 -- 172.21.15.132:0/2225939750 learned_addr learned my addr 172.21.15.132:0/2225939750 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:16:14.553 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:14.532+0000 7fc7520cc700 1 -- 172.21.15.132:0/2225939750 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc748005040 con 0x7fc74c0f62a0 2024-02-20T00:16:14.553 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:14.532+0000 7fc7520cc700 1 --2- 172.21.15.132:0/2225939750 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc74c0f62a0 0x7fc74c06b9a0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0x7fc7480089f0 tx=0x7fc74800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:16:14.553 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:14.532+0000 7fc742ffd700 1 -- 172.21.15.132:0/2225939750 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc748009320 con 0x7fc74c0f62a0 2024-02-20T00:16:14.553 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:14.532+0000 7fc7530ce700 1 -- 172.21.15.132:0/2225939750 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc74c069fc0 con 0x7fc74c0f62a0 2024-02-20T00:16:14.553 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:14.532+0000 7fc742ffd700 1 -- 172.21.15.132:0/2225939750 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc748013380 con 0x7fc74c0f62a0 2024-02-20T00:16:14.553 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:14.532+0000 7fc742ffd700 1 -- 172.21.15.132:0/2225939750 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc748014e20 con 0x7fc74c0f62a0 2024-02-20T00:16:14.554 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:14.532+0000 7fc7530ce700 1 -- 172.21.15.132:0/2225939750 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc74c06a450 con 0x7fc74c0f62a0 2024-02-20T00:16:14.554 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:14.533+0000 7fc742ffd700 1 -- 172.21.15.132:0/2225939750 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc7480263a0 con 0x7fc74c0f62a0 2024-02-20T00:16:14.554 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:14.533+0000 7fc742ffd700 1 --2- 172.21.15.132:0/2225939750 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fc73802ff10 0x7fc7380323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:16:14.554 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:14.533+0000 7fc7530ce700 1 -- 172.21.15.132:0/2225939750 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc74c059100 con 0x7fc74c0f62a0 2024-02-20T00:16:14.554 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:14.533+0000 7fc742ffd700 1 -- 172.21.15.132:0/2225939750 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc748041240 con 0x7fc74c0f62a0 2024-02-20T00:16:14.554 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:14.533+0000 7fc7518cb700 1 --2- 172.21.15.132:0/2225939750 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fc73802ff10 0x7fc7380323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:16:14.554 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:14.534+0000 7fc7518cb700 1 --2- 172.21.15.132:0/2225939750 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fc73802ff10 0x7fc7380323d0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7fc73c0096a0 tx=0x7fc73c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:16:14.554 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:14.538+0000 7fc742ffd700 1 -- 172.21.15.132:0/2225939750 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc74801cb10 con 0x7fc74c0f62a0 2024-02-20T00:16:14.745 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:14.743+0000 7fc7530ce700 1 -- 172.21.15.132:0/2225939750 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc74c0fbe30 con 0x7fc74c0f62a0 2024-02-20T00:16:14.746 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:14.743+0000 7fc742ffd700 1 -- 172.21.15.132:0/2225939750 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc74801cb10 con 0x7fc74c0f62a0 2024-02-20T00:16:14.746 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:16:14.746 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:16:14.747 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:14.745+0000 7fc7530ce700 1 -- 172.21.15.132:0/2225939750 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fc73802ff10 msgr2=0x7fc7380323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:16:14.748 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:14.745+0000 7fc7530ce700 1 --2- 172.21.15.132:0/2225939750 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fc73802ff10 0x7fc7380323d0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7fc73c0096a0 tx=0x7fc73c006b40).stop 2024-02-20T00:16:14.748 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:14.745+0000 7fc7530ce700 1 -- 172.21.15.132:0/2225939750 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc74c0f62a0 msgr2=0x7fc74c06b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:16:14.748 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:14.745+0000 7fc7530ce700 1 --2- 172.21.15.132:0/2225939750 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc74c0f62a0 0x7fc74c06b9a0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0x7fc7480089f0 tx=0x7fc74800b040).stop 2024-02-20T00:16:14.748 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:14.745+0000 7fc7530ce700 1 -- 172.21.15.132:0/2225939750 shutdown_connections 2024-02-20T00:16:14.748 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:14.745+0000 7fc7530ce700 1 --2- 172.21.15.132:0/2225939750 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fc73802ff10 0x7fc7380323d0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:16:14.748 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:14.745+0000 7fc7530ce700 1 --2- 172.21.15.132:0/2225939750 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc74c0f62a0 0x7fc74c06b9a0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:16:14.748 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:14.745+0000 7fc7530ce700 1 -- 172.21.15.132:0/2225939750 >> 172.21.15.132:0/2225939750 conn(0x7fc74c0f1aa0 msgr2=0x7fc74c0f2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:16:14.748 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:14.745+0000 7fc7530ce700 1 -- 172.21.15.132:0/2225939750 shutdown_connections 2024-02-20T00:16:14.748 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:14.745+0000 7fc7530ce700 1 -- 172.21.15.132:0/2225939750 wait complete. 2024-02-20T00:16:14.749 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:16:15.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:16:14 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:16:14.745087+0000 mon.smithi084 (mon.0) 230 : audit [DBG] from='client.? 172.21.15.132:0/2225939750' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:16:16.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:16:15 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:16:14.894901+0000 mgr.smithi084.bmqrnf (mgr.14162) 195 : cluster [DBG] pgmap v183: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:16:16.420 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:16:16.420 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:16:18.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:16:17 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:16:16.895254+0000 mgr.smithi084.bmqrnf (mgr.14162) 196 : cluster [DBG] pgmap v184: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:16:18.363 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:18.359+0000 7f8a8ca67700 1 -- 172.21.15.132:0/3601509070 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8a88060280 msgr2=0x7f8a880f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:16:18.363 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:18.359+0000 7f8a8ca67700 1 --2- 172.21.15.132:0/3601509070 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8a88060280 0x7f8a880f90f0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 rx=0x7f8a78009fb0 tx=0x7f8a78009b30).stop 2024-02-20T00:16:18.363 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:18.359+0000 7f8a8ca67700 1 -- 172.21.15.132:0/3601509070 shutdown_connections 2024-02-20T00:16:18.363 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:18.359+0000 7f8a8ca67700 1 --2- 172.21.15.132:0/3601509070 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8a88060280 0x7f8a880f90f0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:16:18.363 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:18.359+0000 7f8a8ca67700 1 -- 172.21.15.132:0/3601509070 >> 172.21.15.132:0/3601509070 conn(0x7f8a880f0b70 msgr2=0x7f8a880f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:16:18.363 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:18.359+0000 7f8a8ca67700 1 -- 172.21.15.132:0/3601509070 shutdown_connections 2024-02-20T00:16:18.364 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:18.359+0000 7f8a8ca67700 1 -- 172.21.15.132:0/3601509070 wait complete. 2024-02-20T00:16:18.364 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:18.360+0000 7f8a8ca67700 1 Processor -- start 2024-02-20T00:16:18.364 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:18.360+0000 7f8a8ca67700 1 -- start start 2024-02-20T00:16:18.364 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:18.360+0000 7f8a8ca67700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8a88060280 0x7f8a88106780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:16:18.364 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:18.360+0000 7f8a8ca67700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a88106c70 con 0x7f8a88060280 2024-02-20T00:16:18.364 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:18.360+0000 7f8a8659c700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8a88060280 0x7f8a88106780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:16:18.364 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:18.360+0000 7f8a8659c700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8a88060280 0x7f8a88106780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:48876/0 (socket says 172.21.15.132:48876) 2024-02-20T00:16:18.364 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:18.360+0000 7f8a8659c700 1 -- 172.21.15.132:0/1914706394 learned_addr learned my addr 172.21.15.132:0/1914706394 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:16:18.364 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:18.360+0000 7f8a8659c700 1 -- 172.21.15.132:0/1914706394 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a78005040 con 0x7f8a88060280 2024-02-20T00:16:18.364 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:18.361+0000 7f8a8659c700 1 --2- 172.21.15.132:0/1914706394 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8a88060280 0x7f8a88106780 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7f8a780089f0 tx=0x7f8a7800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:16:18.364 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:18.361+0000 7f8a7effd700 1 -- 172.21.15.132:0/1914706394 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8a78009320 con 0x7f8a88060280 2024-02-20T00:16:18.365 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:18.361+0000 7f8a8ca67700 1 -- 172.21.15.132:0/1914706394 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a88106e70 con 0x7f8a88060280 2024-02-20T00:16:18.365 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:18.361+0000 7f8a7effd700 1 -- 172.21.15.132:0/1914706394 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8a78013380 con 0x7f8a88060280 2024-02-20T00:16:18.365 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:18.361+0000 7f8a7effd700 1 -- 172.21.15.132:0/1914706394 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8a78014e20 con 0x7f8a88060280 2024-02-20T00:16:18.365 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:18.361+0000 7f8a8ca67700 1 -- 172.21.15.132:0/1914706394 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a88107300 con 0x7f8a88060280 2024-02-20T00:16:18.366 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:18.362+0000 7f8a7effd700 1 -- 172.21.15.132:0/1914706394 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8a780095f0 con 0x7f8a88060280 2024-02-20T00:16:18.366 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:18.362+0000 7f8a8ca67700 1 -- 172.21.15.132:0/1914706394 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a88045ad0 con 0x7f8a88060280 2024-02-20T00:16:18.366 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:18.362+0000 7f8a7effd700 1 --2- 172.21.15.132:0/1914706394 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f8a7402ff10 0x7f8a740323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:16:18.366 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:18.362+0000 7f8a7effd700 1 -- 172.21.15.132:0/1914706394 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8a78040e90 con 0x7f8a88060280 2024-02-20T00:16:18.366 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:18.362+0000 7f8a85d9b700 1 --2- 172.21.15.132:0/1914706394 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f8a7402ff10 0x7f8a740323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:16:18.366 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:18.363+0000 7f8a85d9b700 1 --2- 172.21.15.132:0/1914706394 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f8a7402ff10 0x7f8a740323d0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f8a700096a0 tx=0x7f8a70006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:16:18.369 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:18.366+0000 7f8a7effd700 1 -- 172.21.15.132:0/1914706394 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8a78014420 con 0x7f8a88060280 2024-02-20T00:16:18.575 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:18.572+0000 7f8a8ca67700 1 -- 172.21.15.132:0/1914706394 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8a881081b0 con 0x7f8a88060280 2024-02-20T00:16:18.575 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:18.573+0000 7f8a7effd700 1 -- 172.21.15.132:0/1914706394 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8a7801d980 con 0x7f8a88060280 2024-02-20T00:16:18.575 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:16:18.575 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:16:18.577 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:18.574+0000 7f8a8ca67700 1 -- 172.21.15.132:0/1914706394 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f8a7402ff10 msgr2=0x7f8a740323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:16:18.577 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:18.574+0000 7f8a8ca67700 1 --2- 172.21.15.132:0/1914706394 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f8a7402ff10 0x7f8a740323d0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f8a700096a0 tx=0x7f8a70006b40).stop 2024-02-20T00:16:18.577 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:18.574+0000 7f8a8ca67700 1 -- 172.21.15.132:0/1914706394 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8a88060280 msgr2=0x7f8a88106780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:16:18.577 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:18.574+0000 7f8a8ca67700 1 --2- 172.21.15.132:0/1914706394 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8a88060280 0x7f8a88106780 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7f8a780089f0 tx=0x7f8a7800b040).stop 2024-02-20T00:16:18.577 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:18.574+0000 7f8a8ca67700 1 -- 172.21.15.132:0/1914706394 shutdown_connections 2024-02-20T00:16:18.578 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:18.574+0000 7f8a8ca67700 1 --2- 172.21.15.132:0/1914706394 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f8a7402ff10 0x7f8a740323d0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:16:18.578 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:18.574+0000 7f8a8ca67700 1 --2- 172.21.15.132:0/1914706394 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8a88060280 0x7f8a88106780 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:16:18.578 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:18.574+0000 7f8a8ca67700 1 -- 172.21.15.132:0/1914706394 >> 172.21.15.132:0/1914706394 conn(0x7f8a880f0b70 msgr2=0x7f8a880f15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:16:18.578 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:18.574+0000 7f8a8ca67700 1 -- 172.21.15.132:0/1914706394 shutdown_connections 2024-02-20T00:16:18.578 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:18.575+0000 7f8a8ca67700 1 -- 172.21.15.132:0/1914706394 wait complete. 2024-02-20T00:16:18.578 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:16:19.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:16:18 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:16:18.574432+0000 mon.smithi084 (mon.0) 231 : audit [DBG] from='client.? 172.21.15.132:0/1914706394' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:16:20.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:16:19 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:16:18.895544+0000 mgr.smithi084.bmqrnf (mgr.14162) 197 : cluster [DBG] pgmap v185: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:16:20.378 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:16:20.378 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:16:22.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:16:21 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:16:20.895792+0000 mgr.smithi084.bmqrnf (mgr.14162) 198 : cluster [DBG] pgmap v186: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:16:22.414 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:22.410+0000 7faff1244700 1 -- 172.21.15.132:0/615283916 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fafec060220 msgr2=0x7fafec0f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:16:22.414 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:22.410+0000 7faff1244700 1 --2- 172.21.15.132:0/615283916 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fafec060220 0x7fafec0f9150 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 rx=0x7fafdc009fb0 tx=0x7fafdc009b30).stop 2024-02-20T00:16:22.414 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:22.410+0000 7faff1244700 1 -- 172.21.15.132:0/615283916 shutdown_connections 2024-02-20T00:16:22.414 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:22.410+0000 7faff1244700 1 --2- 172.21.15.132:0/615283916 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fafec060220 0x7fafec0f9150 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:16:22.414 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:22.410+0000 7faff1244700 1 -- 172.21.15.132:0/615283916 >> 172.21.15.132:0/615283916 conn(0x7fafec0f0b70 msgr2=0x7fafec0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:16:22.414 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:22.410+0000 7faff1244700 1 -- 172.21.15.132:0/615283916 shutdown_connections 2024-02-20T00:16:22.414 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:22.410+0000 7faff1244700 1 -- 172.21.15.132:0/615283916 wait complete. 2024-02-20T00:16:22.414 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:22.410+0000 7faff1244700 1 Processor -- start 2024-02-20T00:16:22.414 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:22.410+0000 7faff1244700 1 -- start start 2024-02-20T00:16:22.414 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:22.410+0000 7faff1244700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fafec060220 0x7fafec0f6bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:16:22.414 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:22.410+0000 7faff1244700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafec0f5050 con 0x7fafec060220 2024-02-20T00:16:22.414 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:22.411+0000 7fafead9d700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fafec060220 0x7fafec0f6bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:16:22.415 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:22.411+0000 7fafead9d700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fafec060220 0x7fafec0f6bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:48894/0 (socket says 172.21.15.132:48894) 2024-02-20T00:16:22.415 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:22.411+0000 7fafead9d700 1 -- 172.21.15.132:0/122341776 learned_addr learned my addr 172.21.15.132:0/122341776 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:16:22.415 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:22.411+0000 7fafead9d700 1 -- 172.21.15.132:0/122341776 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafdc005040 con 0x7fafec060220 2024-02-20T00:16:22.415 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:22.411+0000 7fafead9d700 1 --2- 172.21.15.132:0/122341776 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fafec060220 0x7fafec0f6bd0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7fafdc0089f0 tx=0x7fafdc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:16:22.415 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:22.411+0000 7fafd37fe700 1 -- 172.21.15.132:0/122341776 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fafdc009320 con 0x7fafec060220 2024-02-20T00:16:22.415 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:22.411+0000 7faff1244700 1 -- 172.21.15.132:0/122341776 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafec0f51f0 con 0x7fafec060220 2024-02-20T00:16:22.415 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:22.411+0000 7fafd37fe700 1 -- 172.21.15.132:0/122341776 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fafdc013380 con 0x7fafec060220 2024-02-20T00:16:22.415 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:22.411+0000 7fafd37fe700 1 -- 172.21.15.132:0/122341776 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fafdc014e20 con 0x7fafec060220 2024-02-20T00:16:22.415 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:22.411+0000 7faff1244700 1 -- 172.21.15.132:0/122341776 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafec0f5680 con 0x7fafec060220 2024-02-20T00:16:22.416 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:22.412+0000 7fafd37fe700 1 -- 172.21.15.132:0/122341776 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fafdc014420 con 0x7fafec060220 2024-02-20T00:16:22.416 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:22.412+0000 7faff1244700 1 -- 172.21.15.132:0/122341776 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafec059100 con 0x7fafec060220 2024-02-20T00:16:22.416 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:22.412+0000 7fafd37fe700 1 --2- 172.21.15.132:0/122341776 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fafd402ff10 0x7fafd40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:16:22.417 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:22.412+0000 7fafd37fe700 1 -- 172.21.15.132:0/122341776 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fafdc040fe0 con 0x7fafec060220 2024-02-20T00:16:22.417 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:22.412+0000 7fafea59c700 1 --2- 172.21.15.132:0/122341776 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fafd402ff10 0x7fafd40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:16:22.417 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:22.413+0000 7fafea59c700 1 --2- 172.21.15.132:0/122341776 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fafd402ff10 0x7fafd40323d0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7fafe00096a0 tx=0x7fafe0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:16:22.419 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:22.417+0000 7fafd37fe700 1 -- 172.21.15.132:0/122341776 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fafdc06d2b0 con 0x7fafec060220 2024-02-20T00:16:22.628 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:22.626+0000 7faff1244700 1 -- 172.21.15.132:0/122341776 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fafec0fc090 con 0x7fafec060220 2024-02-20T00:16:22.629 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:22.626+0000 7fafd37fe700 1 -- 172.21.15.132:0/122341776 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fafdc0095f0 con 0x7fafec060220 2024-02-20T00:16:22.629 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:16:22.629 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:16:22.630 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:22.628+0000 7faff1244700 1 -- 172.21.15.132:0/122341776 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fafd402ff10 msgr2=0x7fafd40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:16:22.631 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:22.628+0000 7faff1244700 1 --2- 172.21.15.132:0/122341776 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fafd402ff10 0x7fafd40323d0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7fafe00096a0 tx=0x7fafe0006b40).stop 2024-02-20T00:16:22.631 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:22.628+0000 7faff1244700 1 -- 172.21.15.132:0/122341776 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fafec060220 msgr2=0x7fafec0f6bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:16:22.631 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:22.628+0000 7faff1244700 1 --2- 172.21.15.132:0/122341776 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fafec060220 0x7fafec0f6bd0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7fafdc0089f0 tx=0x7fafdc00b040).stop 2024-02-20T00:16:22.631 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:22.628+0000 7faff1244700 1 -- 172.21.15.132:0/122341776 shutdown_connections 2024-02-20T00:16:22.631 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:22.628+0000 7faff1244700 1 --2- 172.21.15.132:0/122341776 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fafd402ff10 0x7fafd40323d0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:16:22.631 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:22.628+0000 7faff1244700 1 --2- 172.21.15.132:0/122341776 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fafec060220 0x7fafec0f6bd0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:16:22.631 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:22.628+0000 7faff1244700 1 -- 172.21.15.132:0/122341776 >> 172.21.15.132:0/122341776 conn(0x7fafec0f0b70 msgr2=0x7fafec0f15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:16:22.631 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:22.628+0000 7faff1244700 1 -- 172.21.15.132:0/122341776 shutdown_connections 2024-02-20T00:16:22.631 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:22.628+0000 7faff1244700 1 -- 172.21.15.132:0/122341776 wait complete. 2024-02-20T00:16:22.631 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:16:23.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:16:22 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:16:22.627958+0000 mon.smithi084 (mon.0) 232 : audit [DBG] from='client.? 172.21.15.132:0/122341776' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:16:24.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:16:23 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:16:22.896069+0000 mgr.smithi084.bmqrnf (mgr.14162) 199 : cluster [DBG] pgmap v187: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:16:24.360 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:16:24.360 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:16:26.329 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:26.325+0000 7f2b99faa700 1 -- 172.21.15.132:0/2628072490 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2b940f5410 msgr2=0x7f2b940f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:16:26.329 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:26.325+0000 7f2b99faa700 1 --2- 172.21.15.132:0/2628072490 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2b940f5410 0x7f2b940f5830 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 rx=0x7f2b84009fb0 tx=0x7f2b84009b30).stop 2024-02-20T00:16:26.330 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:26.325+0000 7f2b99faa700 1 -- 172.21.15.132:0/2628072490 shutdown_connections 2024-02-20T00:16:26.330 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:26.325+0000 7f2b99faa700 1 --2- 172.21.15.132:0/2628072490 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2b940f5410 0x7f2b940f5830 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:16:26.330 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:26.325+0000 7f2b99faa700 1 -- 172.21.15.132:0/2628072490 >> 172.21.15.132:0/2628072490 conn(0x7f2b940f0b70 msgr2=0x7f2b940f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:16:26.330 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:26.325+0000 7f2b99faa700 1 -- 172.21.15.132:0/2628072490 shutdown_connections 2024-02-20T00:16:26.330 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:26.325+0000 7f2b99faa700 1 -- 172.21.15.132:0/2628072490 wait complete. 2024-02-20T00:16:26.330 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:26.325+0000 7f2b99faa700 1 Processor -- start 2024-02-20T00:16:26.331 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:26.326+0000 7f2b99faa700 1 -- start start 2024-02-20T00:16:26.331 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:26.326+0000 7f2b99faa700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2b940f5410 0x7f2b941094e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:16:26.331 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:26.326+0000 7f2b99faa700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b94060240 con 0x7f2b940f5410 2024-02-20T00:16:26.331 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:26.326+0000 7f2b937fe700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2b940f5410 0x7f2b941094e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:16:26.331 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:26.326+0000 7f2b937fe700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2b940f5410 0x7f2b941094e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:48906/0 (socket says 172.21.15.132:48906) 2024-02-20T00:16:26.331 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:26.326+0000 7f2b937fe700 1 -- 172.21.15.132:0/2773555182 learned_addr learned my addr 172.21.15.132:0/2773555182 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:16:26.332 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:26.326+0000 7f2b937fe700 1 -- 172.21.15.132:0/2773555182 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b84005040 con 0x7f2b940f5410 2024-02-20T00:16:26.332 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:26.327+0000 7f2b937fe700 1 --2- 172.21.15.132:0/2773555182 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2b940f5410 0x7f2b941094e0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7f2b84005010 tx=0x7f2b8400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:16:26.332 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:26.327+0000 7f2b7bfff700 1 -- 172.21.15.132:0/2773555182 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2b84009320 con 0x7f2b940f5410 2024-02-20T00:16:26.332 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:26.327+0000 7f2b7bfff700 1 -- 172.21.15.132:0/2773555182 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2b84013380 con 0x7f2b940f5410 2024-02-20T00:16:26.332 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:26.327+0000 7f2b99faa700 1 -- 172.21.15.132:0/2773555182 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b94060440 con 0x7f2b940f5410 2024-02-20T00:16:26.332 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:26.327+0000 7f2b7bfff700 1 -- 172.21.15.132:0/2773555182 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2b84014e50 con 0x7f2b940f5410 2024-02-20T00:16:26.332 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:26.327+0000 7f2b99faa700 1 -- 172.21.15.132:0/2773555182 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b94109b90 con 0x7f2b940f5410 2024-02-20T00:16:26.334 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:26.328+0000 7f2b7bfff700 1 -- 172.21.15.132:0/2773555182 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2b840095f0 con 0x7f2b940f5410 2024-02-20T00:16:26.334 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:26.328+0000 7f2b99faa700 1 -- 172.21.15.132:0/2773555182 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b94045ad0 con 0x7f2b940f5410 2024-02-20T00:16:26.334 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:26.328+0000 7f2b7bfff700 1 --2- 172.21.15.132:0/2773555182 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f2b7c02ff10 0x7f2b7c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:16:26.334 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:26.328+0000 7f2b7bfff700 1 -- 172.21.15.132:0/2773555182 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2b84040e70 con 0x7f2b940f5410 2024-02-20T00:16:26.334 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:26.328+0000 7f2b92ffd700 1 --2- 172.21.15.132:0/2773555182 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f2b7c02ff10 0x7f2b7c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:16:26.335 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:26.328+0000 7f2b92ffd700 1 --2- 172.21.15.132:0/2773555182 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f2b7c02ff10 0x7f2b7c0323d0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7f2b880096a0 tx=0x7f2b88006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:16:26.335 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:26.333+0000 7f2b7bfff700 1 -- 172.21.15.132:0/2773555182 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2b84014420 con 0x7f2b940f5410 2024-02-20T00:16:26.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:16:25 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:16:24.896404+0000 mgr.smithi084.bmqrnf (mgr.14162) 200 : cluster [DBG] pgmap v188: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:16:26.538 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:26.536+0000 7f2b99faa700 1 -- 172.21.15.132:0/2773555182 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2b9410a3b0 con 0x7f2b940f5410 2024-02-20T00:16:26.539 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:26.536+0000 7f2b7bfff700 1 -- 172.21.15.132:0/2773555182 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2b84010230 con 0x7f2b940f5410 2024-02-20T00:16:26.539 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:16:26.539 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:16:26.540 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:26.538+0000 7f2b99faa700 1 -- 172.21.15.132:0/2773555182 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f2b7c02ff10 msgr2=0x7f2b7c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:16:26.541 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:26.538+0000 7f2b99faa700 1 --2- 172.21.15.132:0/2773555182 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f2b7c02ff10 0x7f2b7c0323d0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7f2b880096a0 tx=0x7f2b88006b40).stop 2024-02-20T00:16:26.541 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:26.538+0000 7f2b99faa700 1 -- 172.21.15.132:0/2773555182 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2b940f5410 msgr2=0x7f2b941094e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:16:26.541 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:26.538+0000 7f2b99faa700 1 --2- 172.21.15.132:0/2773555182 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2b940f5410 0x7f2b941094e0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7f2b84005010 tx=0x7f2b8400b040).stop 2024-02-20T00:16:26.541 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:26.538+0000 7f2b99faa700 1 -- 172.21.15.132:0/2773555182 shutdown_connections 2024-02-20T00:16:26.541 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:26.538+0000 7f2b99faa700 1 --2- 172.21.15.132:0/2773555182 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f2b7c02ff10 0x7f2b7c0323d0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:16:26.541 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:26.538+0000 7f2b99faa700 1 --2- 172.21.15.132:0/2773555182 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2b940f5410 0x7f2b941094e0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:16:26.541 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:26.538+0000 7f2b99faa700 1 -- 172.21.15.132:0/2773555182 >> 172.21.15.132:0/2773555182 conn(0x7f2b940f0b70 msgr2=0x7f2b940f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:16:26.541 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:26.538+0000 7f2b99faa700 1 -- 172.21.15.132:0/2773555182 shutdown_connections 2024-02-20T00:16:26.541 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:26.538+0000 7f2b99faa700 1 -- 172.21.15.132:0/2773555182 wait complete. 2024-02-20T00:16:26.542 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:16:27.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:16:26 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:16:26.537824+0000 mon.smithi084 (mon.0) 233 : audit [DBG] from='client.? 172.21.15.132:0/2773555182' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:16:28.254 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:16:28.254 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:16:28.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:16:27 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:16:26.896779+0000 mgr.smithi084.bmqrnf (mgr.14162) 201 : cluster [DBG] pgmap v189: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:16:30.214 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:30.210+0000 7fbd3dd82700 1 -- 172.21.15.132:0/3710285637 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbd38060220 msgr2=0x7fbd380fb550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:16:30.215 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:30.210+0000 7fbd3dd82700 1 --2- 172.21.15.132:0/3710285637 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbd38060220 0x7fbd380fb550 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 rx=0x7fbd28009fb0 tx=0x7fbd28009b30).stop 2024-02-20T00:16:30.215 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:30.211+0000 7fbd3dd82700 1 -- 172.21.15.132:0/3710285637 shutdown_connections 2024-02-20T00:16:30.215 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:30.211+0000 7fbd3dd82700 1 --2- 172.21.15.132:0/3710285637 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbd38060220 0x7fbd380fb550 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:16:30.215 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:30.211+0000 7fbd3dd82700 1 -- 172.21.15.132:0/3710285637 >> 172.21.15.132:0/3710285637 conn(0x7fbd380f0b70 msgr2=0x7fbd380f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:16:30.215 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:30.211+0000 7fbd3dd82700 1 -- 172.21.15.132:0/3710285637 shutdown_connections 2024-02-20T00:16:30.215 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:30.211+0000 7fbd3dd82700 1 -- 172.21.15.132:0/3710285637 wait complete. 2024-02-20T00:16:30.215 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:30.211+0000 7fbd3dd82700 1 Processor -- start 2024-02-20T00:16:30.215 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:30.211+0000 7fbd3dd82700 1 -- start start 2024-02-20T00:16:30.215 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:30.211+0000 7fbd3dd82700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbd38060220 0x7fbd381089a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:16:30.216 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:30.211+0000 7fbd3dd82700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd38108e90 con 0x7fbd38060220 2024-02-20T00:16:30.216 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:30.212+0000 7fbd377fe700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbd38060220 0x7fbd381089a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:16:30.216 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:30.212+0000 7fbd377fe700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbd38060220 0x7fbd381089a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:33950/0 (socket says 172.21.15.132:33950) 2024-02-20T00:16:30.216 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:30.212+0000 7fbd377fe700 1 -- 172.21.15.132:0/3966207450 learned_addr learned my addr 172.21.15.132:0/3966207450 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:16:30.216 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:30.212+0000 7fbd377fe700 1 -- 172.21.15.132:0/3966207450 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd28005040 con 0x7fbd38060220 2024-02-20T00:16:30.216 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:30.212+0000 7fbd377fe700 1 --2- 172.21.15.132:0/3966207450 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbd38060220 0x7fbd381089a0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0x7fbd28009830 tx=0x7fbd2800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:16:30.216 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:30.212+0000 7fbd1ffff700 1 -- 172.21.15.132:0/3966207450 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbd28009320 con 0x7fbd38060220 2024-02-20T00:16:30.216 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:30.212+0000 7fbd3dd82700 1 -- 172.21.15.132:0/3966207450 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd38109090 con 0x7fbd38060220 2024-02-20T00:16:30.216 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:30.212+0000 7fbd1ffff700 1 -- 172.21.15.132:0/3966207450 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbd28013380 con 0x7fbd38060220 2024-02-20T00:16:30.216 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:30.212+0000 7fbd1ffff700 1 -- 172.21.15.132:0/3966207450 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbd28014420 con 0x7fbd38060220 2024-02-20T00:16:30.216 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:30.212+0000 7fbd3dd82700 1 -- 172.21.15.132:0/3966207450 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd38109520 con 0x7fbd38060220 2024-02-20T00:16:30.217 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:30.213+0000 7fbd1ffff700 1 -- 172.21.15.132:0/3966207450 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbd28014580 con 0x7fbd38060220 2024-02-20T00:16:30.217 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:30.213+0000 7fbd3dd82700 1 -- 172.21.15.132:0/3966207450 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd38045ad0 con 0x7fbd38060220 2024-02-20T00:16:30.217 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:30.213+0000 7fbd1ffff700 1 --2- 172.21.15.132:0/3966207450 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fbd2002ff60 0x7fbd20032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:16:30.218 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:30.213+0000 7fbd1ffff700 1 -- 172.21.15.132:0/3966207450 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbd28040dd0 con 0x7fbd38060220 2024-02-20T00:16:30.218 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:30.213+0000 7fbd36ffd700 1 --2- 172.21.15.132:0/3966207450 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fbd2002ff60 0x7fbd20032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:16:30.218 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:30.214+0000 7fbd36ffd700 1 --2- 172.21.15.132:0/3966207450 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fbd2002ff60 0x7fbd20032420 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7fbd2c0096a0 tx=0x7fbd2c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:16:30.220 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:30.218+0000 7fbd1ffff700 1 -- 172.21.15.132:0/3966207450 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbd280273f0 con 0x7fbd38060220 2024-02-20T00:16:30.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:16:29 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:16:28.897124+0000 mgr.smithi084.bmqrnf (mgr.14162) 202 : cluster [DBG] pgmap v190: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:16:30.425 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:30.423+0000 7fbd3dd82700 1 -- 172.21.15.132:0/3966207450 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbd380fe290 con 0x7fbd38060220 2024-02-20T00:16:30.425 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:30.423+0000 7fbd1ffff700 1 -- 172.21.15.132:0/3966207450 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fbd2802c2d0 con 0x7fbd38060220 2024-02-20T00:16:30.426 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:16:30.426 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:16:30.427 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:30.425+0000 7fbd3dd82700 1 -- 172.21.15.132:0/3966207450 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fbd2002ff60 msgr2=0x7fbd20032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:16:30.427 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:30.425+0000 7fbd3dd82700 1 --2- 172.21.15.132:0/3966207450 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fbd2002ff60 0x7fbd20032420 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7fbd2c0096a0 tx=0x7fbd2c006b40).stop 2024-02-20T00:16:30.427 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:30.425+0000 7fbd3dd82700 1 -- 172.21.15.132:0/3966207450 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbd38060220 msgr2=0x7fbd381089a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:16:30.427 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:30.425+0000 7fbd3dd82700 1 --2- 172.21.15.132:0/3966207450 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbd38060220 0x7fbd381089a0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0x7fbd28009830 tx=0x7fbd2800b040).stop 2024-02-20T00:16:30.427 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:30.425+0000 7fbd3dd82700 1 -- 172.21.15.132:0/3966207450 shutdown_connections 2024-02-20T00:16:30.428 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:30.425+0000 7fbd3dd82700 1 --2- 172.21.15.132:0/3966207450 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fbd2002ff60 0x7fbd20032420 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:16:30.428 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:30.425+0000 7fbd3dd82700 1 --2- 172.21.15.132:0/3966207450 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbd38060220 0x7fbd381089a0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:16:30.428 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:30.425+0000 7fbd3dd82700 1 -- 172.21.15.132:0/3966207450 >> 172.21.15.132:0/3966207450 conn(0x7fbd380f0b70 msgr2=0x7fbd380f1600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:16:30.428 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:30.425+0000 7fbd3dd82700 1 -- 172.21.15.132:0/3966207450 shutdown_connections 2024-02-20T00:16:30.428 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:30.425+0000 7fbd3dd82700 1 -- 172.21.15.132:0/3966207450 wait complete. 2024-02-20T00:16:30.428 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:16:31.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:16:30 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:16:30.424893+0000 mon.smithi084 (mon.0) 234 : audit [DBG] from='client.? 172.21.15.132:0/3966207450' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:16:32.151 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:16:32.151 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:16:32.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:16:31 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:16:30.897475+0000 mgr.smithi084.bmqrnf (mgr.14162) 203 : cluster [DBG] pgmap v191: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:16:34.050 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:34.045+0000 7ff86aaec700 1 -- 172.21.15.132:0/2512875070 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff8640f5410 msgr2=0x7ff8640f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:16:34.050 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:34.045+0000 7ff86aaec700 1 --2- 172.21.15.132:0/2512875070 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff8640f5410 0x7ff8640f5830 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 rx=0x7ff858009fb0 tx=0x7ff858009b30).stop 2024-02-20T00:16:34.050 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:34.045+0000 7ff86aaec700 1 -- 172.21.15.132:0/2512875070 shutdown_connections 2024-02-20T00:16:34.050 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:34.045+0000 7ff86aaec700 1 --2- 172.21.15.132:0/2512875070 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff8640f5410 0x7ff8640f5830 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:16:34.050 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:34.045+0000 7ff86aaec700 1 -- 172.21.15.132:0/2512875070 >> 172.21.15.132:0/2512875070 conn(0x7ff8640f0b70 msgr2=0x7ff8640f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:16:34.050 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:34.045+0000 7ff86aaec700 1 -- 172.21.15.132:0/2512875070 shutdown_connections 2024-02-20T00:16:34.050 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:34.045+0000 7ff86aaec700 1 -- 172.21.15.132:0/2512875070 wait complete. 2024-02-20T00:16:34.050 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:34.046+0000 7ff86aaec700 1 Processor -- start 2024-02-20T00:16:34.050 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:34.046+0000 7ff86aaec700 1 -- start start 2024-02-20T00:16:34.050 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:34.046+0000 7ff86aaec700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff8640f5410 0x7ff8641067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:16:34.050 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:34.046+0000 7ff86aaec700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff864106cd0 con 0x7ff8640f5410 2024-02-20T00:16:34.051 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:34.046+0000 7ff868888700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff8640f5410 0x7ff8641067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:16:34.051 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:34.046+0000 7ff868888700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff8640f5410 0x7ff8641067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:33966/0 (socket says 172.21.15.132:33966) 2024-02-20T00:16:34.051 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:34.046+0000 7ff868888700 1 -- 172.21.15.132:0/793677201 learned_addr learned my addr 172.21.15.132:0/793677201 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:16:34.051 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:34.047+0000 7ff868888700 1 -- 172.21.15.132:0/793677201 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff858005040 con 0x7ff8640f5410 2024-02-20T00:16:34.051 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:34.047+0000 7ff868888700 1 --2- 172.21.15.132:0/793677201 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff8640f5410 0x7ff8641067e0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7ff8580089f0 tx=0x7ff85800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:16:34.051 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:34.047+0000 7ff8617fa700 1 -- 172.21.15.132:0/793677201 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff858009320 con 0x7ff8640f5410 2024-02-20T00:16:34.051 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:34.047+0000 7ff86aaec700 1 -- 172.21.15.132:0/793677201 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff864106ed0 con 0x7ff8640f5410 2024-02-20T00:16:34.051 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:34.047+0000 7ff8617fa700 1 -- 172.21.15.132:0/793677201 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff858013380 con 0x7ff8640f5410 2024-02-20T00:16:34.051 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:34.047+0000 7ff8617fa700 1 -- 172.21.15.132:0/793677201 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff85801ed80 con 0x7ff8640f5410 2024-02-20T00:16:34.051 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:34.047+0000 7ff86aaec700 1 -- 172.21.15.132:0/793677201 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff864107360 con 0x7ff8640f5410 2024-02-20T00:16:34.052 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:34.048+0000 7ff8617fa700 1 -- 172.21.15.132:0/793677201 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff85801e420 con 0x7ff8640f5410 2024-02-20T00:16:34.052 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:34.048+0000 7ff8617fa700 1 --2- 172.21.15.132:0/793677201 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7ff85402ff60 0x7ff854032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:16:34.052 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:34.048+0000 7ff8617fa700 1 -- 172.21.15.132:0/793677201 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff858041060 con 0x7ff8640f5410 2024-02-20T00:16:34.052 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:34.048+0000 7ff86aaec700 1 -- 172.21.15.132:0/793677201 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff864059100 con 0x7ff8640f5410 2024-02-20T00:16:34.052 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:34.048+0000 7ff863fff700 1 --2- 172.21.15.132:0/793677201 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7ff85402ff60 0x7ff854032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:16:34.052 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:34.049+0000 7ff863fff700 1 --2- 172.21.15.132:0/793677201 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7ff85402ff60 0x7ff854032420 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7ff8500096a0 tx=0x7ff850006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:16:34.055 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:34.053+0000 7ff8617fa700 1 -- 172.21.15.132:0/793677201 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff85801cb10 con 0x7ff8640f5410 2024-02-20T00:16:34.259 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:34.257+0000 7ff86aaec700 1 -- 172.21.15.132:0/793677201 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff8640fafa0 con 0x7ff8640f5410 2024-02-20T00:16:34.260 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:34.257+0000 7ff8617fa700 1 -- 172.21.15.132:0/793677201 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff85801cb10 con 0x7ff8640f5410 2024-02-20T00:16:34.260 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:16:34.260 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:16:34.261 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:34.259+0000 7ff86aaec700 1 -- 172.21.15.132:0/793677201 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7ff85402ff60 msgr2=0x7ff854032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:16:34.262 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:34.259+0000 7ff86aaec700 1 --2- 172.21.15.132:0/793677201 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7ff85402ff60 0x7ff854032420 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7ff8500096a0 tx=0x7ff850006b40).stop 2024-02-20T00:16:34.262 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:34.259+0000 7ff86aaec700 1 -- 172.21.15.132:0/793677201 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff8640f5410 msgr2=0x7ff8641067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:16:34.262 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:34.259+0000 7ff86aaec700 1 --2- 172.21.15.132:0/793677201 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff8640f5410 0x7ff8641067e0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7ff8580089f0 tx=0x7ff85800b040).stop 2024-02-20T00:16:34.262 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:34.259+0000 7ff86aaec700 1 -- 172.21.15.132:0/793677201 shutdown_connections 2024-02-20T00:16:34.262 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:34.259+0000 7ff86aaec700 1 --2- 172.21.15.132:0/793677201 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7ff85402ff60 0x7ff854032420 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:16:34.262 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:34.259+0000 7ff86aaec700 1 --2- 172.21.15.132:0/793677201 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff8640f5410 0x7ff8641067e0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:16:34.262 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:34.259+0000 7ff86aaec700 1 -- 172.21.15.132:0/793677201 >> 172.21.15.132:0/793677201 conn(0x7ff8640f0b70 msgr2=0x7ff8640f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:16:34.262 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:34.259+0000 7ff86aaec700 1 -- 172.21.15.132:0/793677201 shutdown_connections 2024-02-20T00:16:34.262 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:34.259+0000 7ff86aaec700 1 -- 172.21.15.132:0/793677201 wait complete. 2024-02-20T00:16:34.263 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:16:34.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:16:33 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:16:32.897878+0000 mgr.smithi084.bmqrnf (mgr.14162) 204 : cluster [DBG] pgmap v192: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:16:35.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:16:34 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:16:34.259063+0000 mon.smithi084 (mon.0) 235 : audit [DBG] from='client.? 172.21.15.132:0/793677201' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:16:36.023 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:16:36.024 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:16:36.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:16:35 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:16:34.898127+0000 mgr.smithi084.bmqrnf (mgr.14162) 205 : cluster [DBG] pgmap v193: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:16:38.081 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:38.078+0000 7f2741724700 1 -- 172.21.15.132:0/3576608071 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f273c0f6510 msgr2=0x7f273c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:16:38.081 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:38.078+0000 7f2741724700 1 --2- 172.21.15.132:0/3576608071 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f273c0f6510 0x7f273c0f6930 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 rx=0x7f272c009fb0 tx=0x7f272c009b30).stop 2024-02-20T00:16:38.081 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:38.078+0000 7f2741724700 1 -- 172.21.15.132:0/3576608071 shutdown_connections 2024-02-20T00:16:38.081 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:38.078+0000 7f2741724700 1 --2- 172.21.15.132:0/3576608071 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f273c0f6510 0x7f273c0f6930 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:16:38.081 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:38.078+0000 7f2741724700 1 -- 172.21.15.132:0/3576608071 >> 172.21.15.132:0/3576608071 conn(0x7f273c0f1c70 msgr2=0x7f273c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:16:38.081 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:38.078+0000 7f2741724700 1 -- 172.21.15.132:0/3576608071 shutdown_connections 2024-02-20T00:16:38.082 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:38.079+0000 7f2741724700 1 -- 172.21.15.132:0/3576608071 wait complete. 2024-02-20T00:16:38.082 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:38.079+0000 7f2741724700 1 Processor -- start 2024-02-20T00:16:38.082 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:38.079+0000 7f2741724700 1 -- start start 2024-02-20T00:16:38.082 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:38.080+0000 7f2741724700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f273c0f6510 0x7f273c1067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:16:38.082 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:38.080+0000 7f2741724700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f273c106cd0 con 0x7f273c0f6510 2024-02-20T00:16:38.082 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:38.080+0000 7f273affd700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f273c0f6510 0x7f273c1067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:16:38.082 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:38.080+0000 7f273affd700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f273c0f6510 0x7f273c1067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:34322/0 (socket says 172.21.15.132:34322) 2024-02-20T00:16:38.082 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:38.080+0000 7f273affd700 1 -- 172.21.15.132:0/1931727554 learned_addr learned my addr 172.21.15.132:0/1931727554 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:16:38.084 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:38.080+0000 7f273affd700 1 -- 172.21.15.132:0/1931727554 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f272c005040 con 0x7f273c0f6510 2024-02-20T00:16:38.084 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:38.080+0000 7f273affd700 1 --2- 172.21.15.132:0/1931727554 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f273c0f6510 0x7f273c1067e0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0x7f272c0089f0 tx=0x7f272c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:16:38.084 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:38.080+0000 7f27237fe700 1 -- 172.21.15.132:0/1931727554 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f272c009320 con 0x7f273c0f6510 2024-02-20T00:16:38.084 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:38.080+0000 7f2741724700 1 -- 172.21.15.132:0/1931727554 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f273c106ed0 con 0x7f273c0f6510 2024-02-20T00:16:38.084 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:38.080+0000 7f27237fe700 1 -- 172.21.15.132:0/1931727554 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f272c013380 con 0x7f273c0f6510 2024-02-20T00:16:38.084 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:38.080+0000 7f27237fe700 1 -- 172.21.15.132:0/1931727554 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f272c01ed80 con 0x7f273c0f6510 2024-02-20T00:16:38.084 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:38.081+0000 7f2741724700 1 -- 172.21.15.132:0/1931727554 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f273c107360 con 0x7f273c0f6510 2024-02-20T00:16:38.084 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:38.081+0000 7f27237fe700 1 -- 172.21.15.132:0/1931727554 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f272c01e420 con 0x7f273c0f6510 2024-02-20T00:16:38.084 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:38.081+0000 7f2741724700 1 -- 172.21.15.132:0/1931727554 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f273c045ad0 con 0x7f273c0f6510 2024-02-20T00:16:38.084 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:38.082+0000 7f27237fe700 1 --2- 172.21.15.132:0/1931727554 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f272402ff60 0x7f2724032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:16:38.085 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:38.082+0000 7f27237fe700 1 -- 172.21.15.132:0/1931727554 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f272c040e70 con 0x7f273c0f6510 2024-02-20T00:16:38.085 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:38.082+0000 7f273a7fc700 1 --2- 172.21.15.132:0/1931727554 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f272402ff60 0x7f2724032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:16:38.085 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:38.082+0000 7f273a7fc700 1 --2- 172.21.15.132:0/1931727554 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f272402ff60 0x7f2724032420 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f27300096a0 tx=0x7f2730006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:16:38.087 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:38.086+0000 7f27237fe700 1 -- 172.21.15.132:0/1931727554 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f272c00e2a0 con 0x7f273c0f6510 2024-02-20T00:16:38.297 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:38.295+0000 7f2741724700 1 -- 172.21.15.132:0/1931727554 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f273c108210 con 0x7f273c0f6510 2024-02-20T00:16:38.297 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:38.296+0000 7f27237fe700 1 -- 172.21.15.132:0/1931727554 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f272c00e2a0 con 0x7f273c0f6510 2024-02-20T00:16:38.298 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:16:38.298 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:16:38.299 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:38.298+0000 7f2741724700 1 -- 172.21.15.132:0/1931727554 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f272402ff60 msgr2=0x7f2724032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:16:38.299 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:38.298+0000 7f2741724700 1 --2- 172.21.15.132:0/1931727554 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f272402ff60 0x7f2724032420 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f27300096a0 tx=0x7f2730006b40).stop 2024-02-20T00:16:38.299 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:38.298+0000 7f2741724700 1 -- 172.21.15.132:0/1931727554 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f273c0f6510 msgr2=0x7f273c1067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:16:38.299 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:38.298+0000 7f2741724700 1 --2- 172.21.15.132:0/1931727554 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f273c0f6510 0x7f273c1067e0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0x7f272c0089f0 tx=0x7f272c00b040).stop 2024-02-20T00:16:38.299 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:38.298+0000 7f2741724700 1 -- 172.21.15.132:0/1931727554 shutdown_connections 2024-02-20T00:16:38.300 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:38.298+0000 7f2741724700 1 --2- 172.21.15.132:0/1931727554 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f272402ff60 0x7f2724032420 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:16:38.300 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:38.298+0000 7f2741724700 1 --2- 172.21.15.132:0/1931727554 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f273c0f6510 0x7f273c1067e0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:16:38.300 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:38.298+0000 7f2741724700 1 -- 172.21.15.132:0/1931727554 >> 172.21.15.132:0/1931727554 conn(0x7f273c0f1c70 msgr2=0x7f273c0f26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:16:38.300 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:38.298+0000 7f2741724700 1 -- 172.21.15.132:0/1931727554 shutdown_connections 2024-02-20T00:16:38.300 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:38.298+0000 7f2741724700 1 -- 172.21.15.132:0/1931727554 wait complete. 2024-02-20T00:16:38.300 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:16:38.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:16:37 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:16:36.898454+0000 mgr.smithi084.bmqrnf (mgr.14162) 206 : cluster [DBG] pgmap v194: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:16:39.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:16:38 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:16:38.296820+0000 mon.smithi084 (mon.0) 236 : audit [DBG] from='client.? 172.21.15.132:0/1931727554' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:16:40.007 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:16:40.007 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:16:40.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:16:39 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:16:38.898859+0000 mgr.smithi084.bmqrnf (mgr.14162) 207 : cluster [DBG] pgmap v195: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:16:41.847 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:41.845+0000 7f89b4bc4700 1 -- 172.21.15.132:0/33442866 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f89b00f5410 msgr2=0x7f89b00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:16:41.848 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:41.845+0000 7f89b4bc4700 1 --2- 172.21.15.132:0/33442866 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f89b00f5410 0x7f89b00f5830 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 rx=0x7f89a4009fb0 tx=0x7f89a4009b30).stop 2024-02-20T00:16:41.848 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:41.845+0000 7f89b4bc4700 1 -- 172.21.15.132:0/33442866 shutdown_connections 2024-02-20T00:16:41.848 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:41.845+0000 7f89b4bc4700 1 --2- 172.21.15.132:0/33442866 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f89b00f5410 0x7f89b00f5830 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:16:41.848 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:41.845+0000 7f89b4bc4700 1 -- 172.21.15.132:0/33442866 >> 172.21.15.132:0/33442866 conn(0x7f89b00f0b70 msgr2=0x7f89b00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:16:41.848 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:41.845+0000 7f89b4bc4700 1 -- 172.21.15.132:0/33442866 shutdown_connections 2024-02-20T00:16:41.848 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:41.845+0000 7f89b4bc4700 1 -- 172.21.15.132:0/33442866 wait complete. 2024-02-20T00:16:41.848 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:41.845+0000 7f89b4bc4700 1 Processor -- start 2024-02-20T00:16:41.848 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:41.845+0000 7f89b4bc4700 1 -- start start 2024-02-20T00:16:41.848 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:41.845+0000 7f89b4bc4700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f89b00f5410 0x7f89b00fd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:16:41.848 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:41.845+0000 7f89b4bc4700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89b00fb960 con 0x7f89b00f5410 2024-02-20T00:16:41.848 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:41.846+0000 7f89ae59c700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f89b00f5410 0x7f89b00fd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:16:41.848 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:41.846+0000 7f89ae59c700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f89b00f5410 0x7f89b00fd4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:34340/0 (socket says 172.21.15.132:34340) 2024-02-20T00:16:41.848 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:41.846+0000 7f89ae59c700 1 -- 172.21.15.132:0/3577985164 learned_addr learned my addr 172.21.15.132:0/3577985164 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:16:41.849 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:41.846+0000 7f89ae59c700 1 -- 172.21.15.132:0/3577985164 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89a4005040 con 0x7f89b00f5410 2024-02-20T00:16:41.849 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:41.846+0000 7f89ae59c700 1 --2- 172.21.15.132:0/3577985164 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f89b00f5410 0x7f89b00fd4e0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0x7f89a4005010 tx=0x7f89a400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:16:41.849 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:41.846+0000 7f899effd700 1 -- 172.21.15.132:0/3577985164 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f89a4009320 con 0x7f89b00f5410 2024-02-20T00:16:41.849 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:41.846+0000 7f89b4bc4700 1 -- 172.21.15.132:0/3577985164 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89b00fbb00 con 0x7f89b00f5410 2024-02-20T00:16:41.849 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:41.846+0000 7f899effd700 1 -- 172.21.15.132:0/3577985164 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f89a4013380 con 0x7f89b00f5410 2024-02-20T00:16:41.849 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:41.846+0000 7f89b4bc4700 1 -- 172.21.15.132:0/3577985164 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89b00fbf90 con 0x7f89b00f5410 2024-02-20T00:16:41.849 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:41.846+0000 7f899effd700 1 -- 172.21.15.132:0/3577985164 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f89a4014e50 con 0x7f89b00f5410 2024-02-20T00:16:41.850 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:41.847+0000 7f89b4bc4700 1 -- 172.21.15.132:0/3577985164 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89b0059100 con 0x7f89b00f5410 2024-02-20T00:16:41.850 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:41.847+0000 7f899effd700 1 -- 172.21.15.132:0/3577985164 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f89a40095f0 con 0x7f89b00f5410 2024-02-20T00:16:41.850 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:41.847+0000 7f899effd700 1 --2- 172.21.15.132:0/3577985164 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f899802ff10 0x7f89980323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:16:41.850 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:41.847+0000 7f899effd700 1 -- 172.21.15.132:0/3577985164 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f89a4040ff0 con 0x7f89b00f5410 2024-02-20T00:16:41.850 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:41.848+0000 7f89add9b700 1 --2- 172.21.15.132:0/3577985164 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f899802ff10 0x7f89980323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:16:41.850 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:41.848+0000 7f89add9b700 1 --2- 172.21.15.132:0/3577985164 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f899802ff10 0x7f89980323d0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7f89a00096a0 tx=0x7f89a0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:16:41.853 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:41.852+0000 7f899effd700 1 -- 172.21.15.132:0/3577985164 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f89a406d2b0 con 0x7f89b00f5410 2024-02-20T00:16:42.057 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:42.056+0000 7f89b4bc4700 1 -- 172.21.15.132:0/3577985164 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f89b0045ad0 con 0x7f89b00f5410 2024-02-20T00:16:42.058 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:42.057+0000 7f899effd700 1 -- 172.21.15.132:0/3577985164 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f89a4014420 con 0x7f89b00f5410 2024-02-20T00:16:42.058 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:16:42.058 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:16:42.060 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:42.058+0000 7f89b4bc4700 1 -- 172.21.15.132:0/3577985164 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f899802ff10 msgr2=0x7f89980323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:16:42.060 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:42.058+0000 7f89b4bc4700 1 --2- 172.21.15.132:0/3577985164 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f899802ff10 0x7f89980323d0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7f89a00096a0 tx=0x7f89a0006b40).stop 2024-02-20T00:16:42.060 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:42.058+0000 7f89b4bc4700 1 -- 172.21.15.132:0/3577985164 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f89b00f5410 msgr2=0x7f89b00fd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:16:42.060 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:42.058+0000 7f89b4bc4700 1 --2- 172.21.15.132:0/3577985164 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f89b00f5410 0x7f89b00fd4e0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0x7f89a4005010 tx=0x7f89a400b040).stop 2024-02-20T00:16:42.060 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:42.058+0000 7f89b4bc4700 1 -- 172.21.15.132:0/3577985164 shutdown_connections 2024-02-20T00:16:42.060 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:42.058+0000 7f89b4bc4700 1 --2- 172.21.15.132:0/3577985164 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f899802ff10 0x7f89980323d0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:16:42.060 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:42.058+0000 7f89b4bc4700 1 --2- 172.21.15.132:0/3577985164 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f89b00f5410 0x7f89b00fd4e0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:16:42.060 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:42.058+0000 7f89b4bc4700 1 -- 172.21.15.132:0/3577985164 >> 172.21.15.132:0/3577985164 conn(0x7f89b00f0b70 msgr2=0x7f89b005fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:16:42.060 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:42.058+0000 7f89b4bc4700 1 -- 172.21.15.132:0/3577985164 shutdown_connections 2024-02-20T00:16:42.060 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:42.058+0000 7f89b4bc4700 1 -- 172.21.15.132:0/3577985164 wait complete. 2024-02-20T00:16:42.061 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:16:42.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:16:41 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:16:40.899207+0000 mgr.smithi084.bmqrnf (mgr.14162) 208 : cluster [DBG] pgmap v196: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:16:43.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:16:42 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:16:42.057248+0000 mon.smithi084 (mon.0) 237 : audit [DBG] from='client.? 172.21.15.132:0/3577985164' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:16:43.710 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:16:43.710 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:16:44.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:16:43 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:16:42.899547+0000 mgr.smithi084.bmqrnf (mgr.14162) 209 : cluster [DBG] pgmap v197: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:16:45.663 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:45.661+0000 7f6cba8f6700 1 -- 172.21.15.132:0/799179279 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6cb40f51a0 msgr2=0x7f6cb40f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:16:45.663 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:45.661+0000 7f6cba8f6700 1 --2- 172.21.15.132:0/799179279 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6cb40f51a0 0x7f6cb40f55c0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 rx=0x7f6cb0009fb0 tx=0x7f6cb0009b30).stop 2024-02-20T00:16:45.663 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:45.661+0000 7f6cba8f6700 1 -- 172.21.15.132:0/799179279 shutdown_connections 2024-02-20T00:16:45.663 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:45.661+0000 7f6cba8f6700 1 --2- 172.21.15.132:0/799179279 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6cb40f51a0 0x7f6cb40f55c0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:16:45.663 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:45.661+0000 7f6cba8f6700 1 -- 172.21.15.132:0/799179279 >> 172.21.15.132:0/799179279 conn(0x7f6cb40f09a0 msgr2=0x7f6cb40f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:16:45.663 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:45.661+0000 7f6cba8f6700 1 -- 172.21.15.132:0/799179279 shutdown_connections 2024-02-20T00:16:45.663 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:45.661+0000 7f6cba8f6700 1 -- 172.21.15.132:0/799179279 wait complete. 2024-02-20T00:16:45.664 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:45.661+0000 7f6cba8f6700 1 Processor -- start 2024-02-20T00:16:45.664 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:45.661+0000 7f6cba8f6700 1 -- start start 2024-02-20T00:16:45.664 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:45.662+0000 7f6cba8f6700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6cb40f51a0 0x7f6cb4109290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:16:45.664 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:45.662+0000 7f6cba8f6700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cb4060280 con 0x7f6cb40f51a0 2024-02-20T00:16:45.666 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:45.662+0000 7f6cb98f4700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6cb40f51a0 0x7f6cb4109290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:16:45.666 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:45.662+0000 7f6cb98f4700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6cb40f51a0 0x7f6cb4109290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:34348/0 (socket says 172.21.15.132:34348) 2024-02-20T00:16:45.666 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:45.662+0000 7f6cb98f4700 1 -- 172.21.15.132:0/2290844123 learned_addr learned my addr 172.21.15.132:0/2290844123 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:16:45.666 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:45.662+0000 7f6cb98f4700 1 -- 172.21.15.132:0/2290844123 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cb0005040 con 0x7f6cb40f51a0 2024-02-20T00:16:45.666 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:45.662+0000 7f6cb98f4700 1 --2- 172.21.15.132:0/2290844123 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6cb40f51a0 0x7f6cb4109290 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7f6cb00089f0 tx=0x7f6cb000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:16:45.666 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:45.663+0000 7f6caa7fc700 1 -- 172.21.15.132:0/2290844123 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6cb0009320 con 0x7f6cb40f51a0 2024-02-20T00:16:45.666 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:45.663+0000 7f6cba8f6700 1 -- 172.21.15.132:0/2290844123 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6cb4060480 con 0x7f6cb40f51a0 2024-02-20T00:16:45.666 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:45.663+0000 7f6caa7fc700 1 -- 172.21.15.132:0/2290844123 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6cb0013380 con 0x7f6cb40f51a0 2024-02-20T00:16:45.666 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:45.663+0000 7f6caa7fc700 1 -- 172.21.15.132:0/2290844123 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6cb0014e20 con 0x7f6cb40f51a0 2024-02-20T00:16:45.666 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:45.663+0000 7f6cba8f6700 1 -- 172.21.15.132:0/2290844123 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cb41099c0 con 0x7f6cb40f51a0 2024-02-20T00:16:45.666 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:45.663+0000 7f6caa7fc700 1 -- 172.21.15.132:0/2290844123 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6cb00095f0 con 0x7f6cb40f51a0 2024-02-20T00:16:45.666 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:45.663+0000 7f6cba8f6700 1 -- 172.21.15.132:0/2290844123 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6cb4045ad0 con 0x7f6cb40f51a0 2024-02-20T00:16:45.667 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:45.664+0000 7f6caa7fc700 1 --2- 172.21.15.132:0/2290844123 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f6ca002ff10 0x7f6ca00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:16:45.667 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:45.664+0000 7f6caa7fc700 1 -- 172.21.15.132:0/2290844123 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6cb0040f50 con 0x7f6cb40f51a0 2024-02-20T00:16:45.667 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:45.664+0000 7f6cb90f3700 1 --2- 172.21.15.132:0/2290844123 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f6ca002ff10 0x7f6ca00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:16:45.667 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:45.664+0000 7f6cb90f3700 1 --2- 172.21.15.132:0/2290844123 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f6ca002ff10 0x7f6ca00323d0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f6ca40096a0 tx=0x7f6ca4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:16:45.669 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:45.668+0000 7f6caa7fc700 1 -- 172.21.15.132:0/2290844123 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6cb0014420 con 0x7f6cb40f51a0 2024-02-20T00:16:45.875 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:45.874+0000 7f6cba8f6700 1 -- 172.21.15.132:0/2290844123 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6cb40fad30 con 0x7f6cb40f51a0 2024-02-20T00:16:45.876 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:45.874+0000 7f6caa7fc700 1 -- 172.21.15.132:0/2290844123 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6cb0010230 con 0x7f6cb40f51a0 2024-02-20T00:16:45.876 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:16:45.876 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:16:45.877 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:45.876+0000 7f6cba8f6700 1 -- 172.21.15.132:0/2290844123 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f6ca002ff10 msgr2=0x7f6ca00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:16:45.878 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:45.876+0000 7f6cba8f6700 1 --2- 172.21.15.132:0/2290844123 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f6ca002ff10 0x7f6ca00323d0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f6ca40096a0 tx=0x7f6ca4006b40).stop 2024-02-20T00:16:45.878 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:45.876+0000 7f6cba8f6700 1 -- 172.21.15.132:0/2290844123 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6cb40f51a0 msgr2=0x7f6cb4109290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:16:45.878 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:45.876+0000 7f6cba8f6700 1 --2- 172.21.15.132:0/2290844123 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6cb40f51a0 0x7f6cb4109290 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7f6cb00089f0 tx=0x7f6cb000b040).stop 2024-02-20T00:16:45.878 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:45.876+0000 7f6cba8f6700 1 -- 172.21.15.132:0/2290844123 shutdown_connections 2024-02-20T00:16:45.878 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:45.876+0000 7f6cba8f6700 1 --2- 172.21.15.132:0/2290844123 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f6ca002ff10 0x7f6ca00323d0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:16:45.878 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:45.876+0000 7f6cba8f6700 1 --2- 172.21.15.132:0/2290844123 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6cb40f51a0 0x7f6cb4109290 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:16:45.878 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:45.876+0000 7f6cba8f6700 1 -- 172.21.15.132:0/2290844123 >> 172.21.15.132:0/2290844123 conn(0x7f6cb40f09a0 msgr2=0x7f6cb40f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:16:45.878 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:45.876+0000 7f6cba8f6700 1 -- 172.21.15.132:0/2290844123 shutdown_connections 2024-02-20T00:16:45.878 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:45.876+0000 7f6cba8f6700 1 -- 172.21.15.132:0/2290844123 wait complete. 2024-02-20T00:16:45.878 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:16:46.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:16:45 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:16:44.899840+0000 mgr.smithi084.bmqrnf (mgr.14162) 210 : cluster [DBG] pgmap v198: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:16:46.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:16:45 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:16:45.874952+0000 mon.smithi084 (mon.0) 238 : audit [DBG] from='client.? 172.21.15.132:0/2290844123' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:16:47.576 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:16:47.577 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:16:48.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:16:47 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:16:46.900169+0000 mgr.smithi084.bmqrnf (mgr.14162) 211 : cluster [DBG] pgmap v199: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:16:49.493 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:49.470+0000 7f07bf6ae700 1 -- 172.21.15.132:0/2351143123 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f07b80f5410 msgr2=0x7f07b80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:16:49.493 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:49.470+0000 7f07bf6ae700 1 --2- 172.21.15.132:0/2351143123 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f07b80f5410 0x7f07b80f5830 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 rx=0x7f07a8009fb0 tx=0x7f07a8009b30).stop 2024-02-20T00:16:49.493 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:49.470+0000 7f07bf6ae700 1 -- 172.21.15.132:0/2351143123 shutdown_connections 2024-02-20T00:16:49.493 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:49.470+0000 7f07bf6ae700 1 --2- 172.21.15.132:0/2351143123 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f07b80f5410 0x7f07b80f5830 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:16:49.493 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:49.470+0000 7f07bf6ae700 1 -- 172.21.15.132:0/2351143123 >> 172.21.15.132:0/2351143123 conn(0x7f07b80f0b70 msgr2=0x7f07b80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:16:49.493 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:49.470+0000 7f07bf6ae700 1 -- 172.21.15.132:0/2351143123 shutdown_connections 2024-02-20T00:16:49.493 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:49.470+0000 7f07bf6ae700 1 -- 172.21.15.132:0/2351143123 wait complete. 2024-02-20T00:16:49.494 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:49.471+0000 7f07bf6ae700 1 Processor -- start 2024-02-20T00:16:49.494 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:49.471+0000 7f07bf6ae700 1 -- start start 2024-02-20T00:16:49.494 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:49.471+0000 7f07bf6ae700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f07b80f5410 0x7f07b81089f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:16:49.494 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:49.471+0000 7f07bf6ae700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07b8108ee0 con 0x7f07b80f5410 2024-02-20T00:16:49.494 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:49.471+0000 7f07bd44a700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f07b80f5410 0x7f07b81089f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:16:49.494 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:49.471+0000 7f07bd44a700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f07b80f5410 0x7f07b81089f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:34718/0 (socket says 172.21.15.132:34718) 2024-02-20T00:16:49.494 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:49.471+0000 7f07bd44a700 1 -- 172.21.15.132:0/2305662157 learned_addr learned my addr 172.21.15.132:0/2305662157 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:16:49.494 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:49.472+0000 7f07bd44a700 1 -- 172.21.15.132:0/2305662157 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07a8005040 con 0x7f07b80f5410 2024-02-20T00:16:49.494 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:49.472+0000 7f07bd44a700 1 --2- 172.21.15.132:0/2305662157 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f07b80f5410 0x7f07b81089f0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7f07a8005010 tx=0x7f07a800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:16:49.494 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:49.472+0000 7f07adffb700 1 -- 172.21.15.132:0/2305662157 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f07a8009320 con 0x7f07b80f5410 2024-02-20T00:16:49.494 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:49.472+0000 7f07bf6ae700 1 -- 172.21.15.132:0/2305662157 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07b81090e0 con 0x7f07b80f5410 2024-02-20T00:16:49.494 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:49.472+0000 7f07adffb700 1 -- 172.21.15.132:0/2305662157 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f07a8013380 con 0x7f07b80f5410 2024-02-20T00:16:49.494 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:49.472+0000 7f07adffb700 1 -- 172.21.15.132:0/2305662157 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f07a8014e50 con 0x7f07b80f5410 2024-02-20T00:16:49.495 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:49.472+0000 7f07bf6ae700 1 -- 172.21.15.132:0/2305662157 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07b8109570 con 0x7f07b80f5410 2024-02-20T00:16:49.495 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:49.473+0000 7f07adffb700 1 -- 172.21.15.132:0/2305662157 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f07a80095f0 con 0x7f07b80f5410 2024-02-20T00:16:49.495 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:49.473+0000 7f07bf6ae700 1 -- 172.21.15.132:0/2305662157 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07b8045ad0 con 0x7f07b80f5410 2024-02-20T00:16:49.495 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:49.473+0000 7f07adffb700 1 --2- 172.21.15.132:0/2305662157 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f07a402ff10 0x7f07a40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:16:49.495 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:49.473+0000 7f07adffb700 1 -- 172.21.15.132:0/2305662157 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f07a8040eb0 con 0x7f07b80f5410 2024-02-20T00:16:49.495 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:49.473+0000 7f07bcc49700 1 --2- 172.21.15.132:0/2305662157 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f07a402ff10 0x7f07a40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:16:49.495 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:49.474+0000 7f07bcc49700 1 --2- 172.21.15.132:0/2305662157 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f07a402ff10 0x7f07a40323d0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f07b40096a0 tx=0x7f07b4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:16:49.495 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:49.478+0000 7f07adffb700 1 -- 172.21.15.132:0/2305662157 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f07a8014420 con 0x7f07b80f5410 2024-02-20T00:16:49.682 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:49.681+0000 7f07bf6ae700 1 -- 172.21.15.132:0/2305662157 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f07b80fafa0 con 0x7f07b80f5410 2024-02-20T00:16:49.683 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:49.682+0000 7f07adffb700 1 -- 172.21.15.132:0/2305662157 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f07a8010230 con 0x7f07b80f5410 2024-02-20T00:16:49.683 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:16:49.683 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:16:49.685 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:49.683+0000 7f07bf6ae700 1 -- 172.21.15.132:0/2305662157 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f07a402ff10 msgr2=0x7f07a40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:16:49.685 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:49.683+0000 7f07bf6ae700 1 --2- 172.21.15.132:0/2305662157 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f07a402ff10 0x7f07a40323d0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f07b40096a0 tx=0x7f07b4006b40).stop 2024-02-20T00:16:49.685 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:49.683+0000 7f07bf6ae700 1 -- 172.21.15.132:0/2305662157 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f07b80f5410 msgr2=0x7f07b81089f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:16:49.685 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:49.683+0000 7f07bf6ae700 1 --2- 172.21.15.132:0/2305662157 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f07b80f5410 0x7f07b81089f0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7f07a8005010 tx=0x7f07a800b040).stop 2024-02-20T00:16:49.685 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:49.683+0000 7f07bf6ae700 1 -- 172.21.15.132:0/2305662157 shutdown_connections 2024-02-20T00:16:49.686 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:49.683+0000 7f07bf6ae700 1 --2- 172.21.15.132:0/2305662157 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f07a402ff10 0x7f07a40323d0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:16:49.686 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:49.683+0000 7f07bf6ae700 1 --2- 172.21.15.132:0/2305662157 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f07b80f5410 0x7f07b81089f0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:16:49.686 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:49.683+0000 7f07bf6ae700 1 -- 172.21.15.132:0/2305662157 >> 172.21.15.132:0/2305662157 conn(0x7f07b80f0b70 msgr2=0x7f07b805fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:16:49.686 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:49.684+0000 7f07bf6ae700 1 -- 172.21.15.132:0/2305662157 shutdown_connections 2024-02-20T00:16:49.686 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:49.684+0000 7f07bf6ae700 1 -- 172.21.15.132:0/2305662157 wait complete. 2024-02-20T00:16:49.686 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:16:50.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:16:49 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:16:48.900560+0000 mgr.smithi084.bmqrnf (mgr.14162) 212 : cluster [DBG] pgmap v200: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:16:50.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:16:49 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:16:49.682432+0000 mon.smithi084 (mon.0) 239 : audit [DBG] from='client.? 172.21.15.132:0/2305662157' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:16:51.366 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:16:51.367 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:16:52.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:16:51 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:16:50.900842+0000 mgr.smithi084.bmqrnf (mgr.14162) 213 : cluster [DBG] pgmap v201: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:16:53.350 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:53.347+0000 7f4c47160700 1 -- 172.21.15.132:0/3367723275 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4c400f5410 msgr2=0x7f4c400f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:16:53.350 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:53.347+0000 7f4c47160700 1 --2- 172.21.15.132:0/3367723275 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4c400f5410 0x7f4c400f5830 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 rx=0x7f4c34009fb0 tx=0x7f4c34009b30).stop 2024-02-20T00:16:53.350 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:53.347+0000 7f4c47160700 1 -- 172.21.15.132:0/3367723275 shutdown_connections 2024-02-20T00:16:53.350 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:53.347+0000 7f4c47160700 1 --2- 172.21.15.132:0/3367723275 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4c400f5410 0x7f4c400f5830 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:16:53.350 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:53.347+0000 7f4c47160700 1 -- 172.21.15.132:0/3367723275 >> 172.21.15.132:0/3367723275 conn(0x7f4c400f0b70 msgr2=0x7f4c400f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:16:53.351 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:53.347+0000 7f4c47160700 1 -- 172.21.15.132:0/3367723275 shutdown_connections 2024-02-20T00:16:53.351 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:53.347+0000 7f4c47160700 1 -- 172.21.15.132:0/3367723275 wait complete. 2024-02-20T00:16:53.351 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:53.348+0000 7f4c47160700 1 Processor -- start 2024-02-20T00:16:53.351 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:53.348+0000 7f4c47160700 1 -- start start 2024-02-20T00:16:53.351 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:53.348+0000 7f4c47160700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4c400f5410 0x7f4c40109500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:16:53.351 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:53.348+0000 7f4c47160700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c40060220 con 0x7f4c400f5410 2024-02-20T00:16:53.353 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:53.348+0000 7f4c44efc700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4c400f5410 0x7f4c40109500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:16:53.353 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:53.348+0000 7f4c44efc700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4c400f5410 0x7f4c40109500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:34732/0 (socket says 172.21.15.132:34732) 2024-02-20T00:16:53.353 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:53.348+0000 7f4c44efc700 1 -- 172.21.15.132:0/3777159276 learned_addr learned my addr 172.21.15.132:0/3777159276 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:16:53.353 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:53.348+0000 7f4c44efc700 1 -- 172.21.15.132:0/3777159276 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c34005040 con 0x7f4c400f5410 2024-02-20T00:16:53.353 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:53.349+0000 7f4c44efc700 1 --2- 172.21.15.132:0/3777159276 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4c400f5410 0x7f4c40109500 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0x7f4c34005010 tx=0x7f4c3400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:16:53.353 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:53.349+0000 7f4c3d7fa700 1 -- 172.21.15.132:0/3777159276 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4c34009320 con 0x7f4c400f5410 2024-02-20T00:16:53.354 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:53.349+0000 7f4c3d7fa700 1 -- 172.21.15.132:0/3777159276 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4c34013380 con 0x7f4c400f5410 2024-02-20T00:16:53.354 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:53.349+0000 7f4c47160700 1 -- 172.21.15.132:0/3777159276 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c40060420 con 0x7f4c400f5410 2024-02-20T00:16:53.354 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:53.349+0000 7f4c3d7fa700 1 -- 172.21.15.132:0/3777159276 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4c34014e50 con 0x7f4c400f5410 2024-02-20T00:16:53.354 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:53.349+0000 7f4c47160700 1 -- 172.21.15.132:0/3777159276 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c40109bb0 con 0x7f4c400f5410 2024-02-20T00:16:53.354 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:53.350+0000 7f4c3d7fa700 1 -- 172.21.15.132:0/3777159276 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4c340095f0 con 0x7f4c400f5410 2024-02-20T00:16:53.354 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:53.350+0000 7f4c47160700 1 -- 172.21.15.132:0/3777159276 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c40059100 con 0x7f4c400f5410 2024-02-20T00:16:53.354 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:53.350+0000 7f4c3d7fa700 1 --2- 172.21.15.132:0/3777159276 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f4c3002ff10 0x7f4c300323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:16:53.354 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:53.350+0000 7f4c3d7fa700 1 -- 172.21.15.132:0/3777159276 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4c34040fa0 con 0x7f4c400f5410 2024-02-20T00:16:53.354 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:53.350+0000 7f4c3ffff700 1 --2- 172.21.15.132:0/3777159276 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f4c3002ff10 0x7f4c300323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:16:53.354 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:53.350+0000 7f4c3ffff700 1 --2- 172.21.15.132:0/3777159276 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f4c3002ff10 0x7f4c300323d0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f4c2c0096a0 tx=0x7f4c2c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:16:53.357 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:53.355+0000 7f4c3d7fa700 1 -- 172.21.15.132:0/3777159276 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4c34014420 con 0x7f4c400f5410 2024-02-20T00:16:53.562 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:53.560+0000 7f4c47160700 1 -- 172.21.15.132:0/3777159276 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4c40045ad0 con 0x7f4c400f5410 2024-02-20T00:16:53.563 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:53.561+0000 7f4c3d7fa700 1 -- 172.21.15.132:0/3777159276 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4c34010230 con 0x7f4c400f5410 2024-02-20T00:16:53.563 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:16:53.563 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:16:53.565 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:53.562+0000 7f4c47160700 1 -- 172.21.15.132:0/3777159276 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f4c3002ff10 msgr2=0x7f4c300323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:16:53.565 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:53.562+0000 7f4c47160700 1 --2- 172.21.15.132:0/3777159276 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f4c3002ff10 0x7f4c300323d0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f4c2c0096a0 tx=0x7f4c2c006b40).stop 2024-02-20T00:16:53.566 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:53.562+0000 7f4c47160700 1 -- 172.21.15.132:0/3777159276 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4c400f5410 msgr2=0x7f4c40109500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:16:53.566 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:53.562+0000 7f4c47160700 1 --2- 172.21.15.132:0/3777159276 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4c400f5410 0x7f4c40109500 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0x7f4c34005010 tx=0x7f4c3400b040).stop 2024-02-20T00:16:53.566 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:53.563+0000 7f4c47160700 1 -- 172.21.15.132:0/3777159276 shutdown_connections 2024-02-20T00:16:53.566 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:53.563+0000 7f4c47160700 1 --2- 172.21.15.132:0/3777159276 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f4c3002ff10 0x7f4c300323d0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:16:53.566 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:53.563+0000 7f4c47160700 1 --2- 172.21.15.132:0/3777159276 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4c400f5410 0x7f4c40109500 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:16:53.566 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:53.563+0000 7f4c47160700 1 -- 172.21.15.132:0/3777159276 >> 172.21.15.132:0/3777159276 conn(0x7f4c400f0b70 msgr2=0x7f4c400f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:16:53.566 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:53.563+0000 7f4c47160700 1 -- 172.21.15.132:0/3777159276 shutdown_connections 2024-02-20T00:16:53.566 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:53.563+0000 7f4c47160700 1 -- 172.21.15.132:0/3777159276 wait complete. 2024-02-20T00:16:53.566 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:16:54.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:16:53 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:16:52.901154+0000 mgr.smithi084.bmqrnf (mgr.14162) 214 : cluster [DBG] pgmap v202: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:16:54.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:16:53 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:16:53.562572+0000 mon.smithi084 (mon.0) 240 : audit [DBG] from='client.? 172.21.15.132:0/3777159276' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:16:55.221 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:16:55.221 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:16:56.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:16:55 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:16:54.901558+0000 mgr.smithi084.bmqrnf (mgr.14162) 215 : cluster [DBG] pgmap v203: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:16:57.154 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:57.141+0000 7f3bc22d7700 1 -- 172.21.15.132:0/1243789664 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3bbc0f6510 msgr2=0x7f3bbc0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:16:57.154 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:57.141+0000 7f3bc22d7700 1 --2- 172.21.15.132:0/1243789664 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3bbc0f6510 0x7f3bbc0f6930 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 rx=0x7f3bac009fb0 tx=0x7f3bac009b30).stop 2024-02-20T00:16:57.154 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:57.141+0000 7f3bc22d7700 1 -- 172.21.15.132:0/1243789664 shutdown_connections 2024-02-20T00:16:57.154 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:57.141+0000 7f3bc22d7700 1 --2- 172.21.15.132:0/1243789664 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3bbc0f6510 0x7f3bbc0f6930 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:16:57.154 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:57.141+0000 7f3bc22d7700 1 -- 172.21.15.132:0/1243789664 >> 172.21.15.132:0/1243789664 conn(0x7f3bbc0f1c70 msgr2=0x7f3bbc0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:16:57.155 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:57.141+0000 7f3bc22d7700 1 -- 172.21.15.132:0/1243789664 shutdown_connections 2024-02-20T00:16:57.155 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:57.141+0000 7f3bc22d7700 1 -- 172.21.15.132:0/1243789664 wait complete. 2024-02-20T00:16:57.155 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:57.141+0000 7f3bc22d7700 1 Processor -- start 2024-02-20T00:16:57.155 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:57.141+0000 7f3bc22d7700 1 -- start start 2024-02-20T00:16:57.155 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:57.141+0000 7f3bc22d7700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3bbc0f6510 0x7f3bbc1067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:16:57.155 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:57.141+0000 7f3bc22d7700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bbc106cd0 con 0x7f3bbc0f6510 2024-02-20T00:16:57.155 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:57.142+0000 7f3bbbfff700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3bbc0f6510 0x7f3bbc1067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:16:57.155 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:57.142+0000 7f3bbbfff700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3bbc0f6510 0x7f3bbc1067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:34738/0 (socket says 172.21.15.132:34738) 2024-02-20T00:16:57.155 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:57.142+0000 7f3bbbfff700 1 -- 172.21.15.132:0/1165559966 learned_addr learned my addr 172.21.15.132:0/1165559966 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:16:57.155 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:57.142+0000 7f3bbbfff700 1 -- 172.21.15.132:0/1165559966 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bac005040 con 0x7f3bbc0f6510 2024-02-20T00:16:57.155 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:57.142+0000 7f3bbbfff700 1 --2- 172.21.15.132:0/1165559966 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3bbc0f6510 0x7f3bbc1067e0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0x7f3bac005010 tx=0x7f3bac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:16:57.155 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:57.142+0000 7f3bb8ff9700 1 -- 172.21.15.132:0/1165559966 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3bac009320 con 0x7f3bbc0f6510 2024-02-20T00:16:57.155 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:57.142+0000 7f3bb8ff9700 1 -- 172.21.15.132:0/1165559966 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3bac013380 con 0x7f3bbc0f6510 2024-02-20T00:16:57.156 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:57.142+0000 7f3bb8ff9700 1 -- 172.21.15.132:0/1165559966 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3bac014e50 con 0x7f3bbc0f6510 2024-02-20T00:16:57.156 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:57.142+0000 7f3bc22d7700 1 -- 172.21.15.132:0/1165559966 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bbc106ed0 con 0x7f3bbc0f6510 2024-02-20T00:16:57.156 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:57.142+0000 7f3bc22d7700 1 -- 172.21.15.132:0/1165559966 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bbc107360 con 0x7f3bbc0f6510 2024-02-20T00:16:57.156 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:57.143+0000 7f3bb8ff9700 1 -- 172.21.15.132:0/1165559966 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3bac0095f0 con 0x7f3bbc0f6510 2024-02-20T00:16:57.156 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:57.143+0000 7f3bc22d7700 1 -- 172.21.15.132:0/1165559966 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bbc045ad0 con 0x7f3bbc0f6510 2024-02-20T00:16:57.156 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:57.143+0000 7f3bb8ff9700 1 --2- 172.21.15.132:0/1165559966 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f3ba402ff10 0x7f3ba40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:16:57.156 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:57.143+0000 7f3bb8ff9700 1 -- 172.21.15.132:0/1165559966 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3bac040e60 con 0x7f3bbc0f6510 2024-02-20T00:16:57.156 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:57.144+0000 7f3bbb7fe700 1 --2- 172.21.15.132:0/1165559966 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f3ba402ff10 0x7f3ba40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:16:57.156 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:57.144+0000 7f3bbb7fe700 1 --2- 172.21.15.132:0/1165559966 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f3ba402ff10 0x7f3ba40323d0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f3bb00096a0 tx=0x7f3bb0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:16:57.156 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:57.148+0000 7f3bb8ff9700 1 -- 172.21.15.132:0/1165559966 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3bac014420 con 0x7f3bbc0f6510 2024-02-20T00:16:57.353 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:57.351+0000 7f3bc22d7700 1 -- 172.21.15.132:0/1165559966 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3bbc108210 con 0x7f3bbc0f6510 2024-02-20T00:16:57.353 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:57.351+0000 7f3bb8ff9700 1 -- 172.21.15.132:0/1165559966 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3bac01d980 con 0x7f3bbc0f6510 2024-02-20T00:16:57.354 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:16:57.354 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:16:57.355 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:57.353+0000 7f3bc22d7700 1 -- 172.21.15.132:0/1165559966 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f3ba402ff10 msgr2=0x7f3ba40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:16:57.355 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:57.353+0000 7f3bc22d7700 1 --2- 172.21.15.132:0/1165559966 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f3ba402ff10 0x7f3ba40323d0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f3bb00096a0 tx=0x7f3bb0006b40).stop 2024-02-20T00:16:57.355 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:57.353+0000 7f3bc22d7700 1 -- 172.21.15.132:0/1165559966 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3bbc0f6510 msgr2=0x7f3bbc1067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:16:57.356 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:57.353+0000 7f3bc22d7700 1 --2- 172.21.15.132:0/1165559966 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3bbc0f6510 0x7f3bbc1067e0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0x7f3bac005010 tx=0x7f3bac00b040).stop 2024-02-20T00:16:57.356 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:57.353+0000 7f3bc22d7700 1 -- 172.21.15.132:0/1165559966 shutdown_connections 2024-02-20T00:16:57.356 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:57.353+0000 7f3bc22d7700 1 --2- 172.21.15.132:0/1165559966 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f3ba402ff10 0x7f3ba40323d0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:16:57.356 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:57.353+0000 7f3bc22d7700 1 --2- 172.21.15.132:0/1165559966 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3bbc0f6510 0x7f3bbc1067e0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:16:57.356 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:57.353+0000 7f3bc22d7700 1 -- 172.21.15.132:0/1165559966 >> 172.21.15.132:0/1165559966 conn(0x7f3bbc0f1c70 msgr2=0x7f3bbc0f26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:16:57.356 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:57.353+0000 7f3bc22d7700 1 -- 172.21.15.132:0/1165559966 shutdown_connections 2024-02-20T00:16:57.356 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:16:57.353+0000 7f3bc22d7700 1 -- 172.21.15.132:0/1165559966 wait complete. 2024-02-20T00:16:57.356 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:16:58.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:16:57 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:16:56.901886+0000 mgr.smithi084.bmqrnf (mgr.14162) 216 : cluster [DBG] pgmap v204: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:16:58.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:16:57 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:16:56.976770+0000 mon.smithi084 (mon.0) 241 : audit [INF] from='mgr.14162 172.21.15.84:0/2152933216' entity='mgr.smithi084.bmqrnf' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi084.bmqrnf/mirror_snapshot_schedule"}]: dispatch 2024-02-20T00:16:58.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:16:57 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:16:56.982563+0000 mon.smithi084 (mon.0) 242 : audit [INF] from='mgr.14162 172.21.15.84:0/2152933216' entity='mgr.smithi084.bmqrnf' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi084.bmqrnf/trash_purge_schedule"}]: dispatch 2024-02-20T00:16:58.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:16:57 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:16:57.352914+0000 mon.smithi084 (mon.0) 243 : audit [DBG] from='client.? 172.21.15.132:0/1165559966' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:16:59.147 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:16:59.148 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:17:00.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:16:59 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:16:58.902144+0000 mgr.smithi084.bmqrnf (mgr.14162) 217 : cluster [DBG] pgmap v205: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:17:01.063 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:01.060+0000 7f0a01d25700 1 -- 172.21.15.132:0/2259378242 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f09fc0f6510 msgr2=0x7f09fc0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:17:01.063 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:01.060+0000 7f0a01d25700 1 --2- 172.21.15.132:0/2259378242 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f09fc0f6510 0x7f09fc0f6930 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 rx=0x7f09ec009fb0 tx=0x7f09ec009b30).stop 2024-02-20T00:17:01.063 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:01.060+0000 7f0a01d25700 1 -- 172.21.15.132:0/2259378242 shutdown_connections 2024-02-20T00:17:01.063 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:01.060+0000 7f0a01d25700 1 --2- 172.21.15.132:0/2259378242 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f09fc0f6510 0x7f09fc0f6930 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:17:01.063 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:01.060+0000 7f0a01d25700 1 -- 172.21.15.132:0/2259378242 >> 172.21.15.132:0/2259378242 conn(0x7f09fc0f1c70 msgr2=0x7f09fc0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:17:01.063 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:01.060+0000 7f0a01d25700 1 -- 172.21.15.132:0/2259378242 shutdown_connections 2024-02-20T00:17:01.063 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:01.060+0000 7f0a01d25700 1 -- 172.21.15.132:0/2259378242 wait complete. 2024-02-20T00:17:01.063 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:01.060+0000 7f0a01d25700 1 Processor -- start 2024-02-20T00:17:01.063 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:01.060+0000 7f0a01d25700 1 -- start start 2024-02-20T00:17:01.063 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:01.060+0000 7f0a01d25700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f09fc0f6510 0x7f09fc1067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:17:01.064 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:01.060+0000 7f0a01d25700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09fc106cd0 con 0x7f09fc0f6510 2024-02-20T00:17:01.064 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:01.061+0000 7f09fb7fe700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f09fc0f6510 0x7f09fc1067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:17:01.064 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:01.061+0000 7f09fb7fe700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f09fc0f6510 0x7f09fc1067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:34672/0 (socket says 172.21.15.132:34672) 2024-02-20T00:17:01.064 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:01.061+0000 7f09fb7fe700 1 -- 172.21.15.132:0/3285935303 learned_addr learned my addr 172.21.15.132:0/3285935303 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:17:01.065 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:01.061+0000 7f09fb7fe700 1 -- 172.21.15.132:0/3285935303 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09ec005040 con 0x7f09fc0f6510 2024-02-20T00:17:01.065 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:01.061+0000 7f09fb7fe700 1 --2- 172.21.15.132:0/3285935303 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f09fc0f6510 0x7f09fc1067e0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7f09ec005010 tx=0x7f09ec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:17:01.065 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:01.061+0000 7f09e3fff700 1 -- 172.21.15.132:0/3285935303 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f09ec009320 con 0x7f09fc0f6510 2024-02-20T00:17:01.066 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:01.061+0000 7f0a01d25700 1 -- 172.21.15.132:0/3285935303 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09fc106ed0 con 0x7f09fc0f6510 2024-02-20T00:17:01.066 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:01.061+0000 7f09e3fff700 1 -- 172.21.15.132:0/3285935303 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f09ec013380 con 0x7f09fc0f6510 2024-02-20T00:17:01.066 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:01.061+0000 7f09e3fff700 1 -- 172.21.15.132:0/3285935303 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f09ec014e50 con 0x7f09fc0f6510 2024-02-20T00:17:01.066 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:01.061+0000 7f0a01d25700 1 -- 172.21.15.132:0/3285935303 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09fc107360 con 0x7f09fc0f6510 2024-02-20T00:17:01.066 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:01.062+0000 7f09e3fff700 1 -- 172.21.15.132:0/3285935303 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f09ec0095f0 con 0x7f09fc0f6510 2024-02-20T00:17:01.066 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:01.062+0000 7f0a01d25700 1 -- 172.21.15.132:0/3285935303 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09fc045ad0 con 0x7f09fc0f6510 2024-02-20T00:17:01.066 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:01.062+0000 7f09e3fff700 1 --2- 172.21.15.132:0/3285935303 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f09e402ff10 0x7f09e40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:17:01.066 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:01.062+0000 7f09e3fff700 1 -- 172.21.15.132:0/3285935303 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f09ec040fa0 con 0x7f09fc0f6510 2024-02-20T00:17:01.066 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:01.062+0000 7f09faffd700 1 --2- 172.21.15.132:0/3285935303 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f09e402ff10 0x7f09e40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:17:01.066 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:01.063+0000 7f09faffd700 1 --2- 172.21.15.132:0/3285935303 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f09e402ff10 0x7f09e40323d0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f09f00096a0 tx=0x7f09f0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:17:01.069 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:01.067+0000 7f09e3fff700 1 -- 172.21.15.132:0/3285935303 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f09ec014420 con 0x7f09fc0f6510 2024-02-20T00:17:01.273 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:01.271+0000 7f0a01d25700 1 -- 172.21.15.132:0/3285935303 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f09fc108210 con 0x7f09fc0f6510 2024-02-20T00:17:01.274 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:01.271+0000 7f09e3fff700 1 -- 172.21.15.132:0/3285935303 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f09ec010230 con 0x7f09fc0f6510 2024-02-20T00:17:01.274 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:17:01.274 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:17:01.275 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:01.273+0000 7f0a01d25700 1 -- 172.21.15.132:0/3285935303 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f09e402ff10 msgr2=0x7f09e40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:17:01.275 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:01.273+0000 7f0a01d25700 1 --2- 172.21.15.132:0/3285935303 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f09e402ff10 0x7f09e40323d0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f09f00096a0 tx=0x7f09f0006b40).stop 2024-02-20T00:17:01.275 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:01.273+0000 7f0a01d25700 1 -- 172.21.15.132:0/3285935303 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f09fc0f6510 msgr2=0x7f09fc1067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:17:01.275 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:01.273+0000 7f0a01d25700 1 --2- 172.21.15.132:0/3285935303 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f09fc0f6510 0x7f09fc1067e0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7f09ec005010 tx=0x7f09ec00b040).stop 2024-02-20T00:17:01.275 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:01.273+0000 7f0a01d25700 1 -- 172.21.15.132:0/3285935303 shutdown_connections 2024-02-20T00:17:01.275 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:01.273+0000 7f0a01d25700 1 --2- 172.21.15.132:0/3285935303 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f09e402ff10 0x7f09e40323d0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:17:01.275 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:01.273+0000 7f0a01d25700 1 --2- 172.21.15.132:0/3285935303 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f09fc0f6510 0x7f09fc1067e0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:17:01.276 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:01.273+0000 7f0a01d25700 1 -- 172.21.15.132:0/3285935303 >> 172.21.15.132:0/3285935303 conn(0x7f09fc0f1c70 msgr2=0x7f09fc0f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:17:01.276 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:01.273+0000 7f0a01d25700 1 -- 172.21.15.132:0/3285935303 shutdown_connections 2024-02-20T00:17:01.276 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:01.273+0000 7f0a01d25700 1 -- 172.21.15.132:0/3285935303 wait complete. 2024-02-20T00:17:01.276 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:17:02.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:17:01 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:17:00.902485+0000 mgr.smithi084.bmqrnf (mgr.14162) 218 : cluster [DBG] pgmap v206: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:17:02.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:17:01 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:17:01.273019+0000 mon.smithi084 (mon.0) 244 : audit [DBG] from='client.? 172.21.15.132:0/3285935303' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:17:03.040 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:17:03.040 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:17:04.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:17:03 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:17:02.902857+0000 mgr.smithi084.bmqrnf (mgr.14162) 219 : cluster [DBG] pgmap v207: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:17:04.924 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:04.918+0000 7f98378fe700 1 -- 172.21.15.132:0/380842971 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f98300f5410 msgr2=0x7f98300f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:17:04.924 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:04.918+0000 7f98378fe700 1 --2- 172.21.15.132:0/380842971 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f98300f5410 0x7f98300f5830 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 rx=0x7f982c009fb0 tx=0x7f982c009b30).stop 2024-02-20T00:17:04.924 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:04.918+0000 7f98378fe700 1 -- 172.21.15.132:0/380842971 shutdown_connections 2024-02-20T00:17:04.924 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:04.918+0000 7f98378fe700 1 --2- 172.21.15.132:0/380842971 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f98300f5410 0x7f98300f5830 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:17:04.924 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:04.918+0000 7f98378fe700 1 -- 172.21.15.132:0/380842971 >> 172.21.15.132:0/380842971 conn(0x7f98300f0b70 msgr2=0x7f98300f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:17:04.924 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:04.918+0000 7f98378fe700 1 -- 172.21.15.132:0/380842971 shutdown_connections 2024-02-20T00:17:04.924 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:04.918+0000 7f98378fe700 1 -- 172.21.15.132:0/380842971 wait complete. 2024-02-20T00:17:04.924 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:04.919+0000 7f98378fe700 1 Processor -- start 2024-02-20T00:17:04.924 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:04.919+0000 7f98378fe700 1 -- start start 2024-02-20T00:17:04.924 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:04.919+0000 7f98378fe700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f98300f5410 0x7f98300fd470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:17:04.925 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:04.919+0000 7f98378fe700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98300fb960 con 0x7f98300f5410 2024-02-20T00:17:04.925 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:04.919+0000 7f983569a700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f98300f5410 0x7f98300fd470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:17:04.925 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:04.919+0000 7f983569a700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f98300f5410 0x7f98300fd470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:34680/0 (socket says 172.21.15.132:34680) 2024-02-20T00:17:04.925 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:04.919+0000 7f983569a700 1 -- 172.21.15.132:0/637712564 learned_addr learned my addr 172.21.15.132:0/637712564 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:17:04.925 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:04.920+0000 7f983569a700 1 -- 172.21.15.132:0/637712564 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f982c005040 con 0x7f98300f5410 2024-02-20T00:17:04.925 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:04.920+0000 7f983569a700 1 --2- 172.21.15.132:0/637712564 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f98300f5410 0x7f98300fd470 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7f982c005010 tx=0x7f982c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:17:04.925 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:04.920+0000 7f9825ffb700 1 -- 172.21.15.132:0/637712564 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f982c009320 con 0x7f98300f5410 2024-02-20T00:17:04.925 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:04.920+0000 7f98378fe700 1 -- 172.21.15.132:0/637712564 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98300fbb60 con 0x7f98300f5410 2024-02-20T00:17:04.925 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:04.920+0000 7f9825ffb700 1 -- 172.21.15.132:0/637712564 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f982c013380 con 0x7f98300f5410 2024-02-20T00:17:04.925 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:04.920+0000 7f9825ffb700 1 -- 172.21.15.132:0/637712564 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f982c014e50 con 0x7f98300f5410 2024-02-20T00:17:04.925 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:04.920+0000 7f98378fe700 1 -- 172.21.15.132:0/637712564 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98300fbff0 con 0x7f98300f5410 2024-02-20T00:17:04.925 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:04.921+0000 7f9825ffb700 1 -- 172.21.15.132:0/637712564 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f982c0263a0 con 0x7f98300f5410 2024-02-20T00:17:04.925 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:04.921+0000 7f98378fe700 1 -- 172.21.15.132:0/637712564 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9830059100 con 0x7f98300f5410 2024-02-20T00:17:04.925 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:04.921+0000 7f9825ffb700 1 --2- 172.21.15.132:0/637712564 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f981c02ff10 0x7f981c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:17:04.925 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:04.921+0000 7f9825ffb700 1 -- 172.21.15.132:0/637712564 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f982c041240 con 0x7f98300f5410 2024-02-20T00:17:04.926 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:04.921+0000 7f9834e99700 1 --2- 172.21.15.132:0/637712564 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f981c02ff10 0x7f981c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:17:04.926 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:04.922+0000 7f9834e99700 1 --2- 172.21.15.132:0/637712564 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f981c02ff10 0x7f981c0323d0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f98200096a0 tx=0x7f9820006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:17:04.928 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:04.926+0000 7f9825ffb700 1 -- 172.21.15.132:0/637712564 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f982c0126f0 con 0x7f98300f5410 2024-02-20T00:17:05.131 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:05.129+0000 7f98378fe700 1 -- 172.21.15.132:0/637712564 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f98300fafa0 con 0x7f98300f5410 2024-02-20T00:17:05.132 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:05.130+0000 7f9825ffb700 1 -- 172.21.15.132:0/637712564 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f982c012330 con 0x7f98300f5410 2024-02-20T00:17:05.132 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:17:05.132 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:17:05.134 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:05.131+0000 7f98378fe700 1 -- 172.21.15.132:0/637712564 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f981c02ff10 msgr2=0x7f981c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:17:05.134 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:05.131+0000 7f98378fe700 1 --2- 172.21.15.132:0/637712564 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f981c02ff10 0x7f981c0323d0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f98200096a0 tx=0x7f9820006b40).stop 2024-02-20T00:17:05.134 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:05.131+0000 7f98378fe700 1 -- 172.21.15.132:0/637712564 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f98300f5410 msgr2=0x7f98300fd470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:17:05.134 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:05.131+0000 7f98378fe700 1 --2- 172.21.15.132:0/637712564 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f98300f5410 0x7f98300fd470 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7f982c005010 tx=0x7f982c00b040).stop 2024-02-20T00:17:05.135 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:05.131+0000 7f98378fe700 1 -- 172.21.15.132:0/637712564 shutdown_connections 2024-02-20T00:17:05.135 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:05.131+0000 7f98378fe700 1 --2- 172.21.15.132:0/637712564 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f981c02ff10 0x7f981c0323d0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:17:05.135 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:05.131+0000 7f98378fe700 1 --2- 172.21.15.132:0/637712564 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f98300f5410 0x7f98300fd470 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:17:05.135 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:05.131+0000 7f98378fe700 1 -- 172.21.15.132:0/637712564 >> 172.21.15.132:0/637712564 conn(0x7f98300f0b70 msgr2=0x7f983005fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:17:05.135 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:05.131+0000 7f98378fe700 1 -- 172.21.15.132:0/637712564 shutdown_connections 2024-02-20T00:17:05.135 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:05.131+0000 7f98378fe700 1 -- 172.21.15.132:0/637712564 wait complete. 2024-02-20T00:17:05.135 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:17:06.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:17:05 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:17:04.903179+0000 mgr.smithi084.bmqrnf (mgr.14162) 220 : cluster [DBG] pgmap v208: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:17:06.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:17:05 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:17:05.131345+0000 mon.smithi084 (mon.0) 245 : audit [DBG] from='client.? 172.21.15.132:0/637712564' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:17:06.836 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:17:06.837 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:17:08.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:17:07 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:17:06.903472+0000 mgr.smithi084.bmqrnf (mgr.14162) 221 : cluster [DBG] pgmap v209: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:17:08.662 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:08.659+0000 7f98ea68c700 1 -- 172.21.15.132:0/1565469886 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f98e40f51a0 msgr2=0x7f98e40f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:17:08.662 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:08.659+0000 7f98ea68c700 1 --2- 172.21.15.132:0/1565469886 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f98e40f51a0 0x7f98e40f55c0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 rx=0x7f98e0009fb0 tx=0x7f98e0009b30).stop 2024-02-20T00:17:08.662 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:08.659+0000 7f98ea68c700 1 -- 172.21.15.132:0/1565469886 shutdown_connections 2024-02-20T00:17:08.662 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:08.659+0000 7f98ea68c700 1 --2- 172.21.15.132:0/1565469886 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f98e40f51a0 0x7f98e40f55c0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:17:08.662 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:08.659+0000 7f98ea68c700 1 -- 172.21.15.132:0/1565469886 >> 172.21.15.132:0/1565469886 conn(0x7f98e40f09a0 msgr2=0x7f98e40f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:17:08.662 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:08.659+0000 7f98ea68c700 1 -- 172.21.15.132:0/1565469886 shutdown_connections 2024-02-20T00:17:08.662 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:08.659+0000 7f98ea68c700 1 -- 172.21.15.132:0/1565469886 wait complete. 2024-02-20T00:17:08.663 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:08.660+0000 7f98ea68c700 1 Processor -- start 2024-02-20T00:17:08.663 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:08.660+0000 7f98ea68c700 1 -- start start 2024-02-20T00:17:08.663 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:08.660+0000 7f98ea68c700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f98e40f51a0 0x7f98e41092d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:17:08.663 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:08.660+0000 7f98ea68c700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98e4060240 con 0x7f98e40f51a0 2024-02-20T00:17:08.665 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:08.660+0000 7f98e968a700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f98e40f51a0 0x7f98e41092d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:17:08.665 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:08.660+0000 7f98e968a700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f98e40f51a0 0x7f98e41092d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:46592/0 (socket says 172.21.15.132:46592) 2024-02-20T00:17:08.665 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:08.660+0000 7f98e968a700 1 -- 172.21.15.132:0/4054698495 learned_addr learned my addr 172.21.15.132:0/4054698495 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:17:08.665 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:08.661+0000 7f98e968a700 1 -- 172.21.15.132:0/4054698495 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98e0005040 con 0x7f98e40f51a0 2024-02-20T00:17:08.665 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:08.661+0000 7f98e968a700 1 --2- 172.21.15.132:0/4054698495 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f98e40f51a0 0x7f98e41092d0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7f98e00089f0 tx=0x7f98e000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:17:08.665 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:08.661+0000 7f98d9ffb700 1 -- 172.21.15.132:0/4054698495 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f98e0009320 con 0x7f98e40f51a0 2024-02-20T00:17:08.665 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:08.661+0000 7f98ea68c700 1 -- 172.21.15.132:0/4054698495 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98e4060440 con 0x7f98e40f51a0 2024-02-20T00:17:08.665 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:08.661+0000 7f98d9ffb700 1 -- 172.21.15.132:0/4054698495 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f98e0013380 con 0x7f98e40f51a0 2024-02-20T00:17:08.665 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:08.661+0000 7f98d9ffb700 1 -- 172.21.15.132:0/4054698495 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f98e0014e20 con 0x7f98e40f51a0 2024-02-20T00:17:08.665 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:08.661+0000 7f98ea68c700 1 -- 172.21.15.132:0/4054698495 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98e4109980 con 0x7f98e40f51a0 2024-02-20T00:17:08.665 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:08.662+0000 7f98d9ffb700 1 -- 172.21.15.132:0/4054698495 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f98e0014420 con 0x7f98e40f51a0 2024-02-20T00:17:08.665 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:08.662+0000 7f98ea68c700 1 -- 172.21.15.132:0/4054698495 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98e4045ad0 con 0x7f98e40f51a0 2024-02-20T00:17:08.665 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:08.662+0000 7f98d9ffb700 1 --2- 172.21.15.132:0/4054698495 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f98d002ff10 0x7f98d00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:17:08.666 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:08.662+0000 7f98d9ffb700 1 -- 172.21.15.132:0/4054698495 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f98e0040ff0 con 0x7f98e40f51a0 2024-02-20T00:17:08.666 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:08.662+0000 7f98e8e89700 1 --2- 172.21.15.132:0/4054698495 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f98d002ff10 0x7f98d00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:17:08.666 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:08.663+0000 7f98e8e89700 1 --2- 172.21.15.132:0/4054698495 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f98d002ff10 0x7f98d00323d0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f98d40096a0 tx=0x7f98d4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:17:08.669 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:08.667+0000 7f98d9ffb700 1 -- 172.21.15.132:0/4054698495 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f98e00095f0 con 0x7f98e40f51a0 2024-02-20T00:17:08.875 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:08.873+0000 7f98ea68c700 1 -- 172.21.15.132:0/4054698495 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f98e410a1a0 con 0x7f98e40f51a0 2024-02-20T00:17:08.875 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:08.873+0000 7f98d9ffb700 1 -- 172.21.15.132:0/4054698495 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f98e001db70 con 0x7f98e40f51a0 2024-02-20T00:17:08.876 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:17:08.876 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:17:08.877 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:08.875+0000 7f98ea68c700 1 -- 172.21.15.132:0/4054698495 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f98d002ff10 msgr2=0x7f98d00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:17:08.877 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:08.875+0000 7f98ea68c700 1 --2- 172.21.15.132:0/4054698495 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f98d002ff10 0x7f98d00323d0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f98d40096a0 tx=0x7f98d4006b40).stop 2024-02-20T00:17:08.878 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:08.875+0000 7f98ea68c700 1 -- 172.21.15.132:0/4054698495 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f98e40f51a0 msgr2=0x7f98e41092d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:17:08.878 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:08.875+0000 7f98ea68c700 1 --2- 172.21.15.132:0/4054698495 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f98e40f51a0 0x7f98e41092d0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7f98e00089f0 tx=0x7f98e000b040).stop 2024-02-20T00:17:08.878 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:08.875+0000 7f98ea68c700 1 -- 172.21.15.132:0/4054698495 shutdown_connections 2024-02-20T00:17:08.878 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:08.875+0000 7f98ea68c700 1 --2- 172.21.15.132:0/4054698495 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f98d002ff10 0x7f98d00323d0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:17:08.878 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:08.875+0000 7f98ea68c700 1 --2- 172.21.15.132:0/4054698495 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f98e40f51a0 0x7f98e41092d0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:17:08.878 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:08.875+0000 7f98ea68c700 1 -- 172.21.15.132:0/4054698495 >> 172.21.15.132:0/4054698495 conn(0x7f98e40f09a0 msgr2=0x7f98e40f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:17:08.878 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:08.875+0000 7f98ea68c700 1 -- 172.21.15.132:0/4054698495 shutdown_connections 2024-02-20T00:17:08.878 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:08.875+0000 7f98ea68c700 1 -- 172.21.15.132:0/4054698495 wait complete. 2024-02-20T00:17:08.878 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:17:09.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:17:08 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:17:08.874840+0000 mon.smithi084 (mon.0) 246 : audit [DBG] from='client.? 172.21.15.132:0/4054698495' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:17:10.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:17:09 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:17:08.903701+0000 mgr.smithi084.bmqrnf (mgr.14162) 222 : cluster [DBG] pgmap v210: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:17:10.623 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:17:10.623 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:17:12.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:17:11 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:17:10.903932+0000 mgr.smithi084.bmqrnf (mgr.14162) 223 : cluster [DBG] pgmap v211: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:17:12.627 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:12.598+0000 7f6e26846700 1 -- 172.21.15.132:0/300727299 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6e200f6510 msgr2=0x7f6e200f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:17:12.627 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:12.598+0000 7f6e26846700 1 --2- 172.21.15.132:0/300727299 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6e200f6510 0x7f6e200f6930 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 rx=0x7f6e14009fb0 tx=0x7f6e14009b30).stop 2024-02-20T00:17:12.627 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:12.598+0000 7f6e26846700 1 -- 172.21.15.132:0/300727299 shutdown_connections 2024-02-20T00:17:12.628 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:12.598+0000 7f6e26846700 1 --2- 172.21.15.132:0/300727299 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6e200f6510 0x7f6e200f6930 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:17:12.628 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:12.598+0000 7f6e26846700 1 -- 172.21.15.132:0/300727299 >> 172.21.15.132:0/300727299 conn(0x7f6e200f1c70 msgr2=0x7f6e200f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:17:12.628 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:12.598+0000 7f6e26846700 1 -- 172.21.15.132:0/300727299 shutdown_connections 2024-02-20T00:17:12.628 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:12.598+0000 7f6e26846700 1 -- 172.21.15.132:0/300727299 wait complete. 2024-02-20T00:17:12.628 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:12.599+0000 7f6e26846700 1 Processor -- start 2024-02-20T00:17:12.628 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:12.599+0000 7f6e26846700 1 -- start start 2024-02-20T00:17:12.628 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:12.600+0000 7f6e26846700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6e200f6510 0x7f6e201067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:17:12.628 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:12.600+0000 7f6e26846700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e20106cd0 con 0x7f6e200f6510 2024-02-20T00:17:12.628 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:12.600+0000 7f6e1ffff700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6e200f6510 0x7f6e201067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:17:12.628 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:12.600+0000 7f6e1ffff700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6e200f6510 0x7f6e201067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:46608/0 (socket says 172.21.15.132:46608) 2024-02-20T00:17:12.628 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:12.600+0000 7f6e1ffff700 1 -- 172.21.15.132:0/4028057598 learned_addr learned my addr 172.21.15.132:0/4028057598 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:17:12.628 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:12.600+0000 7f6e1ffff700 1 -- 172.21.15.132:0/4028057598 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e14005040 con 0x7f6e200f6510 2024-02-20T00:17:12.628 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:12.600+0000 7f6e1ffff700 1 --2- 172.21.15.132:0/4028057598 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6e200f6510 0x7f6e201067e0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7f6e14005010 tx=0x7f6e1400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:17:12.629 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:12.601+0000 7f6e1cff9700 1 -- 172.21.15.132:0/4028057598 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6e14009320 con 0x7f6e200f6510 2024-02-20T00:17:12.629 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:12.601+0000 7f6e26846700 1 -- 172.21.15.132:0/4028057598 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e20106ed0 con 0x7f6e200f6510 2024-02-20T00:17:12.629 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:12.601+0000 7f6e26846700 1 -- 172.21.15.132:0/4028057598 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e20107360 con 0x7f6e200f6510 2024-02-20T00:17:12.629 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:12.601+0000 7f6e1cff9700 1 -- 172.21.15.132:0/4028057598 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6e14013380 con 0x7f6e200f6510 2024-02-20T00:17:12.629 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:12.601+0000 7f6e1cff9700 1 -- 172.21.15.132:0/4028057598 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6e14014df0 con 0x7f6e200f6510 2024-02-20T00:17:12.629 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:12.601+0000 7f6e26846700 1 -- 172.21.15.132:0/4028057598 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e20045ad0 con 0x7f6e200f6510 2024-02-20T00:17:12.629 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:12.601+0000 7f6e1cff9700 1 -- 172.21.15.132:0/4028057598 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6e140095f0 con 0x7f6e200f6510 2024-02-20T00:17:12.629 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:12.602+0000 7f6e1cff9700 1 --2- 172.21.15.132:0/4028057598 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f6e0802ff10 0x7f6e080323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:17:12.629 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:12.602+0000 7f6e1cff9700 1 -- 172.21.15.132:0/4028057598 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6e14040e90 con 0x7f6e200f6510 2024-02-20T00:17:12.629 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:12.602+0000 7f6e1f7fe700 1 --2- 172.21.15.132:0/4028057598 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f6e0802ff10 0x7f6e080323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:17:12.629 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:12.602+0000 7f6e1f7fe700 1 --2- 172.21.15.132:0/4028057598 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f6e0802ff10 0x7f6e080323d0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7f6e100096a0 tx=0x7f6e10006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:17:12.629 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:12.607+0000 7f6e1cff9700 1 -- 172.21.15.132:0/4028057598 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6e14014420 con 0x7f6e200f6510 2024-02-20T00:17:12.814 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:12.812+0000 7f6e26846700 1 -- 172.21.15.132:0/4028057598 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6e20108210 con 0x7f6e200f6510 2024-02-20T00:17:12.815 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:12.814+0000 7f6e1cff9700 1 -- 172.21.15.132:0/4028057598 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6e1401d9d0 con 0x7f6e200f6510 2024-02-20T00:17:12.815 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:17:12.815 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:17:12.817 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:12.814+0000 7f6e26846700 1 -- 172.21.15.132:0/4028057598 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f6e0802ff10 msgr2=0x7f6e080323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:17:12.817 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:12.814+0000 7f6e26846700 1 --2- 172.21.15.132:0/4028057598 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f6e0802ff10 0x7f6e080323d0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7f6e100096a0 tx=0x7f6e10006b40).stop 2024-02-20T00:17:12.817 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:12.814+0000 7f6e26846700 1 -- 172.21.15.132:0/4028057598 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6e200f6510 msgr2=0x7f6e201067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:17:12.817 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:12.814+0000 7f6e26846700 1 --2- 172.21.15.132:0/4028057598 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6e200f6510 0x7f6e201067e0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7f6e14005010 tx=0x7f6e1400b040).stop 2024-02-20T00:17:12.817 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:12.814+0000 7f6e26846700 1 -- 172.21.15.132:0/4028057598 shutdown_connections 2024-02-20T00:17:12.817 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:12.814+0000 7f6e26846700 1 --2- 172.21.15.132:0/4028057598 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f6e0802ff10 0x7f6e080323d0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:17:12.817 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:12.814+0000 7f6e26846700 1 --2- 172.21.15.132:0/4028057598 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6e200f6510 0x7f6e201067e0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:17:12.817 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:12.814+0000 7f6e26846700 1 -- 172.21.15.132:0/4028057598 >> 172.21.15.132:0/4028057598 conn(0x7f6e200f1c70 msgr2=0x7f6e200f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:17:12.817 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:12.814+0000 7f6e26846700 1 -- 172.21.15.132:0/4028057598 shutdown_connections 2024-02-20T00:17:12.817 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:12.814+0000 7f6e26846700 1 -- 172.21.15.132:0/4028057598 wait complete. 2024-02-20T00:17:12.818 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:17:13.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:17:12 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:17:12.814382+0000 mon.smithi084 (mon.0) 247 : audit [DBG] from='client.? 172.21.15.132:0/4028057598' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:17:14.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:17:13 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:17:12.904275+0000 mgr.smithi084.bmqrnf (mgr.14162) 224 : cluster [DBG] pgmap v212: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:17:14.509 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:17:14.510 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:17:16.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:17:15 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:17:14.904495+0000 mgr.smithi084.bmqrnf (mgr.14162) 225 : cluster [DBG] pgmap v213: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:17:16.438 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:16.436+0000 7fc2ee4d5700 1 -- 172.21.15.132:0/3577668716 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc2e80f6510 msgr2=0x7fc2e80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:17:16.438 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:16.436+0000 7fc2ee4d5700 1 --2- 172.21.15.132:0/3577668716 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc2e80f6510 0x7fc2e80f6930 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 rx=0x7fc2d8009fb0 tx=0x7fc2d8009b30).stop 2024-02-20T00:17:16.438 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:16.436+0000 7fc2ee4d5700 1 -- 172.21.15.132:0/3577668716 shutdown_connections 2024-02-20T00:17:16.438 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:16.436+0000 7fc2ee4d5700 1 --2- 172.21.15.132:0/3577668716 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc2e80f6510 0x7fc2e80f6930 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:17:16.438 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:16.436+0000 7fc2ee4d5700 1 -- 172.21.15.132:0/3577668716 >> 172.21.15.132:0/3577668716 conn(0x7fc2e80f1c70 msgr2=0x7fc2e80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:17:16.438 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:16.436+0000 7fc2ee4d5700 1 -- 172.21.15.132:0/3577668716 shutdown_connections 2024-02-20T00:17:16.438 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:16.436+0000 7fc2ee4d5700 1 -- 172.21.15.132:0/3577668716 wait complete. 2024-02-20T00:17:16.440 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:16.437+0000 7fc2ee4d5700 1 Processor -- start 2024-02-20T00:17:16.440 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:16.437+0000 7fc2ee4d5700 1 -- start start 2024-02-20T00:17:16.440 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:16.438+0000 7fc2ee4d5700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc2e80f6510 0x7fc2e81067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:17:16.440 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:16.438+0000 7fc2ee4d5700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2e8106cd0 con 0x7fc2e80f6510 2024-02-20T00:17:16.440 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:16.438+0000 7fc2e7fff700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc2e80f6510 0x7fc2e81067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:17:16.440 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:16.438+0000 7fc2e7fff700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc2e80f6510 0x7fc2e81067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:46630/0 (socket says 172.21.15.132:46630) 2024-02-20T00:17:16.440 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:16.438+0000 7fc2e7fff700 1 -- 172.21.15.132:0/1274434611 learned_addr learned my addr 172.21.15.132:0/1274434611 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:17:16.440 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:16.438+0000 7fc2e7fff700 1 -- 172.21.15.132:0/1274434611 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2d8005040 con 0x7fc2e80f6510 2024-02-20T00:17:16.440 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:16.438+0000 7fc2e7fff700 1 --2- 172.21.15.132:0/1274434611 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc2e80f6510 0x7fc2e81067e0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7fc2d8005010 tx=0x7fc2d800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:17:16.440 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:16.438+0000 7fc2e4ff9700 1 -- 172.21.15.132:0/1274434611 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc2d8009320 con 0x7fc2e80f6510 2024-02-20T00:17:16.440 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:16.438+0000 7fc2ee4d5700 1 -- 172.21.15.132:0/1274434611 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2e8106ed0 con 0x7fc2e80f6510 2024-02-20T00:17:16.440 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:16.438+0000 7fc2e4ff9700 1 -- 172.21.15.132:0/1274434611 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc2d8013380 con 0x7fc2e80f6510 2024-02-20T00:17:16.441 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:16.438+0000 7fc2e4ff9700 1 -- 172.21.15.132:0/1274434611 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc2d8014e50 con 0x7fc2e80f6510 2024-02-20T00:17:16.441 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:16.438+0000 7fc2ee4d5700 1 -- 172.21.15.132:0/1274434611 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2e8107360 con 0x7fc2e80f6510 2024-02-20T00:17:16.441 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:16.439+0000 7fc2e4ff9700 1 -- 172.21.15.132:0/1274434611 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc2d8014420 con 0x7fc2e80f6510 2024-02-20T00:17:16.441 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:16.439+0000 7fc2e4ff9700 1 --2- 172.21.15.132:0/1274434611 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fc2d002ff10 0x7fc2d00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:17:16.442 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:16.439+0000 7fc2ee4d5700 1 -- 172.21.15.132:0/1274434611 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2e8045ad0 con 0x7fc2e80f6510 2024-02-20T00:17:16.442 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:16.439+0000 7fc2e4ff9700 1 -- 172.21.15.132:0/1274434611 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc2d8040f40 con 0x7fc2e80f6510 2024-02-20T00:17:16.442 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:16.440+0000 7fc2e77fe700 1 --2- 172.21.15.132:0/1274434611 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fc2d002ff10 0x7fc2d00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:17:16.442 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:16.440+0000 7fc2e77fe700 1 --2- 172.21.15.132:0/1274434611 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fc2d002ff10 0x7fc2d00323d0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7fc2dc0096a0 tx=0x7fc2dc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:17:16.445 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:16.444+0000 7fc2e4ff9700 1 -- 172.21.15.132:0/1274434611 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc2d801dd50 con 0x7fc2e80f6510 2024-02-20T00:17:16.649 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:16.648+0000 7fc2ee4d5700 1 -- 172.21.15.132:0/1274434611 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc2e8108210 con 0x7fc2e80f6510 2024-02-20T00:17:16.650 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:16.649+0000 7fc2e4ff9700 1 -- 172.21.15.132:0/1274434611 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc2d8012330 con 0x7fc2e80f6510 2024-02-20T00:17:16.650 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:17:16.650 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:17:16.652 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:16.651+0000 7fc2ee4d5700 1 -- 172.21.15.132:0/1274434611 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fc2d002ff10 msgr2=0x7fc2d00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:17:16.653 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:16.651+0000 7fc2ee4d5700 1 --2- 172.21.15.132:0/1274434611 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fc2d002ff10 0x7fc2d00323d0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7fc2dc0096a0 tx=0x7fc2dc006b40).stop 2024-02-20T00:17:16.653 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:16.651+0000 7fc2ee4d5700 1 -- 172.21.15.132:0/1274434611 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc2e80f6510 msgr2=0x7fc2e81067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:17:16.653 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:16.651+0000 7fc2ee4d5700 1 --2- 172.21.15.132:0/1274434611 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc2e80f6510 0x7fc2e81067e0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7fc2d8005010 tx=0x7fc2d800b040).stop 2024-02-20T00:17:16.653 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:16.651+0000 7fc2ee4d5700 1 -- 172.21.15.132:0/1274434611 shutdown_connections 2024-02-20T00:17:16.653 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:16.651+0000 7fc2ee4d5700 1 --2- 172.21.15.132:0/1274434611 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fc2d002ff10 0x7fc2d00323d0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:17:16.653 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:16.651+0000 7fc2ee4d5700 1 --2- 172.21.15.132:0/1274434611 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc2e80f6510 0x7fc2e81067e0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:17:16.653 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:16.651+0000 7fc2ee4d5700 1 -- 172.21.15.132:0/1274434611 >> 172.21.15.132:0/1274434611 conn(0x7fc2e80f1c70 msgr2=0x7fc2e80f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:17:16.653 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:16.651+0000 7fc2ee4d5700 1 -- 172.21.15.132:0/1274434611 shutdown_connections 2024-02-20T00:17:16.653 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:16.651+0000 7fc2ee4d5700 1 -- 172.21.15.132:0/1274434611 wait complete. 2024-02-20T00:17:16.653 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:17:17.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:17:16 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:17:16.649591+0000 mon.smithi084 (mon.0) 248 : audit [DBG] from='client.? 172.21.15.132:0/1274434611' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:17:18.340 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:17:18.341 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:17:18.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:17:17 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:17:16.904851+0000 mgr.smithi084.bmqrnf (mgr.14162) 226 : cluster [DBG] pgmap v214: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:17:20.127 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:20.124+0000 7f1c12921700 1 -- 172.21.15.132:0/1469383650 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1c0c0f5410 msgr2=0x7f1c0c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:17:20.128 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:20.124+0000 7f1c12921700 1 --2- 172.21.15.132:0/1469383650 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1c0c0f5410 0x7f1c0c0f5830 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 rx=0x7f1bfc009fb0 tx=0x7f1bfc009b30).stop 2024-02-20T00:17:20.128 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:20.124+0000 7f1c12921700 1 -- 172.21.15.132:0/1469383650 shutdown_connections 2024-02-20T00:17:20.128 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:20.124+0000 7f1c12921700 1 --2- 172.21.15.132:0/1469383650 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1c0c0f5410 0x7f1c0c0f5830 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:17:20.128 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:20.124+0000 7f1c12921700 1 -- 172.21.15.132:0/1469383650 >> 172.21.15.132:0/1469383650 conn(0x7f1c0c0f0b70 msgr2=0x7f1c0c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:17:20.128 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:20.124+0000 7f1c12921700 1 -- 172.21.15.132:0/1469383650 shutdown_connections 2024-02-20T00:17:20.128 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:20.124+0000 7f1c12921700 1 -- 172.21.15.132:0/1469383650 wait complete. 2024-02-20T00:17:20.128 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:20.125+0000 7f1c12921700 1 Processor -- start 2024-02-20T00:17:20.128 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:20.125+0000 7f1c12921700 1 -- start start 2024-02-20T00:17:20.128 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:20.125+0000 7f1c12921700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1c0c0f5410 0x7f1c0c0fd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:17:20.128 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:20.125+0000 7f1c12921700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c0c0fb960 con 0x7f1c0c0f5410 2024-02-20T00:17:20.128 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:20.125+0000 7f1c0bfff700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1c0c0f5410 0x7f1c0c0fd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:17:20.128 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:20.125+0000 7f1c0bfff700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1c0c0f5410 0x7f1c0c0fd4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:56568/0 (socket says 172.21.15.132:56568) 2024-02-20T00:17:20.129 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:20.125+0000 7f1c0bfff700 1 -- 172.21.15.132:0/1568330320 learned_addr learned my addr 172.21.15.132:0/1568330320 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:17:20.129 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:20.126+0000 7f1c0bfff700 1 -- 172.21.15.132:0/1568330320 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bfc005040 con 0x7f1c0c0f5410 2024-02-20T00:17:20.129 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:20.126+0000 7f1c0bfff700 1 --2- 172.21.15.132:0/1568330320 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1c0c0f5410 0x7f1c0c0fd4e0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0x7f1bfc005010 tx=0x7f1bfc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:17:20.129 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:20.126+0000 7f1c08ff9700 1 -- 172.21.15.132:0/1568330320 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1bfc009320 con 0x7f1c0c0f5410 2024-02-20T00:17:20.129 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:20.126+0000 7f1c08ff9700 1 -- 172.21.15.132:0/1568330320 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1bfc013380 con 0x7f1c0c0f5410 2024-02-20T00:17:20.129 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:20.126+0000 7f1c08ff9700 1 -- 172.21.15.132:0/1568330320 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1bfc014e50 con 0x7f1c0c0f5410 2024-02-20T00:17:20.129 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:20.126+0000 7f1c12921700 1 -- 172.21.15.132:0/1568330320 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c0c0fbb00 con 0x7f1c0c0f5410 2024-02-20T00:17:20.129 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:20.126+0000 7f1c12921700 1 -- 172.21.15.132:0/1568330320 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c0c0fbf90 con 0x7f1c0c0f5410 2024-02-20T00:17:20.130 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:20.127+0000 7f1c08ff9700 1 -- 172.21.15.132:0/1568330320 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1bfc0095f0 con 0x7f1c0c0f5410 2024-02-20T00:17:20.130 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:20.127+0000 7f1c12921700 1 -- 172.21.15.132:0/1568330320 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c0c059100 con 0x7f1c0c0f5410 2024-02-20T00:17:20.130 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:20.127+0000 7f1c08ff9700 1 --2- 172.21.15.132:0/1568330320 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f1bf402ff10 0x7f1bf40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:17:20.131 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:20.127+0000 7f1c08ff9700 1 -- 172.21.15.132:0/1568330320 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1bfc040f70 con 0x7f1c0c0f5410 2024-02-20T00:17:20.131 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:20.127+0000 7f1c0b7fe700 1 --2- 172.21.15.132:0/1568330320 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f1bf402ff10 0x7f1bf40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:17:20.131 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:20.128+0000 7f1c0b7fe700 1 --2- 172.21.15.132:0/1568330320 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f1bf402ff10 0x7f1bf40323d0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f1c000096a0 tx=0x7f1c00006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:17:20.133 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:20.132+0000 7f1c08ff9700 1 -- 172.21.15.132:0/1568330320 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1bfc014420 con 0x7f1c0c0f5410 2024-02-20T00:17:20.337 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:20.336+0000 7f1c12921700 1 -- 172.21.15.132:0/1568330320 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1c0c0fafa0 con 0x7f1c0c0f5410 2024-02-20T00:17:20.338 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:20.337+0000 7f1c08ff9700 1 -- 172.21.15.132:0/1568330320 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1bfc01d980 con 0x7f1c0c0f5410 2024-02-20T00:17:20.338 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:17:20.338 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:17:20.340 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:20.338+0000 7f1c12921700 1 -- 172.21.15.132:0/1568330320 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f1bf402ff10 msgr2=0x7f1bf40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:17:20.340 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:20.338+0000 7f1c12921700 1 --2- 172.21.15.132:0/1568330320 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f1bf402ff10 0x7f1bf40323d0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f1c000096a0 tx=0x7f1c00006b40).stop 2024-02-20T00:17:20.340 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:20.338+0000 7f1c12921700 1 -- 172.21.15.132:0/1568330320 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1c0c0f5410 msgr2=0x7f1c0c0fd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:17:20.340 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:20.338+0000 7f1c12921700 1 --2- 172.21.15.132:0/1568330320 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1c0c0f5410 0x7f1c0c0fd4e0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0x7f1bfc005010 tx=0x7f1bfc00b040).stop 2024-02-20T00:17:20.340 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:20.338+0000 7f1c12921700 1 -- 172.21.15.132:0/1568330320 shutdown_connections 2024-02-20T00:17:20.340 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:20.338+0000 7f1c12921700 1 --2- 172.21.15.132:0/1568330320 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f1bf402ff10 0x7f1bf40323d0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:17:20.340 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:20.338+0000 7f1c12921700 1 --2- 172.21.15.132:0/1568330320 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1c0c0f5410 0x7f1c0c0fd4e0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:17:20.340 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:20.338+0000 7f1c12921700 1 -- 172.21.15.132:0/1568330320 >> 172.21.15.132:0/1568330320 conn(0x7f1c0c0f0b70 msgr2=0x7f1c0c05fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:17:20.340 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:20.338+0000 7f1c12921700 1 -- 172.21.15.132:0/1568330320 shutdown_connections 2024-02-20T00:17:20.340 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:20.338+0000 7f1c12921700 1 -- 172.21.15.132:0/1568330320 wait complete. 2024-02-20T00:17:20.341 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:17:20.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:17:19 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:17:18.905220+0000 mgr.smithi084.bmqrnf (mgr.14162) 227 : cluster [DBG] pgmap v215: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:17:21.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:17:20 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:17:20.337399+0000 mon.smithi084 (mon.0) 249 : audit [DBG] from='client.? 172.21.15.132:0/1568330320' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:17:22.077 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:17:22.077 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:17:22.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:17:21 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:17:20.905571+0000 mgr.smithi084.bmqrnf (mgr.14162) 228 : cluster [DBG] pgmap v216: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:17:24.092 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:24.089+0000 7ffb58175700 1 -- 172.21.15.132:0/2031773729 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ffb500f6510 msgr2=0x7ffb500f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:17:24.092 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:24.089+0000 7ffb58175700 1 --2- 172.21.15.132:0/2031773729 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ffb500f6510 0x7ffb500f6930 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 rx=0x7ffb40009fb0 tx=0x7ffb40009b30).stop 2024-02-20T00:17:24.092 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:24.089+0000 7ffb58175700 1 -- 172.21.15.132:0/2031773729 shutdown_connections 2024-02-20T00:17:24.093 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:24.089+0000 7ffb58175700 1 --2- 172.21.15.132:0/2031773729 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ffb500f6510 0x7ffb500f6930 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:17:24.093 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:24.089+0000 7ffb58175700 1 -- 172.21.15.132:0/2031773729 >> 172.21.15.132:0/2031773729 conn(0x7ffb500f1c70 msgr2=0x7ffb500f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:17:24.093 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:24.089+0000 7ffb58175700 1 -- 172.21.15.132:0/2031773729 shutdown_connections 2024-02-20T00:17:24.093 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:24.089+0000 7ffb58175700 1 -- 172.21.15.132:0/2031773729 wait complete. 2024-02-20T00:17:24.093 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:24.089+0000 7ffb58175700 1 Processor -- start 2024-02-20T00:17:24.093 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:24.089+0000 7ffb58175700 1 -- start start 2024-02-20T00:17:24.093 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:24.090+0000 7ffb58175700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ffb500f6510 0x7ffb501067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:17:24.093 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:24.090+0000 7ffb58175700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb50106cd0 con 0x7ffb500f6510 2024-02-20T00:17:24.094 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:24.090+0000 7ffb55f11700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ffb500f6510 0x7ffb501067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:17:24.094 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:24.090+0000 7ffb55f11700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ffb500f6510 0x7ffb501067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:56576/0 (socket says 172.21.15.132:56576) 2024-02-20T00:17:24.094 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:24.090+0000 7ffb55f11700 1 -- 172.21.15.132:0/1277360208 learned_addr learned my addr 172.21.15.132:0/1277360208 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:17:24.094 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:24.090+0000 7ffb55f11700 1 -- 172.21.15.132:0/1277360208 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb40005040 con 0x7ffb500f6510 2024-02-20T00:17:24.094 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:24.090+0000 7ffb55f11700 1 --2- 172.21.15.132:0/1277360208 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ffb500f6510 0x7ffb501067e0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x7ffb40005010 tx=0x7ffb4000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:17:24.094 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:24.090+0000 7ffb467fc700 1 -- 172.21.15.132:0/1277360208 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffb40009320 con 0x7ffb500f6510 2024-02-20T00:17:24.094 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:24.090+0000 7ffb58175700 1 -- 172.21.15.132:0/1277360208 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb50106ed0 con 0x7ffb500f6510 2024-02-20T00:17:24.094 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:24.090+0000 7ffb467fc700 1 -- 172.21.15.132:0/1277360208 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ffb40013380 con 0x7ffb500f6510 2024-02-20T00:17:24.094 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:24.090+0000 7ffb467fc700 1 -- 172.21.15.132:0/1277360208 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffb40014e50 con 0x7ffb500f6510 2024-02-20T00:17:24.094 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:24.090+0000 7ffb58175700 1 -- 172.21.15.132:0/1277360208 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb50107360 con 0x7ffb500f6510 2024-02-20T00:17:24.096 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:24.091+0000 7ffb467fc700 1 -- 172.21.15.132:0/1277360208 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ffb400095f0 con 0x7ffb500f6510 2024-02-20T00:17:24.096 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:24.091+0000 7ffb58175700 1 -- 172.21.15.132:0/1277360208 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb50045ad0 con 0x7ffb500f6510 2024-02-20T00:17:24.096 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:24.091+0000 7ffb467fc700 1 --2- 172.21.15.132:0/1277360208 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7ffb3c02ff10 0x7ffb3c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:17:24.096 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:24.091+0000 7ffb467fc700 1 -- 172.21.15.132:0/1277360208 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ffb40040e90 con 0x7ffb500f6510 2024-02-20T00:17:24.096 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:24.092+0000 7ffb55710700 1 --2- 172.21.15.132:0/1277360208 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7ffb3c02ff10 0x7ffb3c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:17:24.096 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:24.092+0000 7ffb55710700 1 --2- 172.21.15.132:0/1277360208 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7ffb3c02ff10 0x7ffb3c0323d0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7ffb4c0096a0 tx=0x7ffb4c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:17:24.097 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:24.096+0000 7ffb467fc700 1 -- 172.21.15.132:0/1277360208 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ffb40014420 con 0x7ffb500f6510 2024-02-20T00:17:24.303 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:24.302+0000 7ffb58175700 1 -- 172.21.15.132:0/1277360208 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ffb50108210 con 0x7ffb500f6510 2024-02-20T00:17:24.303 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:24.302+0000 7ffb467fc700 1 -- 172.21.15.132:0/1277360208 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ffb40010230 con 0x7ffb500f6510 2024-02-20T00:17:24.304 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:17:24.304 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:17:24.305 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:24.304+0000 7ffb58175700 1 -- 172.21.15.132:0/1277360208 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7ffb3c02ff10 msgr2=0x7ffb3c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:17:24.306 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:24.304+0000 7ffb58175700 1 --2- 172.21.15.132:0/1277360208 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7ffb3c02ff10 0x7ffb3c0323d0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7ffb4c0096a0 tx=0x7ffb4c006b40).stop 2024-02-20T00:17:24.306 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:24.304+0000 7ffb58175700 1 -- 172.21.15.132:0/1277360208 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ffb500f6510 msgr2=0x7ffb501067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:17:24.306 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:24.304+0000 7ffb58175700 1 --2- 172.21.15.132:0/1277360208 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ffb500f6510 0x7ffb501067e0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x7ffb40005010 tx=0x7ffb4000b040).stop 2024-02-20T00:17:24.306 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:24.304+0000 7ffb58175700 1 -- 172.21.15.132:0/1277360208 shutdown_connections 2024-02-20T00:17:24.306 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:24.304+0000 7ffb58175700 1 --2- 172.21.15.132:0/1277360208 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7ffb3c02ff10 0x7ffb3c0323d0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:17:24.306 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:24.304+0000 7ffb58175700 1 --2- 172.21.15.132:0/1277360208 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ffb500f6510 0x7ffb501067e0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:17:24.306 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:24.304+0000 7ffb58175700 1 -- 172.21.15.132:0/1277360208 >> 172.21.15.132:0/1277360208 conn(0x7ffb500f1c70 msgr2=0x7ffb500f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:17:24.306 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:24.304+0000 7ffb58175700 1 -- 172.21.15.132:0/1277360208 shutdown_connections 2024-02-20T00:17:24.306 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:24.304+0000 7ffb58175700 1 -- 172.21.15.132:0/1277360208 wait complete. 2024-02-20T00:17:24.306 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:17:24.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:17:23 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:17:22.905879+0000 mgr.smithi084.bmqrnf (mgr.14162) 229 : cluster [DBG] pgmap v217: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:17:25.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:17:24 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:17:24.303004+0000 mon.smithi084 (mon.0) 250 : audit [DBG] from='client.? 172.21.15.132:0/1277360208' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:17:25.936 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:17:25.936 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:17:26.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:17:25 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:17:24.906156+0000 mgr.smithi084.bmqrnf (mgr.14162) 230 : cluster [DBG] pgmap v218: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:17:28.036 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:28.019+0000 7f7ee7508700 1 -- 172.21.15.132:0/2293211180 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7ee00f5410 msgr2=0x7f7ee00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:17:28.036 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:28.019+0000 7f7ee7508700 1 --2- 172.21.15.132:0/2293211180 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7ee00f5410 0x7f7ee00f5830 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 rx=0x7f7edc009fb0 tx=0x7f7edc009b30).stop 2024-02-20T00:17:28.036 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:28.019+0000 7f7ee7508700 1 -- 172.21.15.132:0/2293211180 shutdown_connections 2024-02-20T00:17:28.036 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:28.019+0000 7f7ee7508700 1 --2- 172.21.15.132:0/2293211180 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7ee00f5410 0x7f7ee00f5830 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:17:28.036 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:28.019+0000 7f7ee7508700 1 -- 172.21.15.132:0/2293211180 >> 172.21.15.132:0/2293211180 conn(0x7f7ee00f0b70 msgr2=0x7f7ee00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:17:28.036 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:28.019+0000 7f7ee7508700 1 -- 172.21.15.132:0/2293211180 shutdown_connections 2024-02-20T00:17:28.036 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:28.019+0000 7f7ee7508700 1 -- 172.21.15.132:0/2293211180 wait complete. 2024-02-20T00:17:28.036 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:28.020+0000 7f7ee7508700 1 Processor -- start 2024-02-20T00:17:28.036 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:28.020+0000 7f7ee7508700 1 -- start start 2024-02-20T00:17:28.037 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:28.020+0000 7f7ee7508700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7ee00f5410 0x7f7ee01056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:17:28.037 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:28.020+0000 7f7ee7508700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ee0105bd0 con 0x7f7ee00f5410 2024-02-20T00:17:28.037 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:28.020+0000 7f7ee52a4700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7ee00f5410 0x7f7ee01056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:17:28.037 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:28.020+0000 7f7ee52a4700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7ee00f5410 0x7f7ee01056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:57840/0 (socket says 172.21.15.132:57840) 2024-02-20T00:17:28.037 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:28.020+0000 7f7ee52a4700 1 -- 172.21.15.132:0/458474438 learned_addr learned my addr 172.21.15.132:0/458474438 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:17:28.037 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:28.020+0000 7f7ee52a4700 1 -- 172.21.15.132:0/458474438 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7edc005040 con 0x7f7ee00f5410 2024-02-20T00:17:28.037 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:28.020+0000 7f7ee52a4700 1 --2- 172.21.15.132:0/458474438 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7ee00f5410 0x7f7ee01056e0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0x7f7edc0089f0 tx=0x7f7edc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:17:28.037 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:28.021+0000 7f7ed5ffb700 1 -- 172.21.15.132:0/458474438 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7edc009320 con 0x7f7ee00f5410 2024-02-20T00:17:28.037 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:28.021+0000 7f7ed5ffb700 1 -- 172.21.15.132:0/458474438 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7edc013380 con 0x7f7ee00f5410 2024-02-20T00:17:28.037 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:28.021+0000 7f7ee7508700 1 -- 172.21.15.132:0/458474438 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ee0105dd0 con 0x7f7ee00f5410 2024-02-20T00:17:28.037 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:28.021+0000 7f7ed5ffb700 1 -- 172.21.15.132:0/458474438 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7edc01ed80 con 0x7f7ee00f5410 2024-02-20T00:17:28.037 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:28.021+0000 7f7ee7508700 1 -- 172.21.15.132:0/458474438 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ee0106260 con 0x7f7ee00f5410 2024-02-20T00:17:28.037 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:28.022+0000 7f7ed5ffb700 1 -- 172.21.15.132:0/458474438 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7edc01e420 con 0x7f7ee00f5410 2024-02-20T00:17:28.037 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:28.022+0000 7f7ee7508700 1 -- 172.21.15.132:0/458474438 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ee0045ad0 con 0x7f7ee00f5410 2024-02-20T00:17:28.038 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:28.022+0000 7f7ed5ffb700 1 --2- 172.21.15.132:0/458474438 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f7ecc02ff60 0x7f7ecc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:17:28.038 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:28.022+0000 7f7ed5ffb700 1 -- 172.21.15.132:0/458474438 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7edc041080 con 0x7f7ee00f5410 2024-02-20T00:17:28.038 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:28.022+0000 7f7ee4aa3700 1 --2- 172.21.15.132:0/458474438 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f7ecc02ff60 0x7f7ecc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:17:28.038 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:28.022+0000 7f7ee4aa3700 1 --2- 172.21.15.132:0/458474438 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f7ecc02ff60 0x7f7ecc032420 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7f7ed00096a0 tx=0x7f7ed0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:17:28.038 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:28.027+0000 7f7ed5ffb700 1 -- 172.21.15.132:0/458474438 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7edc0095f0 con 0x7f7ee00f5410 2024-02-20T00:17:28.235 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:28.234+0000 7f7ee7508700 1 -- 172.21.15.132:0/458474438 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7ee01070b0 con 0x7f7ee00f5410 2024-02-20T00:17:28.236 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:28.234+0000 7f7ed5ffb700 1 -- 172.21.15.132:0/458474438 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7edc01cb10 con 0x7f7ee00f5410 2024-02-20T00:17:28.236 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:17:28.236 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:17:28.237 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:28.236+0000 7f7ee7508700 1 -- 172.21.15.132:0/458474438 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f7ecc02ff60 msgr2=0x7f7ecc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:17:28.238 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:28.236+0000 7f7ee7508700 1 --2- 172.21.15.132:0/458474438 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f7ecc02ff60 0x7f7ecc032420 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7f7ed00096a0 tx=0x7f7ed0006b40).stop 2024-02-20T00:17:28.238 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:28.236+0000 7f7ee7508700 1 -- 172.21.15.132:0/458474438 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7ee00f5410 msgr2=0x7f7ee01056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:17:28.238 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:28.236+0000 7f7ee7508700 1 --2- 172.21.15.132:0/458474438 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7ee00f5410 0x7f7ee01056e0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0x7f7edc0089f0 tx=0x7f7edc00b040).stop 2024-02-20T00:17:28.238 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:28.236+0000 7f7ee7508700 1 -- 172.21.15.132:0/458474438 shutdown_connections 2024-02-20T00:17:28.238 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:28.236+0000 7f7ee7508700 1 --2- 172.21.15.132:0/458474438 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f7ecc02ff60 0x7f7ecc032420 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:17:28.238 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:28.236+0000 7f7ee7508700 1 --2- 172.21.15.132:0/458474438 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7ee00f5410 0x7f7ee01056e0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:17:28.238 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:28.236+0000 7f7ee7508700 1 -- 172.21.15.132:0/458474438 >> 172.21.15.132:0/458474438 conn(0x7f7ee00f0b70 msgr2=0x7f7ee00f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:17:28.238 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:28.236+0000 7f7ee7508700 1 -- 172.21.15.132:0/458474438 shutdown_connections 2024-02-20T00:17:28.238 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:28.236+0000 7f7ee7508700 1 -- 172.21.15.132:0/458474438 wait complete. 2024-02-20T00:17:28.239 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:17:28.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:17:27 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:17:26.906397+0000 mgr.smithi084.bmqrnf (mgr.14162) 231 : cluster [DBG] pgmap v219: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:17:29.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:17:28 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:17:28.235116+0000 mon.smithi084 (mon.0) 251 : audit [DBG] from='client.? 172.21.15.132:0/458474438' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:17:30.016 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:17:30.016 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:17:30.359 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:17:29 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:17:28.906708+0000 mgr.smithi084.bmqrnf (mgr.14162) 232 : cluster [DBG] pgmap v220: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:17:32.107 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:32.104+0000 7fde6d382700 1 -- 172.21.15.132:0/2690760318 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fde6806b920 msgr2=0x7fde68069ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:17:32.107 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:32.104+0000 7fde6d382700 1 --2- 172.21.15.132:0/2690760318 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fde6806b920 0x7fde68069ec0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 rx=0x7fde5c009fb0 tx=0x7fde5c009b30).stop 2024-02-20T00:17:32.107 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:32.104+0000 7fde6d382700 1 -- 172.21.15.132:0/2690760318 shutdown_connections 2024-02-20T00:17:32.107 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:32.104+0000 7fde6d382700 1 --2- 172.21.15.132:0/2690760318 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fde6806b920 0x7fde68069ec0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:17:32.107 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:32.104+0000 7fde6d382700 1 -- 172.21.15.132:0/2690760318 >> 172.21.15.132:0/2690760318 conn(0x7fde680f1aa0 msgr2=0x7fde680f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:17:32.107 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:32.104+0000 7fde6d382700 1 -- 172.21.15.132:0/2690760318 shutdown_connections 2024-02-20T00:17:32.108 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:32.104+0000 7fde6d382700 1 -- 172.21.15.132:0/2690760318 wait complete. 2024-02-20T00:17:32.108 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:32.104+0000 7fde6d382700 1 Processor -- start 2024-02-20T00:17:32.108 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:32.105+0000 7fde6d382700 1 -- start start 2024-02-20T00:17:32.108 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:32.105+0000 7fde6d382700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fde6806b920 0x7fde681087b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:17:32.108 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:32.105+0000 7fde6d382700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde68108ca0 con 0x7fde6806b920 2024-02-20T00:17:32.109 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:32.105+0000 7fde67fff700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fde6806b920 0x7fde681087b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:17:32.109 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:32.105+0000 7fde67fff700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fde6806b920 0x7fde681087b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:57860/0 (socket says 172.21.15.132:57860) 2024-02-20T00:17:32.110 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:32.105+0000 7fde67fff700 1 -- 172.21.15.132:0/1022088068 learned_addr learned my addr 172.21.15.132:0/1022088068 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:17:32.110 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:32.105+0000 7fde67fff700 1 -- 172.21.15.132:0/1022088068 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde5c005040 con 0x7fde6806b920 2024-02-20T00:17:32.110 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:32.106+0000 7fde67fff700 1 --2- 172.21.15.132:0/1022088068 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fde6806b920 0x7fde681087b0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7fde5c005010 tx=0x7fde5c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:17:32.110 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:32.106+0000 7fde64ff9700 1 -- 172.21.15.132:0/1022088068 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fde5c009320 con 0x7fde6806b920 2024-02-20T00:17:32.110 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:32.106+0000 7fde6d382700 1 -- 172.21.15.132:0/1022088068 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde68108ea0 con 0x7fde6806b920 2024-02-20T00:17:32.110 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:32.106+0000 7fde64ff9700 1 -- 172.21.15.132:0/1022088068 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fde5c013380 con 0x7fde6806b920 2024-02-20T00:17:32.110 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:32.106+0000 7fde64ff9700 1 -- 172.21.15.132:0/1022088068 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fde5c014e50 con 0x7fde6806b920 2024-02-20T00:17:32.110 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:32.106+0000 7fde6d382700 1 -- 172.21.15.132:0/1022088068 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde68109330 con 0x7fde6806b920 2024-02-20T00:17:32.110 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:32.107+0000 7fde64ff9700 1 -- 172.21.15.132:0/1022088068 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fde5c0095f0 con 0x7fde6806b920 2024-02-20T00:17:32.110 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:32.107+0000 7fde6d382700 1 -- 172.21.15.132:0/1022088068 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde68045ad0 con 0x7fde6806b920 2024-02-20T00:17:32.110 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:32.107+0000 7fde64ff9700 1 --2- 172.21.15.132:0/1022088068 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fde5002ff10 0x7fde500323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:17:32.110 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:32.107+0000 7fde64ff9700 1 -- 172.21.15.132:0/1022088068 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fde5c040e70 con 0x7fde6806b920 2024-02-20T00:17:32.110 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:32.107+0000 7fde677fe700 1 --2- 172.21.15.132:0/1022088068 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fde5002ff10 0x7fde500323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:17:32.110 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:32.107+0000 7fde677fe700 1 --2- 172.21.15.132:0/1022088068 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fde5002ff10 0x7fde500323d0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7fde580096a0 tx=0x7fde58006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:17:32.114 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:32.111+0000 7fde64ff9700 1 -- 172.21.15.132:0/1022088068 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fde5c014420 con 0x7fde6806b920 2024-02-20T00:17:32.321 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:32.318+0000 7fde6d382700 1 -- 172.21.15.132:0/1022088068 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fde6810a180 con 0x7fde6806b920 2024-02-20T00:17:32.321 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:32.319+0000 7fde64ff9700 1 -- 172.21.15.132:0/1022088068 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fde5c010230 con 0x7fde6806b920 2024-02-20T00:17:32.321 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:17:32.322 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:17:32.323 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:32.320+0000 7fde6d382700 1 -- 172.21.15.132:0/1022088068 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fde5002ff10 msgr2=0x7fde500323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:17:32.323 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:32.320+0000 7fde6d382700 1 --2- 172.21.15.132:0/1022088068 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fde5002ff10 0x7fde500323d0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7fde580096a0 tx=0x7fde58006b40).stop 2024-02-20T00:17:32.323 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:32.320+0000 7fde6d382700 1 -- 172.21.15.132:0/1022088068 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fde6806b920 msgr2=0x7fde681087b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:17:32.323 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:32.320+0000 7fde6d382700 1 --2- 172.21.15.132:0/1022088068 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fde6806b920 0x7fde681087b0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7fde5c005010 tx=0x7fde5c00b040).stop 2024-02-20T00:17:32.323 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:32.320+0000 7fde6d382700 1 -- 172.21.15.132:0/1022088068 shutdown_connections 2024-02-20T00:17:32.323 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:32.320+0000 7fde6d382700 1 --2- 172.21.15.132:0/1022088068 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fde5002ff10 0x7fde500323d0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:17:32.323 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:32.320+0000 7fde6d382700 1 --2- 172.21.15.132:0/1022088068 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fde6806b920 0x7fde681087b0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:17:32.323 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:32.320+0000 7fde6d382700 1 -- 172.21.15.132:0/1022088068 >> 172.21.15.132:0/1022088068 conn(0x7fde680f1aa0 msgr2=0x7fde680f24f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:17:32.324 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:32.321+0000 7fde6d382700 1 -- 172.21.15.132:0/1022088068 shutdown_connections 2024-02-20T00:17:32.324 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:32.321+0000 7fde6d382700 1 -- 172.21.15.132:0/1022088068 wait complete. 2024-02-20T00:17:32.324 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:17:32.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:17:31 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:17:30.907054+0000 mgr.smithi084.bmqrnf (mgr.14162) 233 : cluster [DBG] pgmap v221: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:17:33.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:17:32 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:17:32.320529+0000 mon.smithi084 (mon.0) 252 : audit [DBG] from='client.? 172.21.15.132:0/1022088068' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:17:34.033 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:17:34.034 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:17:34.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:17:33 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cephadm 2024-02-20T00:17:32.896851+0000 mgr.smithi084.bmqrnf (mgr.14162) 234 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Deploy daemon node-exporter.smithi084 ... 2024-02-20T00:17:34.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:17:33 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: Verifying port 9100 ... 2024-02-20T00:17:34.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:17:33 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: Non-zero exit code 1 from systemctl start ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715@node-exporter.smithi084 2024-02-20T00:17:34.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:17:33 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: systemctl: stderr Job for ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715@node-exporter.smithi084.service failed because a timeout was exceeded. 2024-02-20T00:17:34.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:17:33 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: systemctl: stderr See "systemctl status ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715@node-exporter.smithi084.service" and "journalctl -xe" for details. 2024-02-20T00:17:34.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:17:33 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: Traceback (most recent call last): 2024-02-20T00:17:34.359 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:17:33 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: File "/var/lib/ceph/2a2e6c42-cf84-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-02-20T00:17:34.359 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:17:33 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: main() 2024-02-20T00:17:34.359 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:17:33 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: File "/var/lib/ceph/2a2e6c42-cf84-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-02-20T00:17:34.359 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:17:33 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: r = ctx.func(ctx) 2024-02-20T00:17:34.359 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:17:33 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: File "/var/lib/ceph/2a2e6c42-cf84-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-02-20T00:17:34.359 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:17:33 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: return func(ctx) 2024-02-20T00:17:34.359 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:17:33 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: File "/var/lib/ceph/2a2e6c42-cf84-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4267, in command_deploy 2024-02-20T00:17:34.359 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:17:33 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: ports=daemon_ports) 2024-02-20T00:17:34.359 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:17:33 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: File "/var/lib/ceph/2a2e6c42-cf84-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2608, in deploy_daemon 2024-02-20T00:17:34.359 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:17:33 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: c, osd_fsid=osd_fsid, ports=ports) 2024-02-20T00:17:34.359 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:17:33 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: File "/var/lib/ceph/2a2e6c42-cf84-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2806, in deploy_daemon_units 2024-02-20T00:17:34.359 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:17:33 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: call_throws(ctx, ['systemctl', 'start', unit_name]) 2024-02-20T00:17:34.359 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:17:33 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: File "/var/lib/ceph/2a2e6c42-cf84-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1464, in call_throws 2024-02-20T00:17:34.359 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:17:33 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-02-20T00:17:34.360 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:17:33 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: RuntimeError: Failed command: systemctl start ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715@node-exporter.smithi084 2024-02-20T00:17:34.360 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:17:33 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: Traceback (most recent call last): 2024-02-20T00:17:34.360 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:17:33 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1144, in _remote_connection 2024-02-20T00:17:34.360 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:17:33 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: yield (conn, connr) 2024-02-20T00:17:34.360 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:17:33 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1059, in _run_cephadm 2024-02-20T00:17:34.360 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:17:33 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: code, '\n'.join(err))) 2024-02-20T00:17:34.360 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:17:33 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Deploy daemon node-exporter.smithi084 ... 2024-02-20T00:17:34.360 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:17:33 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: Verifying port 9100 ... 2024-02-20T00:17:34.360 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:17:33 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: Non-zero exit code 1 from systemctl start ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715@node-exporter.smithi084 2024-02-20T00:17:34.360 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:17:33 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: systemctl: stderr Job for ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715@node-exporter.smithi084.service failed because a timeout was exceeded. 2024-02-20T00:17:34.360 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:17:33 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: systemctl: stderr See "systemctl status ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715@node-exporter.smithi084.service" and "journalctl -xe" for details. 2024-02-20T00:17:34.360 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:17:33 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: Traceback (most recent call last): 2024-02-20T00:17:34.360 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:17:33 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: File "/var/lib/ceph/2a2e6c42-cf84-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-02-20T00:17:34.360 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:17:33 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: main() 2024-02-20T00:17:34.361 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:17:33 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: File "/var/lib/ceph/2a2e6c42-cf84-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-02-20T00:17:34.361 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:17:33 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: r = ctx.func(ctx) 2024-02-20T00:17:34.361 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:17:33 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: File "/var/lib/ceph/2a2e6c42-cf84-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-02-20T00:17:34.361 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:17:33 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: return func(ctx) 2024-02-20T00:17:34.361 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:17:33 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: File "/var/lib/ceph/2a2e6c42-cf84-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4267, in command_deploy 2024-02-20T00:17:34.361 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:17:33 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: ports=daemon_ports) 2024-02-20T00:17:34.361 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:17:33 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: File "/var/lib/ceph/2a2e6c42-cf84-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2608, in deploy_daemon 2024-02-20T00:17:34.361 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:17:33 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: c, osd_fsid=osd_fsid, ports=ports) 2024-02-20T00:17:34.361 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:17:33 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: File "/var/lib/ceph/2a2e6c42-cf84-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2806, in deploy_daemon_units 2024-02-20T00:17:34.361 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:17:33 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: call_throws(ctx, ['systemctl', 'start', unit_name]) 2024-02-20T00:17:34.361 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:17:33 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: File "/var/lib/ceph/2a2e6c42-cf84-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1464, in call_throws 2024-02-20T00:17:34.361 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:17:33 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-02-20T00:17:34.361 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:17:33 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: RuntimeError: Failed command: systemctl start ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715@node-exporter.smithi084 2024-02-20T00:17:34.362 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:17:33 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cephadm 2024-02-20T00:17:32.902443+0000 mgr.smithi084.bmqrnf (mgr.14162) 235 : cephadm [INF] Deploying daemon prometheus.smithi084 on smithi084 2024-02-20T00:17:34.362 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:17:33 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:17:32.907338+0000 mgr.smithi084.bmqrnf (mgr.14162) 236 : cluster [DBG] pgmap v222: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:17:35.984 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:35.980+0000 7fe84b7db700 1 -- 172.21.15.132:0/2507204129 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe8440f47c0 msgr2=0x7fe8440f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:17:35.984 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:35.980+0000 7fe84b7db700 1 --2- 172.21.15.132:0/2507204129 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe8440f47c0 0x7fe8440f4be0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 rx=0x7fe834009fb0 tx=0x7fe834009b30).stop 2024-02-20T00:17:35.984 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:35.980+0000 7fe84b7db700 1 -- 172.21.15.132:0/2507204129 shutdown_connections 2024-02-20T00:17:35.984 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:35.980+0000 7fe84b7db700 1 --2- 172.21.15.132:0/2507204129 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe8440f47c0 0x7fe8440f4be0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:17:35.984 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:35.980+0000 7fe84b7db700 1 -- 172.21.15.132:0/2507204129 >> 172.21.15.132:0/2507204129 conn(0x7fe8440f02e0 msgr2=0x7fe8440f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:17:35.984 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:35.980+0000 7fe84b7db700 1 -- 172.21.15.132:0/2507204129 shutdown_connections 2024-02-20T00:17:35.985 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:35.980+0000 7fe84b7db700 1 -- 172.21.15.132:0/2507204129 wait complete. 2024-02-20T00:17:35.985 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:35.980+0000 7fe84b7db700 1 Processor -- start 2024-02-20T00:17:35.985 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:35.981+0000 7fe84b7db700 1 -- start start 2024-02-20T00:17:35.985 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:35.981+0000 7fe84b7db700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe8440f47c0 0x7fe8441067d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:17:35.985 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:35.981+0000 7fe84b7db700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe844106cc0 con 0x7fe8440f47c0 2024-02-20T00:17:35.985 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:35.981+0000 7fe849577700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe8440f47c0 0x7fe8441067d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:17:35.985 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:35.981+0000 7fe849577700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe8440f47c0 0x7fe8441067d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:57878/0 (socket says 172.21.15.132:57878) 2024-02-20T00:17:35.985 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:35.981+0000 7fe849577700 1 -- 172.21.15.132:0/789592841 learned_addr learned my addr 172.21.15.132:0/789592841 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:17:35.985 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:35.981+0000 7fe849577700 1 -- 172.21.15.132:0/789592841 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe834005040 con 0x7fe8440f47c0 2024-02-20T00:17:35.985 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:35.982+0000 7fe849577700 1 --2- 172.21.15.132:0/789592841 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe8440f47c0 0x7fe8441067d0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0x7fe834005010 tx=0x7fe83400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:17:35.985 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:35.982+0000 7fe839ffb700 1 -- 172.21.15.132:0/789592841 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe834009320 con 0x7fe8440f47c0 2024-02-20T00:17:35.985 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:35.982+0000 7fe839ffb700 1 -- 172.21.15.132:0/789592841 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe834013380 con 0x7fe8440f47c0 2024-02-20T00:17:35.985 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:35.982+0000 7fe84b7db700 1 -- 172.21.15.132:0/789592841 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe844106ec0 con 0x7fe8440f47c0 2024-02-20T00:17:35.985 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:35.982+0000 7fe839ffb700 1 -- 172.21.15.132:0/789592841 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe834014e50 con 0x7fe8440f47c0 2024-02-20T00:17:35.986 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:35.982+0000 7fe84b7db700 1 -- 172.21.15.132:0/789592841 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe844107350 con 0x7fe8440f47c0 2024-02-20T00:17:35.986 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:35.983+0000 7fe839ffb700 1 -- 172.21.15.132:0/789592841 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe8340095f0 con 0x7fe8440f47c0 2024-02-20T00:17:35.987 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:35.983+0000 7fe84b7db700 1 -- 172.21.15.132:0/789592841 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe844045ad0 con 0x7fe8440f47c0 2024-02-20T00:17:35.987 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:35.983+0000 7fe839ffb700 1 --2- 172.21.15.132:0/789592841 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fe83002ff10 0x7fe8300323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:17:35.987 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:35.983+0000 7fe839ffb700 1 -- 172.21.15.132:0/789592841 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe834040eb0 con 0x7fe8440f47c0 2024-02-20T00:17:35.987 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:35.983+0000 7fe848d76700 1 --2- 172.21.15.132:0/789592841 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fe83002ff10 0x7fe8300323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:17:35.987 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:35.983+0000 7fe848d76700 1 --2- 172.21.15.132:0/789592841 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fe83002ff10 0x7fe8300323d0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7fe8400096a0 tx=0x7fe840006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:17:35.990 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:35.988+0000 7fe839ffb700 1 -- 172.21.15.132:0/789592841 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe834014420 con 0x7fe8440f47c0 2024-02-20T00:17:36.196 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:36.194+0000 7fe84b7db700 1 -- 172.21.15.132:0/789592841 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe844108180 con 0x7fe8440f47c0 2024-02-20T00:17:36.197 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:36.195+0000 7fe839ffb700 1 -- 172.21.15.132:0/789592841 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe834010230 con 0x7fe8440f47c0 2024-02-20T00:17:36.197 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:17:36.197 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:17:36.198 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:36.196+0000 7fe84b7db700 1 -- 172.21.15.132:0/789592841 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fe83002ff10 msgr2=0x7fe8300323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:17:36.199 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:36.196+0000 7fe84b7db700 1 --2- 172.21.15.132:0/789592841 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fe83002ff10 0x7fe8300323d0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7fe8400096a0 tx=0x7fe840006b40).stop 2024-02-20T00:17:36.199 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:36.196+0000 7fe84b7db700 1 -- 172.21.15.132:0/789592841 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe8440f47c0 msgr2=0x7fe8441067d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:17:36.199 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:36.196+0000 7fe84b7db700 1 --2- 172.21.15.132:0/789592841 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe8440f47c0 0x7fe8441067d0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0x7fe834005010 tx=0x7fe83400b040).stop 2024-02-20T00:17:36.199 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:36.196+0000 7fe84b7db700 1 -- 172.21.15.132:0/789592841 shutdown_connections 2024-02-20T00:17:36.199 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:36.196+0000 7fe84b7db700 1 --2- 172.21.15.132:0/789592841 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fe83002ff10 0x7fe8300323d0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:17:36.199 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:36.196+0000 7fe84b7db700 1 --2- 172.21.15.132:0/789592841 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe8440f47c0 0x7fe8441067d0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:17:36.199 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:36.196+0000 7fe84b7db700 1 -- 172.21.15.132:0/789592841 >> 172.21.15.132:0/789592841 conn(0x7fe8440f02e0 msgr2=0x7fe8440f0d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:17:36.199 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:36.196+0000 7fe84b7db700 1 -- 172.21.15.132:0/789592841 shutdown_connections 2024-02-20T00:17:36.199 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:36.196+0000 7fe84b7db700 1 -- 172.21.15.132:0/789592841 wait complete. 2024-02-20T00:17:36.200 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:17:36.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:17:35 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:17:34.907644+0000 mgr.smithi084.bmqrnf (mgr.14162) 237 : cluster [DBG] pgmap v223: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:17:37.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:17:36 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:17:36.196211+0000 mon.smithi084 (mon.0) 253 : audit [DBG] from='client.? 172.21.15.132:0/789592841' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:17:37.906 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:17:37.906 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:17:38.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:17:37 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:17:36.907935+0000 mgr.smithi084.bmqrnf (mgr.14162) 238 : cluster [DBG] pgmap v224: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:17:39.743 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:39.739+0000 7f87fe25c700 1 -- 172.21.15.132:0/4179407777 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f87f80f6480 msgr2=0x7f87f80f68a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:17:39.743 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:39.739+0000 7f87fe25c700 1 --2- 172.21.15.132:0/4179407777 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f87f80f6480 0x7f87f80f68a0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 rx=0x7f87ec009fb0 tx=0x7f87ec009b30).stop 2024-02-20T00:17:39.743 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:39.739+0000 7f87fe25c700 1 -- 172.21.15.132:0/4179407777 shutdown_connections 2024-02-20T00:17:39.743 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:39.739+0000 7f87fe25c700 1 --2- 172.21.15.132:0/4179407777 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f87f80f6480 0x7f87f80f68a0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:17:39.743 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:39.739+0000 7f87fe25c700 1 -- 172.21.15.132:0/4179407777 >> 172.21.15.132:0/4179407777 conn(0x7f87f80f1c00 msgr2=0x7f87f80f4020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:17:39.743 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:39.740+0000 7f87fe25c700 1 -- 172.21.15.132:0/4179407777 shutdown_connections 2024-02-20T00:17:39.743 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:39.740+0000 7f87fe25c700 1 -- 172.21.15.132:0/4179407777 wait complete. 2024-02-20T00:17:39.744 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:39.740+0000 7f87fe25c700 1 Processor -- start 2024-02-20T00:17:39.744 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:39.740+0000 7f87fe25c700 1 -- start start 2024-02-20T00:17:39.744 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:39.740+0000 7f87fe25c700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f87f80f6480 0x7f87f81067f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:17:39.744 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:39.740+0000 7f87fe25c700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87f8106ce0 con 0x7f87f80f6480 2024-02-20T00:17:39.744 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:39.741+0000 7f87f77fe700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f87f80f6480 0x7f87f81067f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:17:39.744 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:39.741+0000 7f87f77fe700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f87f80f6480 0x7f87f81067f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:49360/0 (socket says 172.21.15.132:49360) 2024-02-20T00:17:39.744 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:39.741+0000 7f87f77fe700 1 -- 172.21.15.132:0/1937226307 learned_addr learned my addr 172.21.15.132:0/1937226307 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:17:39.745 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:39.741+0000 7f87f77fe700 1 -- 172.21.15.132:0/1937226307 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87ec005040 con 0x7f87f80f6480 2024-02-20T00:17:39.746 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:39.741+0000 7f87f77fe700 1 --2- 172.21.15.132:0/1937226307 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f87f80f6480 0x7f87f81067f0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7f87ec005010 tx=0x7f87ec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:17:39.746 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:39.741+0000 7f87dffff700 1 -- 172.21.15.132:0/1937226307 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f87ec009320 con 0x7f87f80f6480 2024-02-20T00:17:39.746 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:39.741+0000 7f87fe25c700 1 -- 172.21.15.132:0/1937226307 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87f8106ee0 con 0x7f87f80f6480 2024-02-20T00:17:39.746 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:39.741+0000 7f87dffff700 1 -- 172.21.15.132:0/1937226307 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f87ec013380 con 0x7f87f80f6480 2024-02-20T00:17:39.746 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:39.741+0000 7f87dffff700 1 -- 172.21.15.132:0/1937226307 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f87ec014e50 con 0x7f87f80f6480 2024-02-20T00:17:39.746 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:39.741+0000 7f87fe25c700 1 -- 172.21.15.132:0/1937226307 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87f8107370 con 0x7f87f80f6480 2024-02-20T00:17:39.746 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:39.742+0000 7f87dffff700 1 -- 172.21.15.132:0/1937226307 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f87ec014420 con 0x7f87f80f6480 2024-02-20T00:17:39.746 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:39.742+0000 7f87fe25c700 1 -- 172.21.15.132:0/1937226307 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87f8045ad0 con 0x7f87f80f6480 2024-02-20T00:17:39.746 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:39.742+0000 7f87dffff700 1 --2- 172.21.15.132:0/1937226307 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f87e002ff10 0x7f87e00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:17:39.746 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:39.742+0000 7f87dffff700 1 -- 172.21.15.132:0/1937226307 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f87ec040f40 con 0x7f87f80f6480 2024-02-20T00:17:39.746 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:39.743+0000 7f87f6ffd700 1 --2- 172.21.15.132:0/1937226307 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f87e002ff10 0x7f87e00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:17:39.746 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:39.743+0000 7f87f6ffd700 1 --2- 172.21.15.132:0/1937226307 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f87e002ff10 0x7f87e00323d0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f87e80096a0 tx=0x7f87e8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:17:39.749 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:39.747+0000 7f87dffff700 1 -- 172.21.15.132:0/1937226307 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f87ec0095f0 con 0x7f87f80f6480 2024-02-20T00:17:39.954 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:39.951+0000 7f87fe25c700 1 -- 172.21.15.132:0/1937226307 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f87f80fc090 con 0x7f87f80f6480 2024-02-20T00:17:39.955 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:39.952+0000 7f87dffff700 1 -- 172.21.15.132:0/1937226307 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f87ec01db70 con 0x7f87f80f6480 2024-02-20T00:17:39.955 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:17:39.955 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:17:39.956 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:39.954+0000 7f87fe25c700 1 -- 172.21.15.132:0/1937226307 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f87e002ff10 msgr2=0x7f87e00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:17:39.957 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:39.954+0000 7f87fe25c700 1 --2- 172.21.15.132:0/1937226307 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f87e002ff10 0x7f87e00323d0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f87e80096a0 tx=0x7f87e8006b40).stop 2024-02-20T00:17:39.957 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:39.954+0000 7f87fe25c700 1 -- 172.21.15.132:0/1937226307 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f87f80f6480 msgr2=0x7f87f81067f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:17:39.957 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:39.954+0000 7f87fe25c700 1 --2- 172.21.15.132:0/1937226307 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f87f80f6480 0x7f87f81067f0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7f87ec005010 tx=0x7f87ec00b040).stop 2024-02-20T00:17:39.957 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:39.954+0000 7f87fe25c700 1 -- 172.21.15.132:0/1937226307 shutdown_connections 2024-02-20T00:17:39.957 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:39.954+0000 7f87fe25c700 1 --2- 172.21.15.132:0/1937226307 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f87e002ff10 0x7f87e00323d0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:17:39.957 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:39.954+0000 7f87fe25c700 1 --2- 172.21.15.132:0/1937226307 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f87f80f6480 0x7f87f81067f0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:17:39.957 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:39.954+0000 7f87fe25c700 1 -- 172.21.15.132:0/1937226307 >> 172.21.15.132:0/1937226307 conn(0x7f87f80f1c00 msgr2=0x7f87f80f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:17:39.958 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:39.954+0000 7f87fe25c700 1 -- 172.21.15.132:0/1937226307 shutdown_connections 2024-02-20T00:17:39.958 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:39.954+0000 7f87fe25c700 1 -- 172.21.15.132:0/1937226307 wait complete. 2024-02-20T00:17:39.958 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:17:40.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:17:39 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:17:38.908213+0000 mgr.smithi084.bmqrnf (mgr.14162) 239 : cluster [DBG] pgmap v225: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:17:40.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:17:39 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:17:39.953719+0000 mon.smithi084 (mon.0) 254 : audit [DBG] from='client.? 172.21.15.132:0/1937226307' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:17:41.606 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:17:41.606 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:17:42.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:17:41 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:17:40.908546+0000 mgr.smithi084.bmqrnf (mgr.14162) 240 : cluster [DBG] pgmap v226: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:17:43.503 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:43.491+0000 7fdef7d67700 1 -- 172.21.15.132:0/884567253 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdef00f5410 msgr2=0x7fdef00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:17:43.503 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:43.491+0000 7fdef7d67700 1 --2- 172.21.15.132:0/884567253 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdef00f5410 0x7fdef00f5830 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 rx=0x7fdee0009fb0 tx=0x7fdee0009b30).stop 2024-02-20T00:17:43.503 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:43.491+0000 7fdef7d67700 1 -- 172.21.15.132:0/884567253 shutdown_connections 2024-02-20T00:17:43.503 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:43.491+0000 7fdef7d67700 1 --2- 172.21.15.132:0/884567253 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdef00f5410 0x7fdef00f5830 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:17:43.503 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:43.491+0000 7fdef7d67700 1 -- 172.21.15.132:0/884567253 >> 172.21.15.132:0/884567253 conn(0x7fdef00f0b70 msgr2=0x7fdef00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:17:43.503 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:43.491+0000 7fdef7d67700 1 -- 172.21.15.132:0/884567253 shutdown_connections 2024-02-20T00:17:43.503 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:43.491+0000 7fdef7d67700 1 -- 172.21.15.132:0/884567253 wait complete. 2024-02-20T00:17:43.503 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:43.492+0000 7fdef7d67700 1 Processor -- start 2024-02-20T00:17:43.504 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:43.492+0000 7fdef7d67700 1 -- start start 2024-02-20T00:17:43.504 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:43.492+0000 7fdef7d67700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdef00f5410 0x7fdef0105710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:17:43.504 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:43.492+0000 7fdef7d67700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdef0105c00 con 0x7fdef00f5410 2024-02-20T00:17:43.504 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:43.492+0000 7fdef5b03700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdef00f5410 0x7fdef0105710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:17:43.504 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:43.492+0000 7fdef5b03700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdef00f5410 0x7fdef0105710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:49368/0 (socket says 172.21.15.132:49368) 2024-02-20T00:17:43.504 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:43.492+0000 7fdef5b03700 1 -- 172.21.15.132:0/3640048840 learned_addr learned my addr 172.21.15.132:0/3640048840 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:17:43.504 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:43.492+0000 7fdef5b03700 1 -- 172.21.15.132:0/3640048840 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdee0005040 con 0x7fdef00f5410 2024-02-20T00:17:43.504 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:43.493+0000 7fdef5b03700 1 --2- 172.21.15.132:0/3640048840 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdef00f5410 0x7fdef0105710 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7fdee0005010 tx=0x7fdee000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:17:43.504 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:43.493+0000 7fdee67fc700 1 -- 172.21.15.132:0/3640048840 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdee0009320 con 0x7fdef00f5410 2024-02-20T00:17:43.504 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:43.493+0000 7fdee67fc700 1 -- 172.21.15.132:0/3640048840 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdee0013380 con 0x7fdef00f5410 2024-02-20T00:17:43.504 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:43.493+0000 7fdef7d67700 1 -- 172.21.15.132:0/3640048840 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdef0105e00 con 0x7fdef00f5410 2024-02-20T00:17:43.504 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:43.493+0000 7fdee67fc700 1 -- 172.21.15.132:0/3640048840 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdee0014e50 con 0x7fdef00f5410 2024-02-20T00:17:43.505 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:43.493+0000 7fdef7d67700 1 -- 172.21.15.132:0/3640048840 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdef0106290 con 0x7fdef00f5410 2024-02-20T00:17:43.505 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:43.493+0000 7fdee67fc700 1 -- 172.21.15.132:0/3640048840 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdee00263a0 con 0x7fdef00f5410 2024-02-20T00:17:43.505 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:43.494+0000 7fdef7d67700 1 -- 172.21.15.132:0/3640048840 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdef0045ad0 con 0x7fdef00f5410 2024-02-20T00:17:43.505 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:43.494+0000 7fdee67fc700 1 --2- 172.21.15.132:0/3640048840 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fdedc02ff10 0x7fdedc0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:17:43.505 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:43.494+0000 7fdee67fc700 1 -- 172.21.15.132:0/3640048840 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdee0041250 con 0x7fdef00f5410 2024-02-20T00:17:43.505 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:43.494+0000 7fdef5302700 1 --2- 172.21.15.132:0/3640048840 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fdedc02ff10 0x7fdedc0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:17:43.505 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:43.494+0000 7fdef5302700 1 --2- 172.21.15.132:0/3640048840 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fdedc02ff10 0x7fdedc0323d0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7fdeec0096a0 tx=0x7fdeec006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:17:43.505 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:43.498+0000 7fdee67fc700 1 -- 172.21.15.132:0/3640048840 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdee001cb10 con 0x7fdef00f5410 2024-02-20T00:17:43.705 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:43.702+0000 7fdef7d67700 1 -- 172.21.15.132:0/3640048840 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdef0107140 con 0x7fdef00f5410 2024-02-20T00:17:43.705 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:43.703+0000 7fdee67fc700 1 -- 172.21.15.132:0/3640048840 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fdee001cb10 con 0x7fdef00f5410 2024-02-20T00:17:43.705 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:17:43.706 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:17:43.707 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:43.704+0000 7fdef7d67700 1 -- 172.21.15.132:0/3640048840 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fdedc02ff10 msgr2=0x7fdedc0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:17:43.707 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:43.704+0000 7fdef7d67700 1 --2- 172.21.15.132:0/3640048840 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fdedc02ff10 0x7fdedc0323d0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7fdeec0096a0 tx=0x7fdeec006b40).stop 2024-02-20T00:17:43.707 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:43.704+0000 7fdef7d67700 1 -- 172.21.15.132:0/3640048840 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdef00f5410 msgr2=0x7fdef0105710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:17:43.707 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:43.704+0000 7fdef7d67700 1 --2- 172.21.15.132:0/3640048840 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdef00f5410 0x7fdef0105710 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7fdee0005010 tx=0x7fdee000b040).stop 2024-02-20T00:17:43.707 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:43.704+0000 7fdef7d67700 1 -- 172.21.15.132:0/3640048840 shutdown_connections 2024-02-20T00:17:43.707 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:43.704+0000 7fdef7d67700 1 --2- 172.21.15.132:0/3640048840 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fdedc02ff10 0x7fdedc0323d0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:17:43.707 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:43.704+0000 7fdef7d67700 1 --2- 172.21.15.132:0/3640048840 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdef00f5410 0x7fdef0105710 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:17:43.707 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:43.704+0000 7fdef7d67700 1 -- 172.21.15.132:0/3640048840 >> 172.21.15.132:0/3640048840 conn(0x7fdef00f0b70 msgr2=0x7fdef00f15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:17:43.707 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:43.704+0000 7fdef7d67700 1 -- 172.21.15.132:0/3640048840 shutdown_connections 2024-02-20T00:17:43.707 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:43.704+0000 7fdef7d67700 1 -- 172.21.15.132:0/3640048840 wait complete. 2024-02-20T00:17:43.708 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:17:44.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:17:43 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:17:42.908905+0000 mgr.smithi084.bmqrnf (mgr.14162) 241 : cluster [DBG] pgmap v227: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:17:44.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:17:43 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:17:43.704572+0000 mon.smithi084 (mon.0) 255 : audit [DBG] from='client.? 172.21.15.132:0/3640048840' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:17:45.464 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:17:45.464 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:17:46.359 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:17:45 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:17:44.909216+0000 mgr.smithi084.bmqrnf (mgr.14162) 242 : cluster [DBG] pgmap v228: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:17:47.438 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:47.435+0000 7fc343fff700 1 -- 172.21.15.132:0/3780004118 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc3440f62a0 msgr2=0x7fc3440f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:17:47.439 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:47.435+0000 7fc343fff700 1 --2- 172.21.15.132:0/3780004118 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc3440f62a0 0x7fc3440f66c0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 rx=0x7fc334009fb0 tx=0x7fc334009b30).stop 2024-02-20T00:17:47.439 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:47.435+0000 7fc343fff700 1 -- 172.21.15.132:0/3780004118 shutdown_connections 2024-02-20T00:17:47.439 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:47.435+0000 7fc343fff700 1 --2- 172.21.15.132:0/3780004118 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc3440f62a0 0x7fc3440f66c0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:17:47.439 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:47.435+0000 7fc343fff700 1 -- 172.21.15.132:0/3780004118 >> 172.21.15.132:0/3780004118 conn(0x7fc3440f1ac0 msgr2=0x7fc3440f3ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:17:47.439 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:47.435+0000 7fc343fff700 1 -- 172.21.15.132:0/3780004118 shutdown_connections 2024-02-20T00:17:47.439 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:47.436+0000 7fc343fff700 1 -- 172.21.15.132:0/3780004118 wait complete. 2024-02-20T00:17:47.439 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:47.436+0000 7fc343fff700 1 Processor -- start 2024-02-20T00:17:47.439 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:47.436+0000 7fc343fff700 1 -- start start 2024-02-20T00:17:47.439 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:47.436+0000 7fc343fff700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc3440f62a0 0x7fc344106570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:17:47.439 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:47.436+0000 7fc343fff700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc344106a60 con 0x7fc3440f62a0 2024-02-20T00:17:47.439 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:47.437+0000 7fc342ffd700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc3440f62a0 0x7fc344106570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:17:47.440 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:47.437+0000 7fc342ffd700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc3440f62a0 0x7fc344106570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:50918/0 (socket says 172.21.15.132:50918) 2024-02-20T00:17:47.440 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:47.437+0000 7fc342ffd700 1 -- 172.21.15.132:0/3033704055 learned_addr learned my addr 172.21.15.132:0/3033704055 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:17:47.441 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:47.437+0000 7fc342ffd700 1 -- 172.21.15.132:0/3033704055 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc334005040 con 0x7fc3440f62a0 2024-02-20T00:17:47.441 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:47.437+0000 7fc342ffd700 1 --2- 172.21.15.132:0/3033704055 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc3440f62a0 0x7fc344106570 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0x7fc3340089f0 tx=0x7fc33400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:17:47.442 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:47.437+0000 7fc32b7fe700 1 -- 172.21.15.132:0/3033704055 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc334009320 con 0x7fc3440f62a0 2024-02-20T00:17:47.442 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:47.437+0000 7fc343fff700 1 -- 172.21.15.132:0/3033704055 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc344106c60 con 0x7fc3440f62a0 2024-02-20T00:17:47.442 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:47.437+0000 7fc32b7fe700 1 -- 172.21.15.132:0/3033704055 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc334013380 con 0x7fc3440f62a0 2024-02-20T00:17:47.442 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:47.437+0000 7fc32b7fe700 1 -- 172.21.15.132:0/3033704055 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc334014e20 con 0x7fc3440f62a0 2024-02-20T00:17:47.442 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:47.437+0000 7fc343fff700 1 -- 172.21.15.132:0/3033704055 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3441070f0 con 0x7fc3440f62a0 2024-02-20T00:17:47.442 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:47.438+0000 7fc32b7fe700 1 -- 172.21.15.132:0/3033704055 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc3340095f0 con 0x7fc3440f62a0 2024-02-20T00:17:47.442 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:47.438+0000 7fc32b7fe700 1 --2- 172.21.15.132:0/3033704055 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fc32c02ff10 0x7fc32c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:17:47.442 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:47.438+0000 7fc343fff700 1 -- 172.21.15.132:0/3033704055 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc344045ad0 con 0x7fc3440f62a0 2024-02-20T00:17:47.442 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:47.438+0000 7fc32b7fe700 1 -- 172.21.15.132:0/3033704055 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc334040f50 con 0x7fc3440f62a0 2024-02-20T00:17:47.442 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:47.438+0000 7fc3427fc700 1 --2- 172.21.15.132:0/3033704055 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fc32c02ff10 0x7fc32c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:17:47.442 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:47.439+0000 7fc3427fc700 1 --2- 172.21.15.132:0/3033704055 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fc32c02ff10 0x7fc32c0323d0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7fc3380096a0 tx=0x7fc338006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:17:47.445 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:47.443+0000 7fc32b7fe700 1 -- 172.21.15.132:0/3033704055 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc33401d980 con 0x7fc3440f62a0 2024-02-20T00:17:47.650 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:47.647+0000 7fc343fff700 1 -- 172.21.15.132:0/3033704055 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc344107fa0 con 0x7fc3440f62a0 2024-02-20T00:17:47.650 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:47.648+0000 7fc32b7fe700 1 -- 172.21.15.132:0/3033704055 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc334015040 con 0x7fc3440f62a0 2024-02-20T00:17:47.651 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:17:47.651 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:17:47.652 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:47.650+0000 7fc343fff700 1 -- 172.21.15.132:0/3033704055 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fc32c02ff10 msgr2=0x7fc32c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:17:47.652 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:47.650+0000 7fc343fff700 1 --2- 172.21.15.132:0/3033704055 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fc32c02ff10 0x7fc32c0323d0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7fc3380096a0 tx=0x7fc338006b40).stop 2024-02-20T00:17:47.652 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:47.650+0000 7fc343fff700 1 -- 172.21.15.132:0/3033704055 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc3440f62a0 msgr2=0x7fc344106570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:17:47.652 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:47.650+0000 7fc343fff700 1 --2- 172.21.15.132:0/3033704055 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc3440f62a0 0x7fc344106570 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0x7fc3340089f0 tx=0x7fc33400b040).stop 2024-02-20T00:17:47.653 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:47.650+0000 7fc343fff700 1 -- 172.21.15.132:0/3033704055 shutdown_connections 2024-02-20T00:17:47.653 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:47.650+0000 7fc343fff700 1 --2- 172.21.15.132:0/3033704055 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fc32c02ff10 0x7fc32c0323d0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:17:47.653 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:47.650+0000 7fc343fff700 1 --2- 172.21.15.132:0/3033704055 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc3440f62a0 0x7fc344106570 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:17:47.653 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:47.650+0000 7fc343fff700 1 -- 172.21.15.132:0/3033704055 >> 172.21.15.132:0/3033704055 conn(0x7fc3440f1ac0 msgr2=0x7fc3440f24b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:17:47.653 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:47.650+0000 7fc343fff700 1 -- 172.21.15.132:0/3033704055 shutdown_connections 2024-02-20T00:17:47.653 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:47.650+0000 7fc343fff700 1 -- 172.21.15.132:0/3033704055 wait complete. 2024-02-20T00:17:47.653 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:17:48.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:17:47 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:17:46.909564+0000 mgr.smithi084.bmqrnf (mgr.14162) 243 : cluster [DBG] pgmap v229: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:17:48.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:17:47 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:17:47.649803+0000 mon.smithi084 (mon.0) 256 : audit [DBG] from='client.? 172.21.15.132:0/3033704055' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:17:49.382 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:17:49.382 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:17:50.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:17:49 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:17:48.909800+0000 mgr.smithi084.bmqrnf (mgr.14162) 244 : cluster [DBG] pgmap v230: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:17:51.316 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:51.314+0000 7f411d41b700 1 -- 172.21.15.132:0/2511373515 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f41180f5410 msgr2=0x7f41180f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:17:51.316 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:51.314+0000 7f411d41b700 1 --2- 172.21.15.132:0/2511373515 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f41180f5410 0x7f41180f5830 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 rx=0x7f4108009fb0 tx=0x7f4108009b30).stop 2024-02-20T00:17:51.316 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:51.314+0000 7f411d41b700 1 -- 172.21.15.132:0/2511373515 shutdown_connections 2024-02-20T00:17:51.316 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:51.314+0000 7f411d41b700 1 --2- 172.21.15.132:0/2511373515 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f41180f5410 0x7f41180f5830 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:17:51.316 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:51.314+0000 7f411d41b700 1 -- 172.21.15.132:0/2511373515 >> 172.21.15.132:0/2511373515 conn(0x7f41180f0b70 msgr2=0x7f41180f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:17:51.317 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:51.314+0000 7f411d41b700 1 -- 172.21.15.132:0/2511373515 shutdown_connections 2024-02-20T00:17:51.317 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:51.314+0000 7f411d41b700 1 -- 172.21.15.132:0/2511373515 wait complete. 2024-02-20T00:17:51.319 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:51.314+0000 7f411d41b700 1 Processor -- start 2024-02-20T00:17:51.319 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:51.314+0000 7f411d41b700 1 -- start start 2024-02-20T00:17:51.319 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:51.314+0000 7f411d41b700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f41180f5410 0x7f41181067f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:17:51.319 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:51.315+0000 7f411d41b700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4118106ce0 con 0x7f41180f5410 2024-02-20T00:17:51.319 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:51.315+0000 7f4116ffd700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f41180f5410 0x7f41181067f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:17:51.319 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:51.315+0000 7f4116ffd700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f41180f5410 0x7f41181067f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:50926/0 (socket says 172.21.15.132:50926) 2024-02-20T00:17:51.319 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:51.315+0000 7f4116ffd700 1 -- 172.21.15.132:0/3574295106 learned_addr learned my addr 172.21.15.132:0/3574295106 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:17:51.319 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:51.315+0000 7f4116ffd700 1 -- 172.21.15.132:0/3574295106 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4108005040 con 0x7f41180f5410 2024-02-20T00:17:51.319 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:51.315+0000 7f4116ffd700 1 --2- 172.21.15.132:0/3574295106 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f41180f5410 0x7f41181067f0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0x7f4108005010 tx=0x7f410800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:17:51.319 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:51.315+0000 7f40ff7fe700 1 -- 172.21.15.132:0/3574295106 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4108009320 con 0x7f41180f5410 2024-02-20T00:17:51.319 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:51.315+0000 7f411d41b700 1 -- 172.21.15.132:0/3574295106 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4118106ee0 con 0x7f41180f5410 2024-02-20T00:17:51.320 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:51.315+0000 7f40ff7fe700 1 -- 172.21.15.132:0/3574295106 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4108013380 con 0x7f41180f5410 2024-02-20T00:17:51.320 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:51.315+0000 7f40ff7fe700 1 -- 172.21.15.132:0/3574295106 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4108014e50 con 0x7f41180f5410 2024-02-20T00:17:51.320 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:51.315+0000 7f411d41b700 1 -- 172.21.15.132:0/3574295106 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4118107370 con 0x7f41180f5410 2024-02-20T00:17:51.320 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:51.316+0000 7f40ff7fe700 1 -- 172.21.15.132:0/3574295106 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4108014420 con 0x7f41180f5410 2024-02-20T00:17:51.320 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:51.316+0000 7f411d41b700 1 -- 172.21.15.132:0/3574295106 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4118045ad0 con 0x7f41180f5410 2024-02-20T00:17:51.320 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:51.316+0000 7f40ff7fe700 1 --2- 172.21.15.132:0/3574295106 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f410002ff10 0x7f41000323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:17:51.320 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:51.316+0000 7f40ff7fe700 1 -- 172.21.15.132:0/3574295106 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4108040f70 con 0x7f41180f5410 2024-02-20T00:17:51.320 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:51.316+0000 7f41167fc700 1 --2- 172.21.15.132:0/3574295106 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f410002ff10 0x7f41000323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:17:51.320 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:51.317+0000 7f41167fc700 1 --2- 172.21.15.132:0/3574295106 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f410002ff10 0x7f41000323d0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7f410c0096a0 tx=0x7f410c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:17:51.323 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:51.322+0000 7f40ff7fe700 1 -- 172.21.15.132:0/3574295106 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f41080095f0 con 0x7f41180f5410 2024-02-20T00:17:51.527 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:51.526+0000 7f411d41b700 1 -- 172.21.15.132:0/3574295106 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4118108220 con 0x7f41180f5410 2024-02-20T00:17:51.528 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:51.526+0000 7f40ff7fe700 1 -- 172.21.15.132:0/3574295106 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f410801db70 con 0x7f41180f5410 2024-02-20T00:17:51.528 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:17:51.528 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:17:51.530 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:51.528+0000 7f411d41b700 1 -- 172.21.15.132:0/3574295106 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f410002ff10 msgr2=0x7f41000323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:17:51.530 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:51.528+0000 7f411d41b700 1 --2- 172.21.15.132:0/3574295106 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f410002ff10 0x7f41000323d0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7f410c0096a0 tx=0x7f410c006b40).stop 2024-02-20T00:17:51.530 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:51.528+0000 7f411d41b700 1 -- 172.21.15.132:0/3574295106 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f41180f5410 msgr2=0x7f41181067f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:17:51.530 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:51.528+0000 7f411d41b700 1 --2- 172.21.15.132:0/3574295106 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f41180f5410 0x7f41181067f0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0x7f4108005010 tx=0x7f410800b040).stop 2024-02-20T00:17:51.530 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:51.528+0000 7f411d41b700 1 -- 172.21.15.132:0/3574295106 shutdown_connections 2024-02-20T00:17:51.530 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:51.528+0000 7f411d41b700 1 --2- 172.21.15.132:0/3574295106 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f410002ff10 0x7f41000323d0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:17:51.530 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:51.528+0000 7f411d41b700 1 --2- 172.21.15.132:0/3574295106 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f41180f5410 0x7f41181067f0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:17:51.530 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:51.528+0000 7f411d41b700 1 -- 172.21.15.132:0/3574295106 >> 172.21.15.132:0/3574295106 conn(0x7f41180f0b70 msgr2=0x7f411805fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:17:51.530 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:51.528+0000 7f411d41b700 1 -- 172.21.15.132:0/3574295106 shutdown_connections 2024-02-20T00:17:51.530 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:51.528+0000 7f411d41b700 1 -- 172.21.15.132:0/3574295106 wait complete. 2024-02-20T00:17:51.531 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:17:52.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:17:51 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:17:50.910159+0000 mgr.smithi084.bmqrnf (mgr.14162) 245 : cluster [DBG] pgmap v231: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:17:52.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:17:51 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:17:51.526977+0000 mon.smithi084 (mon.0) 257 : audit [DBG] from='client.? 172.21.15.132:0/3574295106' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:17:53.150 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:17:53.151 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:17:54.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:17:53 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:17:52.910555+0000 mgr.smithi084.bmqrnf (mgr.14162) 246 : cluster [DBG] pgmap v232: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:17:55.341 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:55.301+0000 7f443ea80700 1 -- 172.21.15.132:0/3350135880 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f44380f5410 msgr2=0x7f44380f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:17:55.341 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:55.301+0000 7f443ea80700 1 --2- 172.21.15.132:0/3350135880 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f44380f5410 0x7f44380f5830 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 rx=0x7f442c009fb0 tx=0x7f442c009b30).stop 2024-02-20T00:17:55.341 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:55.301+0000 7f443ea80700 1 -- 172.21.15.132:0/3350135880 shutdown_connections 2024-02-20T00:17:55.342 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:55.301+0000 7f443ea80700 1 --2- 172.21.15.132:0/3350135880 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f44380f5410 0x7f44380f5830 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:17:55.342 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:55.301+0000 7f443ea80700 1 -- 172.21.15.132:0/3350135880 >> 172.21.15.132:0/3350135880 conn(0x7f44380f0b70 msgr2=0x7f44380f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:17:55.342 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:55.301+0000 7f443ea80700 1 -- 172.21.15.132:0/3350135880 shutdown_connections 2024-02-20T00:17:55.342 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:55.301+0000 7f443ea80700 1 -- 172.21.15.132:0/3350135880 wait complete. 2024-02-20T00:17:55.342 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:55.302+0000 7f443ea80700 1 Processor -- start 2024-02-20T00:17:55.342 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:55.302+0000 7f443ea80700 1 -- start start 2024-02-20T00:17:55.342 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:55.302+0000 7f443ea80700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f44380f5410 0x7f44380fd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:17:55.342 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:55.302+0000 7f443ea80700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44380fb960 con 0x7f44380f5410 2024-02-20T00:17:55.342 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:55.302+0000 7f443c81c700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f44380f5410 0x7f44380fd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:17:55.342 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:55.302+0000 7f443c81c700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f44380f5410 0x7f44380fd4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:50944/0 (socket says 172.21.15.132:50944) 2024-02-20T00:17:55.342 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:55.302+0000 7f443c81c700 1 -- 172.21.15.132:0/3587288413 learned_addr learned my addr 172.21.15.132:0/3587288413 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:17:55.342 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:55.303+0000 7f443c81c700 1 -- 172.21.15.132:0/3587288413 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f442c005040 con 0x7f44380f5410 2024-02-20T00:17:55.342 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:55.303+0000 7f443c81c700 1 --2- 172.21.15.132:0/3587288413 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f44380f5410 0x7f44380fd4e0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0x7f442c0089f0 tx=0x7f442c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:17:55.343 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:55.303+0000 7f44357fa700 1 -- 172.21.15.132:0/3587288413 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f442c009320 con 0x7f44380f5410 2024-02-20T00:17:55.343 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:55.303+0000 7f443ea80700 1 -- 172.21.15.132:0/3587288413 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44380fbb00 con 0x7f44380f5410 2024-02-20T00:17:55.343 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:55.303+0000 7f44357fa700 1 -- 172.21.15.132:0/3587288413 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f442c013380 con 0x7f44380f5410 2024-02-20T00:17:55.343 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:55.303+0000 7f44357fa700 1 -- 172.21.15.132:0/3587288413 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f442c01ed80 con 0x7f44380f5410 2024-02-20T00:17:55.343 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:55.303+0000 7f443ea80700 1 -- 172.21.15.132:0/3587288413 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44380fbf90 con 0x7f44380f5410 2024-02-20T00:17:55.343 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:55.304+0000 7f44357fa700 1 -- 172.21.15.132:0/3587288413 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f442c01e420 con 0x7f44380f5410 2024-02-20T00:17:55.343 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:55.304+0000 7f443ea80700 1 -- 172.21.15.132:0/3587288413 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4438059100 con 0x7f44380f5410 2024-02-20T00:17:55.343 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:55.304+0000 7f44357fa700 1 --2- 172.21.15.132:0/3587288413 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f442802ff60 0x7f4428032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:17:55.343 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:55.304+0000 7f44357fa700 1 -- 172.21.15.132:0/3587288413 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f442c040fa0 con 0x7f44380f5410 2024-02-20T00:17:55.343 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:55.304+0000 7f4437fff700 1 --2- 172.21.15.132:0/3587288413 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f442802ff60 0x7f4428032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:17:55.343 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:55.305+0000 7f4437fff700 1 --2- 172.21.15.132:0/3587288413 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f442802ff60 0x7f4428032420 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f44240096a0 tx=0x7f4424006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:17:55.343 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:55.309+0000 7f44357fa700 1 -- 172.21.15.132:0/3587288413 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f442c0095f0 con 0x7f44380f5410 2024-02-20T00:17:55.518 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:55.516+0000 7f443ea80700 1 -- 172.21.15.132:0/3587288413 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f44380fafa0 con 0x7f44380f5410 2024-02-20T00:17:55.518 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:55.517+0000 7f44357fa700 1 -- 172.21.15.132:0/3587288413 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f442c01cb10 con 0x7f44380f5410 2024-02-20T00:17:55.519 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:17:55.519 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:17:55.520 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:55.519+0000 7f443ea80700 1 -- 172.21.15.132:0/3587288413 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f442802ff60 msgr2=0x7f4428032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:17:55.520 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:55.519+0000 7f443ea80700 1 --2- 172.21.15.132:0/3587288413 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f442802ff60 0x7f4428032420 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f44240096a0 tx=0x7f4424006b40).stop 2024-02-20T00:17:55.520 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:55.519+0000 7f443ea80700 1 -- 172.21.15.132:0/3587288413 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f44380f5410 msgr2=0x7f44380fd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:17:55.520 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:55.519+0000 7f443ea80700 1 --2- 172.21.15.132:0/3587288413 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f44380f5410 0x7f44380fd4e0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0x7f442c0089f0 tx=0x7f442c00b040).stop 2024-02-20T00:17:55.520 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:55.519+0000 7f443ea80700 1 -- 172.21.15.132:0/3587288413 shutdown_connections 2024-02-20T00:17:55.521 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:55.519+0000 7f443ea80700 1 --2- 172.21.15.132:0/3587288413 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f442802ff60 0x7f4428032420 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:17:55.521 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:55.519+0000 7f443ea80700 1 --2- 172.21.15.132:0/3587288413 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f44380f5410 0x7f44380fd4e0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:17:55.521 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:55.519+0000 7f443ea80700 1 -- 172.21.15.132:0/3587288413 >> 172.21.15.132:0/3587288413 conn(0x7f44380f0b70 msgr2=0x7f443805fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:17:55.521 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:55.519+0000 7f443ea80700 1 -- 172.21.15.132:0/3587288413 shutdown_connections 2024-02-20T00:17:55.521 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:55.519+0000 7f443ea80700 1 -- 172.21.15.132:0/3587288413 wait complete. 2024-02-20T00:17:55.521 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:17:56.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:17:55 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:17:54.910756+0000 mgr.smithi084.bmqrnf (mgr.14162) 247 : cluster [DBG] pgmap v233: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:17:56.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:17:55 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:17:55.517685+0000 mon.smithi084 (mon.0) 258 : audit [DBG] from='client.? 172.21.15.132:0/3587288413' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:17:57.172 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:17:57.173 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:17:58.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:17:57 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:17:56.911127+0000 mgr.smithi084.bmqrnf (mgr.14162) 248 : cluster [DBG] pgmap v234: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:17:58.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:17:57 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:17:56.977756+0000 mon.smithi084 (mon.0) 259 : audit [INF] from='mgr.14162 172.21.15.84:0/2152933216' entity='mgr.smithi084.bmqrnf' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi084.bmqrnf/mirror_snapshot_schedule"}]: dispatch 2024-02-20T00:17:58.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:17:57 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:17:56.983618+0000 mon.smithi084 (mon.0) 260 : audit [INF] from='mgr.14162 172.21.15.84:0/2152933216' entity='mgr.smithi084.bmqrnf' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi084.bmqrnf/trash_purge_schedule"}]: dispatch 2024-02-20T00:17:59.113 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:59.111+0000 7fc241f18700 1 -- 172.21.15.132:0/3239290129 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc23c0f6510 msgr2=0x7fc23c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:17:59.113 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:59.111+0000 7fc241f18700 1 --2- 172.21.15.132:0/3239290129 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc23c0f6510 0x7fc23c0f6930 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 rx=0x7fc230009fb0 tx=0x7fc230009b30).stop 2024-02-20T00:17:59.113 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:59.111+0000 7fc241f18700 1 -- 172.21.15.132:0/3239290129 shutdown_connections 2024-02-20T00:17:59.113 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:59.111+0000 7fc241f18700 1 --2- 172.21.15.132:0/3239290129 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc23c0f6510 0x7fc23c0f6930 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:17:59.113 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:59.111+0000 7fc241f18700 1 -- 172.21.15.132:0/3239290129 >> 172.21.15.132:0/3239290129 conn(0x7fc23c0f1c70 msgr2=0x7fc23c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:17:59.113 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:59.111+0000 7fc241f18700 1 -- 172.21.15.132:0/3239290129 shutdown_connections 2024-02-20T00:17:59.114 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:59.111+0000 7fc241f18700 1 -- 172.21.15.132:0/3239290129 wait complete. 2024-02-20T00:17:59.114 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:59.112+0000 7fc241f18700 1 Processor -- start 2024-02-20T00:17:59.114 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:59.112+0000 7fc241f18700 1 -- start start 2024-02-20T00:17:59.115 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:59.112+0000 7fc241f18700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc23c0f6510 0x7fc23c06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:17:59.115 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:59.112+0000 7fc241f18700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc23c069e20 con 0x7fc23c0f6510 2024-02-20T00:17:59.115 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:59.112+0000 7fc23b7fe700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc23c0f6510 0x7fc23c06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:17:59.115 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:59.112+0000 7fc23b7fe700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc23c0f6510 0x7fc23c06b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:39722/0 (socket says 172.21.15.132:39722) 2024-02-20T00:17:59.116 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:59.112+0000 7fc23b7fe700 1 -- 172.21.15.132:0/2731934030 learned_addr learned my addr 172.21.15.132:0/2731934030 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:17:59.116 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:59.113+0000 7fc23b7fe700 1 -- 172.21.15.132:0/2731934030 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc230005040 con 0x7fc23c0f6510 2024-02-20T00:17:59.116 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:59.113+0000 7fc23b7fe700 1 --2- 172.21.15.132:0/2731934030 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc23c0f6510 0x7fc23c06b9a0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0x7fc230005010 tx=0x7fc23000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:17:59.116 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:59.113+0000 7fc223fff700 1 -- 172.21.15.132:0/2731934030 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc230009320 con 0x7fc23c0f6510 2024-02-20T00:17:59.116 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:59.113+0000 7fc241f18700 1 -- 172.21.15.132:0/2731934030 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc23c069fc0 con 0x7fc23c0f6510 2024-02-20T00:17:59.116 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:59.113+0000 7fc223fff700 1 -- 172.21.15.132:0/2731934030 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc230013380 con 0x7fc23c0f6510 2024-02-20T00:17:59.116 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:59.113+0000 7fc223fff700 1 -- 172.21.15.132:0/2731934030 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc230014e50 con 0x7fc23c0f6510 2024-02-20T00:17:59.116 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:59.113+0000 7fc241f18700 1 -- 172.21.15.132:0/2731934030 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc23c06a450 con 0x7fc23c0f6510 2024-02-20T00:17:59.116 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:59.114+0000 7fc223fff700 1 -- 172.21.15.132:0/2731934030 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc2300263a0 con 0x7fc23c0f6510 2024-02-20T00:17:59.116 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:59.114+0000 7fc241f18700 1 -- 172.21.15.132:0/2731934030 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc23c059100 con 0x7fc23c0f6510 2024-02-20T00:17:59.116 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:59.114+0000 7fc223fff700 1 --2- 172.21.15.132:0/2731934030 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fc22402ff10 0x7fc2240323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:17:59.117 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:59.114+0000 7fc223fff700 1 -- 172.21.15.132:0/2731934030 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc230041140 con 0x7fc23c0f6510 2024-02-20T00:17:59.117 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:59.114+0000 7fc23affd700 1 --2- 172.21.15.132:0/2731934030 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fc22402ff10 0x7fc2240323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:17:59.117 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:59.115+0000 7fc23affd700 1 --2- 172.21.15.132:0/2731934030 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fc22402ff10 0x7fc2240323d0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7fc22c0096a0 tx=0x7fc22c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:17:59.120 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:59.119+0000 7fc223fff700 1 -- 172.21.15.132:0/2731934030 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc230026e60 con 0x7fc23c0f6510 2024-02-20T00:17:59.326 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:59.324+0000 7fc241f18700 1 -- 172.21.15.132:0/2731934030 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc23c0fc0a0 con 0x7fc23c0f6510 2024-02-20T00:17:59.326 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:59.325+0000 7fc223fff700 1 -- 172.21.15.132:0/2731934030 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc230012330 con 0x7fc23c0f6510 2024-02-20T00:17:59.327 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:17:59.327 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:17:59.328 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:59.327+0000 7fc241f18700 1 -- 172.21.15.132:0/2731934030 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fc22402ff10 msgr2=0x7fc2240323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:17:59.328 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:59.327+0000 7fc241f18700 1 --2- 172.21.15.132:0/2731934030 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fc22402ff10 0x7fc2240323d0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7fc22c0096a0 tx=0x7fc22c006b40).stop 2024-02-20T00:17:59.329 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:59.327+0000 7fc241f18700 1 -- 172.21.15.132:0/2731934030 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc23c0f6510 msgr2=0x7fc23c06b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:17:59.329 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:59.327+0000 7fc241f18700 1 --2- 172.21.15.132:0/2731934030 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc23c0f6510 0x7fc23c06b9a0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0x7fc230005010 tx=0x7fc23000b040).stop 2024-02-20T00:17:59.329 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:59.327+0000 7fc241f18700 1 -- 172.21.15.132:0/2731934030 shutdown_connections 2024-02-20T00:17:59.329 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:59.327+0000 7fc241f18700 1 --2- 172.21.15.132:0/2731934030 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fc22402ff10 0x7fc2240323d0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:17:59.329 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:59.327+0000 7fc241f18700 1 --2- 172.21.15.132:0/2731934030 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc23c0f6510 0x7fc23c06b9a0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:17:59.329 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:59.327+0000 7fc241f18700 1 -- 172.21.15.132:0/2731934030 >> 172.21.15.132:0/2731934030 conn(0x7fc23c0f1c70 msgr2=0x7fc23c0f26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:17:59.329 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:59.327+0000 7fc241f18700 1 -- 172.21.15.132:0/2731934030 shutdown_connections 2024-02-20T00:17:59.329 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:17:59.327+0000 7fc241f18700 1 -- 172.21.15.132:0/2731934030 wait complete. 2024-02-20T00:17:59.330 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:18:00.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:17:59 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:17:58.911494+0000 mgr.smithi084.bmqrnf (mgr.14162) 249 : cluster [DBG] pgmap v235: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:18:00.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:17:59 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:17:59.325923+0000 mon.smithi084 (mon.0) 261 : audit [DBG] from='client.? 172.21.15.132:0/2731934030' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:18:00.985 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:18:00.985 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:18:02.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:18:01 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:18:00.911760+0000 mgr.smithi084.bmqrnf (mgr.14162) 250 : cluster [DBG] pgmap v236: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:18:02.928 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:02.925+0000 7fdd06c66700 1 -- 172.21.15.132:0/383787794 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdd000f6510 msgr2=0x7fdd000f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:18:02.928 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:02.925+0000 7fdd06c66700 1 --2- 172.21.15.132:0/383787794 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdd000f6510 0x7fdd000f6930 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 rx=0x7fdcf4009fb0 tx=0x7fdcf4009b30).stop 2024-02-20T00:18:02.928 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:02.925+0000 7fdd06c66700 1 -- 172.21.15.132:0/383787794 shutdown_connections 2024-02-20T00:18:02.928 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:02.925+0000 7fdd06c66700 1 --2- 172.21.15.132:0/383787794 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdd000f6510 0x7fdd000f6930 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:18:02.928 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:02.925+0000 7fdd06c66700 1 -- 172.21.15.132:0/383787794 >> 172.21.15.132:0/383787794 conn(0x7fdd000f1c70 msgr2=0x7fdd000f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:18:02.928 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:02.925+0000 7fdd06c66700 1 -- 172.21.15.132:0/383787794 shutdown_connections 2024-02-20T00:18:02.928 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:02.925+0000 7fdd06c66700 1 -- 172.21.15.132:0/383787794 wait complete. 2024-02-20T00:18:02.929 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:02.926+0000 7fdd06c66700 1 Processor -- start 2024-02-20T00:18:02.929 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:02.926+0000 7fdd06c66700 1 -- start start 2024-02-20T00:18:02.929 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:02.926+0000 7fdd06c66700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdd000f6510 0x7fdd001067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:18:02.929 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:02.926+0000 7fdd06c66700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd00106cd0 con 0x7fdd000f6510 2024-02-20T00:18:02.929 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:02.926+0000 7fdd04a02700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdd000f6510 0x7fdd001067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:18:02.929 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:02.926+0000 7fdd04a02700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdd000f6510 0x7fdd001067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:39744/0 (socket says 172.21.15.132:39744) 2024-02-20T00:18:02.929 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:02.926+0000 7fdd04a02700 1 -- 172.21.15.132:0/491087411 learned_addr learned my addr 172.21.15.132:0/491087411 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:18:02.929 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:02.927+0000 7fdd04a02700 1 -- 172.21.15.132:0/491087411 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcf4005040 con 0x7fdd000f6510 2024-02-20T00:18:02.929 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:02.927+0000 7fdd04a02700 1 --2- 172.21.15.132:0/491087411 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdd000f6510 0x7fdd001067e0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0x7fdcf4005010 tx=0x7fdcf400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:18:02.930 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:02.927+0000 7fdcfd7fa700 1 -- 172.21.15.132:0/491087411 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdcf4009320 con 0x7fdd000f6510 2024-02-20T00:18:02.931 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:02.927+0000 7fdd06c66700 1 -- 172.21.15.132:0/491087411 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd00106ed0 con 0x7fdd000f6510 2024-02-20T00:18:02.931 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:02.927+0000 7fdcfd7fa700 1 -- 172.21.15.132:0/491087411 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdcf4013380 con 0x7fdd000f6510 2024-02-20T00:18:02.931 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:02.927+0000 7fdcfd7fa700 1 -- 172.21.15.132:0/491087411 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdcf4014e50 con 0x7fdd000f6510 2024-02-20T00:18:02.931 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:02.927+0000 7fdd06c66700 1 -- 172.21.15.132:0/491087411 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd00107360 con 0x7fdd000f6510 2024-02-20T00:18:02.931 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:02.928+0000 7fdcfd7fa700 1 -- 172.21.15.132:0/491087411 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdcf40263a0 con 0x7fdd000f6510 2024-02-20T00:18:02.931 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:02.928+0000 7fdd06c66700 1 -- 172.21.15.132:0/491087411 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd00045ad0 con 0x7fdd000f6510 2024-02-20T00:18:02.931 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:02.928+0000 7fdcfd7fa700 1 --2- 172.21.15.132:0/491087411 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fdcf002ff10 0x7fdcf00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:18:02.931 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:02.928+0000 7fdcfd7fa700 1 -- 172.21.15.132:0/491087411 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdcf4041250 con 0x7fdd000f6510 2024-02-20T00:18:02.931 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:02.928+0000 7fdcfffff700 1 --2- 172.21.15.132:0/491087411 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fdcf002ff10 0x7fdcf00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:18:02.931 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:02.929+0000 7fdcfffff700 1 --2- 172.21.15.132:0/491087411 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fdcf002ff10 0x7fdcf00323d0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7fdcec0096a0 tx=0x7fdcec006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:18:02.934 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:02.933+0000 7fdcfd7fa700 1 -- 172.21.15.132:0/491087411 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdcf40126f0 con 0x7fdd000f6510 2024-02-20T00:18:03.137 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:03.136+0000 7fdd06c66700 1 -- 172.21.15.132:0/491087411 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdd00108210 con 0x7fdd000f6510 2024-02-20T00:18:03.138 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:03.137+0000 7fdcfd7fa700 1 -- 172.21.15.132:0/491087411 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fdcf4012330 con 0x7fdd000f6510 2024-02-20T00:18:03.138 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:18:03.138 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:18:03.140 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:03.138+0000 7fdd06c66700 1 -- 172.21.15.132:0/491087411 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fdcf002ff10 msgr2=0x7fdcf00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:18:03.140 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:03.138+0000 7fdd06c66700 1 --2- 172.21.15.132:0/491087411 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fdcf002ff10 0x7fdcf00323d0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7fdcec0096a0 tx=0x7fdcec006b40).stop 2024-02-20T00:18:03.140 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:03.138+0000 7fdd06c66700 1 -- 172.21.15.132:0/491087411 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdd000f6510 msgr2=0x7fdd001067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:18:03.140 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:03.138+0000 7fdd06c66700 1 --2- 172.21.15.132:0/491087411 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdd000f6510 0x7fdd001067e0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0x7fdcf4005010 tx=0x7fdcf400b040).stop 2024-02-20T00:18:03.140 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:03.138+0000 7fdd06c66700 1 -- 172.21.15.132:0/491087411 shutdown_connections 2024-02-20T00:18:03.140 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:03.138+0000 7fdd06c66700 1 --2- 172.21.15.132:0/491087411 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fdcf002ff10 0x7fdcf00323d0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:18:03.140 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:03.138+0000 7fdd06c66700 1 --2- 172.21.15.132:0/491087411 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdd000f6510 0x7fdd001067e0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:18:03.140 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:03.138+0000 7fdd06c66700 1 -- 172.21.15.132:0/491087411 >> 172.21.15.132:0/491087411 conn(0x7fdd000f1c70 msgr2=0x7fdd000f26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:18:03.140 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:03.138+0000 7fdd06c66700 1 -- 172.21.15.132:0/491087411 shutdown_connections 2024-02-20T00:18:03.140 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:03.138+0000 7fdd06c66700 1 -- 172.21.15.132:0/491087411 wait complete. 2024-02-20T00:18:03.141 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:18:04.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:18:03 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:18:02.912110+0000 mgr.smithi084.bmqrnf (mgr.14162) 251 : cluster [DBG] pgmap v237: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:18:04.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:18:03 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:18:03.137417+0000 mon.smithi084 (mon.0) 262 : audit [DBG] from='client.? 172.21.15.132:0/491087411' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:18:04.816 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:18:04.816 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:18:06.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:18:05 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:18:04.912473+0000 mgr.smithi084.bmqrnf (mgr.14162) 252 : cluster [DBG] pgmap v238: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:18:06.842 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:06.834+0000 7f1e5a6ab700 1 -- 172.21.15.132:0/1588754473 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1e54060280 msgr2=0x7f1e540f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:18:06.842 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:06.834+0000 7f1e5a6ab700 1 --2- 172.21.15.132:0/1588754473 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1e54060280 0x7f1e540f90f0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 rx=0x7f1e48009fb0 tx=0x7f1e48009b30).stop 2024-02-20T00:18:06.842 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:06.834+0000 7f1e5a6ab700 1 -- 172.21.15.132:0/1588754473 shutdown_connections 2024-02-20T00:18:06.842 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:06.834+0000 7f1e5a6ab700 1 --2- 172.21.15.132:0/1588754473 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1e54060280 0x7f1e540f90f0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:18:06.842 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:06.834+0000 7f1e5a6ab700 1 -- 172.21.15.132:0/1588754473 >> 172.21.15.132:0/1588754473 conn(0x7f1e540f0b70 msgr2=0x7f1e540f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:18:06.843 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:06.834+0000 7f1e5a6ab700 1 -- 172.21.15.132:0/1588754473 shutdown_connections 2024-02-20T00:18:06.843 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:06.834+0000 7f1e5a6ab700 1 -- 172.21.15.132:0/1588754473 wait complete. 2024-02-20T00:18:06.843 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:06.835+0000 7f1e5a6ab700 1 Processor -- start 2024-02-20T00:18:06.843 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:06.835+0000 7f1e5a6ab700 1 -- start start 2024-02-20T00:18:06.843 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:06.835+0000 7f1e5a6ab700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1e54060280 0x7f1e540f6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:18:06.843 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:06.835+0000 7f1e5a6ab700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e540f4ff0 con 0x7f1e54060280 2024-02-20T00:18:06.843 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:06.835+0000 7f1e53fff700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1e54060280 0x7f1e540f6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:18:06.844 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:06.835+0000 7f1e53fff700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1e54060280 0x7f1e540f6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:39764/0 (socket says 172.21.15.132:39764) 2024-02-20T00:18:06.844 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:06.835+0000 7f1e53fff700 1 -- 172.21.15.132:0/3515636968 learned_addr learned my addr 172.21.15.132:0/3515636968 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:18:06.844 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:06.836+0000 7f1e53fff700 1 -- 172.21.15.132:0/3515636968 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e48005040 con 0x7f1e54060280 2024-02-20T00:18:06.844 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:06.836+0000 7f1e53fff700 1 --2- 172.21.15.132:0/3515636968 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1e54060280 0x7f1e540f6b70 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0x7f1e480089f0 tx=0x7f1e4800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:18:06.844 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:06.836+0000 7f1e50ff9700 1 -- 172.21.15.132:0/3515636968 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1e48009320 con 0x7f1e54060280 2024-02-20T00:18:06.844 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:06.836+0000 7f1e5a6ab700 1 -- 172.21.15.132:0/3515636968 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e540f5190 con 0x7f1e54060280 2024-02-20T00:18:06.845 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:06.836+0000 7f1e50ff9700 1 -- 172.21.15.132:0/3515636968 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1e48013380 con 0x7f1e54060280 2024-02-20T00:18:06.845 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:06.836+0000 7f1e50ff9700 1 -- 172.21.15.132:0/3515636968 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1e48014e20 con 0x7f1e54060280 2024-02-20T00:18:06.845 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:06.836+0000 7f1e5a6ab700 1 -- 172.21.15.132:0/3515636968 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e540f5620 con 0x7f1e54060280 2024-02-20T00:18:06.845 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:06.837+0000 7f1e50ff9700 1 -- 172.21.15.132:0/3515636968 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1e480095f0 con 0x7f1e54060280 2024-02-20T00:18:06.845 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:06.837+0000 7f1e5a6ab700 1 -- 172.21.15.132:0/3515636968 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e54059100 con 0x7f1e54060280 2024-02-20T00:18:06.845 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:06.837+0000 7f1e50ff9700 1 --2- 172.21.15.132:0/3515636968 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f1e3c02ff10 0x7f1e3c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:18:06.846 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:06.837+0000 7f1e50ff9700 1 -- 172.21.15.132:0/3515636968 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1e48040f80 con 0x7f1e54060280 2024-02-20T00:18:06.846 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:06.837+0000 7f1e537fe700 1 --2- 172.21.15.132:0/3515636968 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f1e3c02ff10 0x7f1e3c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:18:06.846 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:06.837+0000 7f1e537fe700 1 --2- 172.21.15.132:0/3515636968 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f1e3c02ff10 0x7f1e3c0323d0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f1e440096a0 tx=0x7f1e44006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:18:06.847 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:06.842+0000 7f1e50ff9700 1 -- 172.21.15.132:0/3515636968 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1e48014420 con 0x7f1e54060280 2024-02-20T00:18:07.051 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:07.050+0000 7f1e5a6ab700 1 -- 172.21.15.132:0/3515636968 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1e54045ad0 con 0x7f1e54060280 2024-02-20T00:18:07.052 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:07.050+0000 7f1e50ff9700 1 -- 172.21.15.132:0/3515636968 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1e48010230 con 0x7f1e54060280 2024-02-20T00:18:07.052 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:18:07.052 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:18:07.053 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:07.052+0000 7f1e5a6ab700 1 -- 172.21.15.132:0/3515636968 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f1e3c02ff10 msgr2=0x7f1e3c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:18:07.054 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:07.052+0000 7f1e5a6ab700 1 --2- 172.21.15.132:0/3515636968 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f1e3c02ff10 0x7f1e3c0323d0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f1e440096a0 tx=0x7f1e44006b40).stop 2024-02-20T00:18:07.054 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:07.052+0000 7f1e5a6ab700 1 -- 172.21.15.132:0/3515636968 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1e54060280 msgr2=0x7f1e540f6b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:18:07.054 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:07.052+0000 7f1e5a6ab700 1 --2- 172.21.15.132:0/3515636968 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1e54060280 0x7f1e540f6b70 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0x7f1e480089f0 tx=0x7f1e4800b040).stop 2024-02-20T00:18:07.054 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:07.052+0000 7f1e5a6ab700 1 -- 172.21.15.132:0/3515636968 shutdown_connections 2024-02-20T00:18:07.054 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:07.052+0000 7f1e5a6ab700 1 --2- 172.21.15.132:0/3515636968 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f1e3c02ff10 0x7f1e3c0323d0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:18:07.054 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:07.052+0000 7f1e5a6ab700 1 --2- 172.21.15.132:0/3515636968 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1e54060280 0x7f1e540f6b70 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:18:07.054 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:07.052+0000 7f1e5a6ab700 1 -- 172.21.15.132:0/3515636968 >> 172.21.15.132:0/3515636968 conn(0x7f1e540f0b70 msgr2=0x7f1e540f15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:18:07.054 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:07.052+0000 7f1e5a6ab700 1 -- 172.21.15.132:0/3515636968 shutdown_connections 2024-02-20T00:18:07.054 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:07.052+0000 7f1e5a6ab700 1 -- 172.21.15.132:0/3515636968 wait complete. 2024-02-20T00:18:07.055 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:18:08.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:18:07 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:18:06.912850+0000 mgr.smithi084.bmqrnf (mgr.14162) 253 : cluster [DBG] pgmap v239: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:18:08.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:18:07 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:18:07.051022+0000 mon.smithi084 (mon.0) 263 : audit [DBG] from='client.? 172.21.15.132:0/3515636968' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:18:08.833 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:18:08.834 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:18:10.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:18:09 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:18:08.913265+0000 mgr.smithi084.bmqrnf (mgr.14162) 254 : cluster [DBG] pgmap v240: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:18:10.731 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:10.728+0000 7f8e03e82700 1 -- 172.21.15.132:0/1405031004 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8dfc0f5410 msgr2=0x7f8dfc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:18:10.731 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:10.728+0000 7f8e03e82700 1 --2- 172.21.15.132:0/1405031004 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8dfc0f5410 0x7f8dfc0f5830 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 rx=0x7f8dec009fb0 tx=0x7f8dec009b30).stop 2024-02-20T00:18:10.731 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:10.728+0000 7f8e03e82700 1 -- 172.21.15.132:0/1405031004 shutdown_connections 2024-02-20T00:18:10.731 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:10.728+0000 7f8e03e82700 1 --2- 172.21.15.132:0/1405031004 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8dfc0f5410 0x7f8dfc0f5830 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:18:10.731 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:10.728+0000 7f8e03e82700 1 -- 172.21.15.132:0/1405031004 >> 172.21.15.132:0/1405031004 conn(0x7f8dfc0f0b70 msgr2=0x7f8dfc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:18:10.731 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:10.728+0000 7f8e03e82700 1 -- 172.21.15.132:0/1405031004 shutdown_connections 2024-02-20T00:18:10.732 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:10.728+0000 7f8e03e82700 1 -- 172.21.15.132:0/1405031004 wait complete. 2024-02-20T00:18:10.732 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:10.729+0000 7f8e03e82700 1 Processor -- start 2024-02-20T00:18:10.732 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:10.729+0000 7f8e03e82700 1 -- start start 2024-02-20T00:18:10.732 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:10.729+0000 7f8e03e82700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8dfc0f5410 0x7f8dfc0fd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:18:10.732 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:10.729+0000 7f8e03e82700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dfc0fb960 con 0x7f8dfc0f5410 2024-02-20T00:18:10.734 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:10.729+0000 7f8e01c1e700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8dfc0f5410 0x7f8dfc0fd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:18:10.734 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:10.729+0000 7f8e01c1e700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8dfc0f5410 0x7f8dfc0fd4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:39450/0 (socket says 172.21.15.132:39450) 2024-02-20T00:18:10.734 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:10.729+0000 7f8e01c1e700 1 -- 172.21.15.132:0/4065093150 learned_addr learned my addr 172.21.15.132:0/4065093150 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:18:10.734 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:10.730+0000 7f8e01c1e700 1 -- 172.21.15.132:0/4065093150 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dec005040 con 0x7f8dfc0f5410 2024-02-20T00:18:10.734 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:10.730+0000 7f8e01c1e700 1 --2- 172.21.15.132:0/4065093150 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8dfc0f5410 0x7f8dfc0fd4e0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0x7f8dec005010 tx=0x7f8dec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:18:10.734 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:10.730+0000 7f8df27fc700 1 -- 172.21.15.132:0/4065093150 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8dec009320 con 0x7f8dfc0f5410 2024-02-20T00:18:10.734 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:10.730+0000 7f8e03e82700 1 -- 172.21.15.132:0/4065093150 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8dfc0fbb00 con 0x7f8dfc0f5410 2024-02-20T00:18:10.734 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:10.730+0000 7f8df27fc700 1 -- 172.21.15.132:0/4065093150 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8dec013380 con 0x7f8dfc0f5410 2024-02-20T00:18:10.734 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:10.730+0000 7f8df27fc700 1 -- 172.21.15.132:0/4065093150 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8dec014e50 con 0x7f8dfc0f5410 2024-02-20T00:18:10.735 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:10.730+0000 7f8e03e82700 1 -- 172.21.15.132:0/4065093150 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8dfc0fbf90 con 0x7f8dfc0f5410 2024-02-20T00:18:10.735 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:10.731+0000 7f8e03e82700 1 -- 172.21.15.132:0/4065093150 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8dfc059100 con 0x7f8dfc0f5410 2024-02-20T00:18:10.735 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:10.731+0000 7f8df27fc700 1 -- 172.21.15.132:0/4065093150 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8dec0095f0 con 0x7f8dfc0f5410 2024-02-20T00:18:10.735 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:10.731+0000 7f8df27fc700 1 --2- 172.21.15.132:0/4065093150 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f8de802ff10 0x7f8de80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:18:10.735 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:10.731+0000 7f8df27fc700 1 -- 172.21.15.132:0/4065093150 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8dec040fa0 con 0x7f8dfc0f5410 2024-02-20T00:18:10.735 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:10.731+0000 7f8e0141d700 1 --2- 172.21.15.132:0/4065093150 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f8de802ff10 0x7f8de80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:18:10.735 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:10.732+0000 7f8e0141d700 1 --2- 172.21.15.132:0/4065093150 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f8de802ff10 0x7f8de80323d0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7f8df80096a0 tx=0x7f8df8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:18:10.737 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:10.735+0000 7f8df27fc700 1 -- 172.21.15.132:0/4065093150 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8dec014420 con 0x7f8dfc0f5410 2024-02-20T00:18:10.942 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:10.940+0000 7f8e03e82700 1 -- 172.21.15.132:0/4065093150 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8dfc0fafa0 con 0x7f8dfc0f5410 2024-02-20T00:18:10.943 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:10.941+0000 7f8df27fc700 1 -- 172.21.15.132:0/4065093150 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8dec010230 con 0x7f8dfc0f5410 2024-02-20T00:18:10.943 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:18:10.943 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:18:10.945 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:10.942+0000 7f8e03e82700 1 -- 172.21.15.132:0/4065093150 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f8de802ff10 msgr2=0x7f8de80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:18:10.945 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:10.942+0000 7f8e03e82700 1 --2- 172.21.15.132:0/4065093150 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f8de802ff10 0x7f8de80323d0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7f8df80096a0 tx=0x7f8df8006b40).stop 2024-02-20T00:18:10.945 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:10.942+0000 7f8e03e82700 1 -- 172.21.15.132:0/4065093150 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8dfc0f5410 msgr2=0x7f8dfc0fd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:18:10.945 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:10.942+0000 7f8e03e82700 1 --2- 172.21.15.132:0/4065093150 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8dfc0f5410 0x7f8dfc0fd4e0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0x7f8dec005010 tx=0x7f8dec00b040).stop 2024-02-20T00:18:10.945 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:10.942+0000 7f8e03e82700 1 -- 172.21.15.132:0/4065093150 shutdown_connections 2024-02-20T00:18:10.945 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:10.942+0000 7f8e03e82700 1 --2- 172.21.15.132:0/4065093150 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f8de802ff10 0x7f8de80323d0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:18:10.945 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:10.942+0000 7f8e03e82700 1 --2- 172.21.15.132:0/4065093150 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8dfc0f5410 0x7f8dfc0fd4e0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:18:10.945 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:10.942+0000 7f8e03e82700 1 -- 172.21.15.132:0/4065093150 >> 172.21.15.132:0/4065093150 conn(0x7f8dfc0f0b70 msgr2=0x7f8dfc05fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:18:10.945 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:10.942+0000 7f8e03e82700 1 -- 172.21.15.132:0/4065093150 shutdown_connections 2024-02-20T00:18:10.945 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:10.942+0000 7f8e03e82700 1 -- 172.21.15.132:0/4065093150 wait complete. 2024-02-20T00:18:10.946 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:18:12.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:18:11 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:18:10.913646+0000 mgr.smithi084.bmqrnf (mgr.14162) 255 : cluster [DBG] pgmap v241: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:18:12.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:18:11 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:18:10.942282+0000 mon.smithi084 (mon.0) 264 : audit [DBG] from='client.? 172.21.15.132:0/4065093150' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:18:12.690 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:18:12.690 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:18:14.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:18:13 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:18:12.913913+0000 mgr.smithi084.bmqrnf (mgr.14162) 256 : cluster [DBG] pgmap v242: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:18:14.506 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:14.503+0000 7ff0d4c4c700 1 -- 172.21.15.132:0/844844470 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff0d00f51a0 msgr2=0x7ff0d00f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:18:14.506 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:14.503+0000 7ff0d4c4c700 1 --2- 172.21.15.132:0/844844470 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff0d00f51a0 0x7ff0d00f55c0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 rx=0x7ff0c0009fb0 tx=0x7ff0c0009b30).stop 2024-02-20T00:18:14.506 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:14.503+0000 7ff0d4c4c700 1 -- 172.21.15.132:0/844844470 shutdown_connections 2024-02-20T00:18:14.506 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:14.503+0000 7ff0d4c4c700 1 --2- 172.21.15.132:0/844844470 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff0d00f51a0 0x7ff0d00f55c0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:18:14.506 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:14.503+0000 7ff0d4c4c700 1 -- 172.21.15.132:0/844844470 >> 172.21.15.132:0/844844470 conn(0x7ff0d00f09a0 msgr2=0x7ff0d00f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:18:14.506 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:14.503+0000 7ff0d4c4c700 1 -- 172.21.15.132:0/844844470 shutdown_connections 2024-02-20T00:18:14.506 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:14.503+0000 7ff0d4c4c700 1 -- 172.21.15.132:0/844844470 wait complete. 2024-02-20T00:18:14.506 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:14.503+0000 7ff0d4c4c700 1 Processor -- start 2024-02-20T00:18:14.506 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:14.504+0000 7ff0d4c4c700 1 -- start start 2024-02-20T00:18:14.506 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:14.504+0000 7ff0d4c4c700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff0d00f51a0 0x7ff0d01065a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:18:14.507 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:14.504+0000 7ff0d4c4c700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0d0106a90 con 0x7ff0d00f51a0 2024-02-20T00:18:14.508 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:14.504+0000 7ff0cf7fe700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff0d00f51a0 0x7ff0d01065a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:18:14.508 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:14.504+0000 7ff0cf7fe700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff0d00f51a0 0x7ff0d01065a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:39454/0 (socket says 172.21.15.132:39454) 2024-02-20T00:18:14.508 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:14.504+0000 7ff0cf7fe700 1 -- 172.21.15.132:0/2964746296 learned_addr learned my addr 172.21.15.132:0/2964746296 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:18:14.508 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:14.504+0000 7ff0cf7fe700 1 -- 172.21.15.132:0/2964746296 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0c0005040 con 0x7ff0d00f51a0 2024-02-20T00:18:14.509 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:14.504+0000 7ff0cf7fe700 1 --2- 172.21.15.132:0/2964746296 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff0d00f51a0 0x7ff0d01065a0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7ff0c00089f0 tx=0x7ff0c000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:18:14.509 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:14.505+0000 7ff0b7fff700 1 -- 172.21.15.132:0/2964746296 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff0c0009320 con 0x7ff0d00f51a0 2024-02-20T00:18:14.509 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:14.505+0000 7ff0d4c4c700 1 -- 172.21.15.132:0/2964746296 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0d0106c90 con 0x7ff0d00f51a0 2024-02-20T00:18:14.509 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:14.505+0000 7ff0d4c4c700 1 -- 172.21.15.132:0/2964746296 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0d0107120 con 0x7ff0d00f51a0 2024-02-20T00:18:14.509 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:14.505+0000 7ff0b7fff700 1 -- 172.21.15.132:0/2964746296 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff0c0013380 con 0x7ff0d00f51a0 2024-02-20T00:18:14.509 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:14.505+0000 7ff0b7fff700 1 -- 172.21.15.132:0/2964746296 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff0c0014e40 con 0x7ff0d00f51a0 2024-02-20T00:18:14.509 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:14.505+0000 7ff0b7fff700 1 -- 172.21.15.132:0/2964746296 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff0c0014420 con 0x7ff0d00f51a0 2024-02-20T00:18:14.509 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:14.505+0000 7ff0d4c4c700 1 -- 172.21.15.132:0/2964746296 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0d0045ad0 con 0x7ff0d00f51a0 2024-02-20T00:18:14.509 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:14.506+0000 7ff0b7fff700 1 --2- 172.21.15.132:0/2964746296 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7ff0b802ff10 0x7ff0b80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:18:14.509 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:14.506+0000 7ff0b7fff700 1 -- 172.21.15.132:0/2964746296 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff0c0041180 con 0x7ff0d00f51a0 2024-02-20T00:18:14.509 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:14.506+0000 7ff0ceffd700 1 --2- 172.21.15.132:0/2964746296 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7ff0b802ff10 0x7ff0b80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:18:14.509 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:14.506+0000 7ff0ceffd700 1 --2- 172.21.15.132:0/2964746296 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7ff0b802ff10 0x7ff0b80323d0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7ff0c40096a0 tx=0x7ff0c4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:18:14.512 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:14.510+0000 7ff0b7fff700 1 -- 172.21.15.132:0/2964746296 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff0c0012940 con 0x7ff0d00f51a0 2024-02-20T00:18:14.719 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:14.716+0000 7ff0d4c4c700 1 -- 172.21.15.132:0/2964746296 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff0d0107fd0 con 0x7ff0d00f51a0 2024-02-20T00:18:14.719 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:14.717+0000 7ff0b7fff700 1 -- 172.21.15.132:0/2964746296 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff0c0012b50 con 0x7ff0d00f51a0 2024-02-20T00:18:14.720 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:18:14.720 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:18:14.722 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:14.719+0000 7ff0d4c4c700 1 -- 172.21.15.132:0/2964746296 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7ff0b802ff10 msgr2=0x7ff0b80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:18:14.722 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:14.719+0000 7ff0d4c4c700 1 --2- 172.21.15.132:0/2964746296 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7ff0b802ff10 0x7ff0b80323d0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7ff0c40096a0 tx=0x7ff0c4006b40).stop 2024-02-20T00:18:14.722 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:14.719+0000 7ff0d4c4c700 1 -- 172.21.15.132:0/2964746296 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff0d00f51a0 msgr2=0x7ff0d01065a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:18:14.722 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:14.719+0000 7ff0d4c4c700 1 --2- 172.21.15.132:0/2964746296 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff0d00f51a0 0x7ff0d01065a0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7ff0c00089f0 tx=0x7ff0c000b040).stop 2024-02-20T00:18:14.722 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:14.719+0000 7ff0d4c4c700 1 -- 172.21.15.132:0/2964746296 shutdown_connections 2024-02-20T00:18:14.722 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:14.719+0000 7ff0d4c4c700 1 --2- 172.21.15.132:0/2964746296 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7ff0b802ff10 0x7ff0b80323d0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:18:14.722 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:14.719+0000 7ff0d4c4c700 1 --2- 172.21.15.132:0/2964746296 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff0d00f51a0 0x7ff0d01065a0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:18:14.722 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:14.719+0000 7ff0d4c4c700 1 -- 172.21.15.132:0/2964746296 >> 172.21.15.132:0/2964746296 conn(0x7ff0d00f09a0 msgr2=0x7ff0d00f13c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:18:14.722 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:14.719+0000 7ff0d4c4c700 1 -- 172.21.15.132:0/2964746296 shutdown_connections 2024-02-20T00:18:14.722 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:14.719+0000 7ff0d4c4c700 1 -- 172.21.15.132:0/2964746296 wait complete. 2024-02-20T00:18:14.722 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:18:15.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:18:14 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:18:14.718587+0000 mon.smithi084 (mon.0) 265 : audit [DBG] from='client.? 172.21.15.132:0/2964746296' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:18:16.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:18:15 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:18:14.914290+0000 mgr.smithi084.bmqrnf (mgr.14162) 257 : cluster [DBG] pgmap v243: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:18:16.475 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:18:16.475 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:18:18.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:18:17 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:18:16.914563+0000 mgr.smithi084.bmqrnf (mgr.14162) 258 : cluster [DBG] pgmap v244: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:18:18.488 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:18.482+0000 7f574063a700 1 -- 172.21.15.132:0/3133258446 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f57380601e0 msgr2=0x7f57380f9190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:18:18.488 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:18.482+0000 7f574063a700 1 --2- 172.21.15.132:0/3133258446 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f57380601e0 0x7f57380f9190 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 rx=0x7f5734009fb0 tx=0x7f5734009b30).stop 2024-02-20T00:18:18.488 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:18.482+0000 7f574063a700 1 -- 172.21.15.132:0/3133258446 shutdown_connections 2024-02-20T00:18:18.488 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:18.482+0000 7f574063a700 1 --2- 172.21.15.132:0/3133258446 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f57380601e0 0x7f57380f9190 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:18:18.489 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:18.482+0000 7f574063a700 1 -- 172.21.15.132:0/3133258446 >> 172.21.15.132:0/3133258446 conn(0x7f57380f0b70 msgr2=0x7f57380f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:18:18.489 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:18.482+0000 7f574063a700 1 -- 172.21.15.132:0/3133258446 shutdown_connections 2024-02-20T00:18:18.489 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:18.482+0000 7f574063a700 1 -- 172.21.15.132:0/3133258446 wait complete. 2024-02-20T00:18:18.491 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:18.483+0000 7f574063a700 1 Processor -- start 2024-02-20T00:18:18.491 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:18.483+0000 7f574063a700 1 -- start start 2024-02-20T00:18:18.491 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:18.483+0000 7f574063a700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f57380601e0 0x7f57381089e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:18:18.491 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:18.483+0000 7f574063a700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5738108ed0 con 0x7f57380601e0 2024-02-20T00:18:18.491 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:18.483+0000 7f573e3d6700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f57380601e0 0x7f57381089e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:18:18.491 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:18.483+0000 7f573e3d6700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f57380601e0 0x7f57381089e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:35106/0 (socket says 172.21.15.132:35106) 2024-02-20T00:18:18.492 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:18.483+0000 7f573e3d6700 1 -- 172.21.15.132:0/3812781380 learned_addr learned my addr 172.21.15.132:0/3812781380 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:18:18.492 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:18.483+0000 7f573e3d6700 1 -- 172.21.15.132:0/3812781380 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5734005040 con 0x7f57380601e0 2024-02-20T00:18:18.492 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:18.484+0000 7f573e3d6700 1 --2- 172.21.15.132:0/3812781380 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f57380601e0 0x7f57381089e0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 rx=0x7f5734007a10 tx=0x7f5734009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:18:18.492 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:18.484+0000 7f572effd700 1 -- 172.21.15.132:0/3812781380 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5734018070 con 0x7f57380601e0 2024-02-20T00:18:18.492 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:18.484+0000 7f574063a700 1 -- 172.21.15.132:0/3812781380 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57381090d0 con 0x7f57380601e0 2024-02-20T00:18:18.492 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:18.484+0000 7f574063a700 1 -- 172.21.15.132:0/3812781380 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5738109560 con 0x7f57380601e0 2024-02-20T00:18:18.492 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:18.484+0000 7f572effd700 1 -- 172.21.15.132:0/3812781380 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5734013410 con 0x7f57380601e0 2024-02-20T00:18:18.492 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:18.484+0000 7f572effd700 1 -- 172.21.15.132:0/3812781380 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5734027440 con 0x7f57380601e0 2024-02-20T00:18:18.493 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:18.485+0000 7f572effd700 1 -- 172.21.15.132:0/3812781380 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f573401d420 con 0x7f57380601e0 2024-02-20T00:18:18.493 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:18.485+0000 7f574063a700 1 -- 172.21.15.132:0/3812781380 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5738045ad0 con 0x7f57380601e0 2024-02-20T00:18:18.493 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:18.485+0000 7f572effd700 1 --2- 172.21.15.132:0/3812781380 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f572402ff60 0x7f5724032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:18:18.493 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:18.485+0000 7f572effd700 1 -- 172.21.15.132:0/3812781380 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5734041500 con 0x7f57380601e0 2024-02-20T00:18:18.493 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:18.485+0000 7f573dbd5700 1 --2- 172.21.15.132:0/3812781380 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f572402ff60 0x7f5724032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:18:18.493 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:18.486+0000 7f573dbd5700 1 --2- 172.21.15.132:0/3812781380 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f572402ff60 0x7f5724032420 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f57280096a0 tx=0x7f5728006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:18:18.493 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:18.490+0000 7f572effd700 1 -- 172.21.15.132:0/3812781380 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5734013ca0 con 0x7f57380601e0 2024-02-20T00:18:18.703 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:18.700+0000 7f574063a700 1 -- 172.21.15.132:0/3812781380 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f573810a410 con 0x7f57380601e0 2024-02-20T00:18:18.703 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:18.701+0000 7f572effd700 1 -- 172.21.15.132:0/3812781380 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5734025b40 con 0x7f57380601e0 2024-02-20T00:18:18.704 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:18:18.704 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:18:18.706 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:18.702+0000 7f574063a700 1 -- 172.21.15.132:0/3812781380 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f572402ff60 msgr2=0x7f5724032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:18:18.706 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:18.702+0000 7f574063a700 1 --2- 172.21.15.132:0/3812781380 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f572402ff60 0x7f5724032420 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f57280096a0 tx=0x7f5728006b40).stop 2024-02-20T00:18:18.706 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:18.702+0000 7f574063a700 1 -- 172.21.15.132:0/3812781380 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f57380601e0 msgr2=0x7f57381089e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:18:18.706 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:18.702+0000 7f574063a700 1 --2- 172.21.15.132:0/3812781380 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f57380601e0 0x7f57381089e0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 rx=0x7f5734007a10 tx=0x7f5734009b30).stop 2024-02-20T00:18:18.706 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:18.703+0000 7f574063a700 1 -- 172.21.15.132:0/3812781380 shutdown_connections 2024-02-20T00:18:18.707 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:18.703+0000 7f574063a700 1 --2- 172.21.15.132:0/3812781380 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f572402ff60 0x7f5724032420 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:18:18.707 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:18.703+0000 7f574063a700 1 --2- 172.21.15.132:0/3812781380 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f57380601e0 0x7f57381089e0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:18:18.707 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:18.703+0000 7f574063a700 1 -- 172.21.15.132:0/3812781380 >> 172.21.15.132:0/3812781380 conn(0x7f57380f0b70 msgr2=0x7f57380f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:18:18.707 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:18.703+0000 7f574063a700 1 -- 172.21.15.132:0/3812781380 shutdown_connections 2024-02-20T00:18:18.707 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:18.703+0000 7f574063a700 1 -- 172.21.15.132:0/3812781380 wait complete. 2024-02-20T00:18:18.707 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:18:19.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:18:18 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:18:18.702385+0000 mon.smithi084 (mon.0) 266 : audit [DBG] from='client.? 172.21.15.132:0/3812781380' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:18:20.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:18:19 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:18:18.914787+0000 mgr.smithi084.bmqrnf (mgr.14162) 259 : cluster [DBG] pgmap v245: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:18:20.385 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:18:20.385 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:18:22.232 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:22.229+0000 7f65f9f1d700 1 -- 172.21.15.132:0/1900861121 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f65f40f6510 msgr2=0x7f65f40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:18:22.232 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:22.229+0000 7f65f9f1d700 1 --2- 172.21.15.132:0/1900861121 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f65f40f6510 0x7f65f40f6930 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 rx=0x7f65e8009fb0 tx=0x7f65e8009b30).stop 2024-02-20T00:18:22.232 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:22.229+0000 7f65f9f1d700 1 -- 172.21.15.132:0/1900861121 shutdown_connections 2024-02-20T00:18:22.232 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:22.229+0000 7f65f9f1d700 1 --2- 172.21.15.132:0/1900861121 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f65f40f6510 0x7f65f40f6930 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:18:22.232 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:22.229+0000 7f65f9f1d700 1 -- 172.21.15.132:0/1900861121 >> 172.21.15.132:0/1900861121 conn(0x7f65f40f1c70 msgr2=0x7f65f40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:18:22.233 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:22.229+0000 7f65f9f1d700 1 -- 172.21.15.132:0/1900861121 shutdown_connections 2024-02-20T00:18:22.233 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:22.229+0000 7f65f9f1d700 1 -- 172.21.15.132:0/1900861121 wait complete. 2024-02-20T00:18:22.233 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:22.230+0000 7f65f9f1d700 1 Processor -- start 2024-02-20T00:18:22.235 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:22.230+0000 7f65f9f1d700 1 -- start start 2024-02-20T00:18:22.235 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:22.230+0000 7f65f9f1d700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f65f40f6510 0x7f65f406b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:18:22.235 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:22.230+0000 7f65f9f1d700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65f4069e20 con 0x7f65f40f6510 2024-02-20T00:18:22.235 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:22.231+0000 7f65f37fe700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f65f40f6510 0x7f65f406b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:18:22.235 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:22.231+0000 7f65f37fe700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f65f40f6510 0x7f65f406b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:35112/0 (socket says 172.21.15.132:35112) 2024-02-20T00:18:22.235 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:22.231+0000 7f65f37fe700 1 -- 172.21.15.132:0/2679168633 learned_addr learned my addr 172.21.15.132:0/2679168633 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:18:22.235 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:22.231+0000 7f65f37fe700 1 -- 172.21.15.132:0/2679168633 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65e8005040 con 0x7f65f40f6510 2024-02-20T00:18:22.235 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:22.231+0000 7f65f37fe700 1 --2- 172.21.15.132:0/2679168633 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f65f40f6510 0x7f65f406b9a0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 rx=0x7f65e8005010 tx=0x7f65e800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:18:22.235 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:22.231+0000 7f65dbfff700 1 -- 172.21.15.132:0/2679168633 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f65e8009320 con 0x7f65f40f6510 2024-02-20T00:18:22.235 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:22.231+0000 7f65f9f1d700 1 -- 172.21.15.132:0/2679168633 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65f4069fc0 con 0x7f65f40f6510 2024-02-20T00:18:22.235 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:22.231+0000 7f65dbfff700 1 -- 172.21.15.132:0/2679168633 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f65e8013380 con 0x7f65f40f6510 2024-02-20T00:18:22.235 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:22.231+0000 7f65dbfff700 1 -- 172.21.15.132:0/2679168633 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f65e8014e50 con 0x7f65f40f6510 2024-02-20T00:18:22.236 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:22.231+0000 7f65f9f1d700 1 -- 172.21.15.132:0/2679168633 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65f406a450 con 0x7f65f40f6510 2024-02-20T00:18:22.236 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:22.232+0000 7f65dbfff700 1 -- 172.21.15.132:0/2679168633 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f65e8014420 con 0x7f65f40f6510 2024-02-20T00:18:22.236 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:22.232+0000 7f65f9f1d700 1 -- 172.21.15.132:0/2679168633 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65f4059100 con 0x7f65f40f6510 2024-02-20T00:18:22.236 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:22.232+0000 7f65dbfff700 1 --2- 172.21.15.132:0/2679168633 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f65dc02ff10 0x7f65dc0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:18:22.236 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:22.232+0000 7f65dbfff700 1 -- 172.21.15.132:0/2679168633 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f65e8040f40 con 0x7f65f40f6510 2024-02-20T00:18:22.236 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:22.232+0000 7f65f2ffd700 1 --2- 172.21.15.132:0/2679168633 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f65dc02ff10 0x7f65dc0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:18:22.236 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:22.233+0000 7f65f2ffd700 1 --2- 172.21.15.132:0/2679168633 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f65dc02ff10 0x7f65dc0323d0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7f65e40096a0 tx=0x7f65e4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:18:22.239 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:22.237+0000 7f65dbfff700 1 -- 172.21.15.132:0/2679168633 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f65e80095f0 con 0x7f65f40f6510 2024-02-20T00:18:22.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:18:21 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:18:20.915041+0000 mgr.smithi084.bmqrnf (mgr.14162) 260 : cluster [DBG] pgmap v246: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:18:22.443 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:22.441+0000 7f65f9f1d700 1 -- 172.21.15.132:0/2679168633 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f65f40fc0a0 con 0x7f65f40f6510 2024-02-20T00:18:22.444 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:22.441+0000 7f65dbfff700 1 -- 172.21.15.132:0/2679168633 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f65e801db70 con 0x7f65f40f6510 2024-02-20T00:18:22.444 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:18:22.444 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:18:22.445 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:22.443+0000 7f65f9f1d700 1 -- 172.21.15.132:0/2679168633 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f65dc02ff10 msgr2=0x7f65dc0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:18:22.445 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:22.443+0000 7f65f9f1d700 1 --2- 172.21.15.132:0/2679168633 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f65dc02ff10 0x7f65dc0323d0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7f65e40096a0 tx=0x7f65e4006b40).stop 2024-02-20T00:18:22.445 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:22.443+0000 7f65f9f1d700 1 -- 172.21.15.132:0/2679168633 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f65f40f6510 msgr2=0x7f65f406b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:18:22.445 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:22.443+0000 7f65f9f1d700 1 --2- 172.21.15.132:0/2679168633 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f65f40f6510 0x7f65f406b9a0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 rx=0x7f65e8005010 tx=0x7f65e800b040).stop 2024-02-20T00:18:22.446 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:22.443+0000 7f65f9f1d700 1 -- 172.21.15.132:0/2679168633 shutdown_connections 2024-02-20T00:18:22.446 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:22.443+0000 7f65f9f1d700 1 --2- 172.21.15.132:0/2679168633 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f65dc02ff10 0x7f65dc0323d0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:18:22.446 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:22.443+0000 7f65f9f1d700 1 --2- 172.21.15.132:0/2679168633 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f65f40f6510 0x7f65f406b9a0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:18:22.446 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:22.443+0000 7f65f9f1d700 1 -- 172.21.15.132:0/2679168633 >> 172.21.15.132:0/2679168633 conn(0x7f65f40f1c70 msgr2=0x7f65f40f26f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:18:22.446 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:22.443+0000 7f65f9f1d700 1 -- 172.21.15.132:0/2679168633 shutdown_connections 2024-02-20T00:18:22.446 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:22.443+0000 7f65f9f1d700 1 -- 172.21.15.132:0/2679168633 wait complete. 2024-02-20T00:18:22.446 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:18:23.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:18:22 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:18:22.442921+0000 mon.smithi084 (mon.0) 267 : audit [DBG] from='client.? 172.21.15.132:0/2679168633' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:18:24.037 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:18:24.037 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:18:24.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:18:23 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:18:22.915418+0000 mgr.smithi084.bmqrnf (mgr.14162) 261 : cluster [DBG] pgmap v247: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:18:26.123 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:26.120+0000 7f34ae269700 1 -- 172.21.15.132:0/1192238206 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f34a80f5410 msgr2=0x7f34a80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:18:26.124 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:26.120+0000 7f34ae269700 1 --2- 172.21.15.132:0/1192238206 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f34a80f5410 0x7f34a80f5830 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 rx=0x7f3498009fb0 tx=0x7f3498009b30).stop 2024-02-20T00:18:26.124 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:26.120+0000 7f34ae269700 1 -- 172.21.15.132:0/1192238206 shutdown_connections 2024-02-20T00:18:26.124 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:26.120+0000 7f34ae269700 1 --2- 172.21.15.132:0/1192238206 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f34a80f5410 0x7f34a80f5830 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:18:26.124 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:26.120+0000 7f34ae269700 1 -- 172.21.15.132:0/1192238206 >> 172.21.15.132:0/1192238206 conn(0x7f34a80f0b70 msgr2=0x7f34a80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:18:26.124 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:26.120+0000 7f34ae269700 1 -- 172.21.15.132:0/1192238206 shutdown_connections 2024-02-20T00:18:26.124 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:26.120+0000 7f34ae269700 1 -- 172.21.15.132:0/1192238206 wait complete. 2024-02-20T00:18:26.124 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:26.121+0000 7f34ae269700 1 Processor -- start 2024-02-20T00:18:26.124 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:26.121+0000 7f34ae269700 1 -- start start 2024-02-20T00:18:26.124 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:26.121+0000 7f34ae269700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f34a80f5410 0x7f34a810b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:18:26.124 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:26.121+0000 7f34ae269700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34a81072f0 con 0x7f34a80f5410 2024-02-20T00:18:26.126 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:26.122+0000 7f34a7fff700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f34a80f5410 0x7f34a810b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:18:26.126 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:26.122+0000 7f34a7fff700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f34a80f5410 0x7f34a810b900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:35120/0 (socket says 172.21.15.132:35120) 2024-02-20T00:18:26.126 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:26.122+0000 7f34a7fff700 1 -- 172.21.15.132:0/1097455272 learned_addr learned my addr 172.21.15.132:0/1097455272 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:18:26.126 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:26.122+0000 7f34a7fff700 1 -- 172.21.15.132:0/1097455272 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3498005040 con 0x7f34a80f5410 2024-02-20T00:18:26.126 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:26.122+0000 7f34a7fff700 1 --2- 172.21.15.132:0/1097455272 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f34a80f5410 0x7f34a810b900 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 rx=0x7f34980089f0 tx=0x7f349800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:18:26.127 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:26.122+0000 7f34a4ff9700 1 -- 172.21.15.132:0/1097455272 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3498009320 con 0x7f34a80f5410 2024-02-20T00:18:26.127 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:26.122+0000 7f34ae269700 1 -- 172.21.15.132:0/1097455272 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34a8060220 con 0x7f34a80f5410 2024-02-20T00:18:26.127 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:26.122+0000 7f34a4ff9700 1 -- 172.21.15.132:0/1097455272 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3498013380 con 0x7f34a80f5410 2024-02-20T00:18:26.127 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:26.122+0000 7f34a4ff9700 1 -- 172.21.15.132:0/1097455272 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f349801ed80 con 0x7f34a80f5410 2024-02-20T00:18:26.127 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:26.122+0000 7f34ae269700 1 -- 172.21.15.132:0/1097455272 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34a810bdf0 con 0x7f34a80f5410 2024-02-20T00:18:26.127 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:26.123+0000 7f34a4ff9700 1 -- 172.21.15.132:0/1097455272 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f349801e420 con 0x7f34a80f5410 2024-02-20T00:18:26.127 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:26.123+0000 7f34ae269700 1 -- 172.21.15.132:0/1097455272 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34a8045ad0 con 0x7f34a80f5410 2024-02-20T00:18:26.127 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:26.123+0000 7f34a4ff9700 1 --2- 172.21.15.132:0/1097455272 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f349002ff60 0x7f3490032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:18:26.127 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:26.123+0000 7f34a4ff9700 1 -- 172.21.15.132:0/1097455272 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3498041080 con 0x7f34a80f5410 2024-02-20T00:18:26.127 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:26.124+0000 7f34a77fe700 1 --2- 172.21.15.132:0/1097455272 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f349002ff60 0x7f3490032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:18:26.127 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:26.124+0000 7f34a77fe700 1 --2- 172.21.15.132:0/1097455272 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f349002ff60 0x7f3490032420 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f349c0096a0 tx=0x7f349c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:18:26.130 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:26.128+0000 7f34a4ff9700 1 -- 172.21.15.132:0/1097455272 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f34980095f0 con 0x7f34a80f5410 2024-02-20T00:18:26.336 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:26.334+0000 7f34ae269700 1 -- 172.21.15.132:0/1097455272 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f34a810c5b0 con 0x7f34a80f5410 2024-02-20T00:18:26.337 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:26.336+0000 7f34a4ff9700 1 -- 172.21.15.132:0/1097455272 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f349801cb10 con 0x7f34a80f5410 2024-02-20T00:18:26.337 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:18:26.337 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:18:26.339 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:26.337+0000 7f34ae269700 1 -- 172.21.15.132:0/1097455272 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f349002ff60 msgr2=0x7f3490032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:18:26.339 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:26.337+0000 7f34ae269700 1 --2- 172.21.15.132:0/1097455272 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f349002ff60 0x7f3490032420 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f349c0096a0 tx=0x7f349c006b40).stop 2024-02-20T00:18:26.339 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:26.337+0000 7f34ae269700 1 -- 172.21.15.132:0/1097455272 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f34a80f5410 msgr2=0x7f34a810b900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:18:26.339 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:26.337+0000 7f34ae269700 1 --2- 172.21.15.132:0/1097455272 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f34a80f5410 0x7f34a810b900 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 rx=0x7f34980089f0 tx=0x7f349800b040).stop 2024-02-20T00:18:26.340 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:26.337+0000 7f34ae269700 1 -- 172.21.15.132:0/1097455272 shutdown_connections 2024-02-20T00:18:26.340 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:26.337+0000 7f34ae269700 1 --2- 172.21.15.132:0/1097455272 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f349002ff60 0x7f3490032420 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:18:26.340 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:26.337+0000 7f34ae269700 1 --2- 172.21.15.132:0/1097455272 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f34a80f5410 0x7f34a810b900 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:18:26.340 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:26.337+0000 7f34ae269700 1 -- 172.21.15.132:0/1097455272 >> 172.21.15.132:0/1097455272 conn(0x7f34a80f0b70 msgr2=0x7f34a80f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:18:26.340 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:26.337+0000 7f34ae269700 1 -- 172.21.15.132:0/1097455272 shutdown_connections 2024-02-20T00:18:26.340 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:26.337+0000 7f34ae269700 1 -- 172.21.15.132:0/1097455272 wait complete. 2024-02-20T00:18:26.340 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:18:26.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:18:25 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:18:24.915788+0000 mgr.smithi084.bmqrnf (mgr.14162) 262 : cluster [DBG] pgmap v248: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:18:27.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:18:26 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:18:26.336378+0000 mon.smithi084 (mon.0) 268 : audit [DBG] from='client.? 172.21.15.132:0/1097455272' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:18:28.054 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:18:28.054 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:18:28.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:18:27 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:18:26.916124+0000 mgr.smithi084.bmqrnf (mgr.14162) 263 : cluster [DBG] pgmap v249: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:18:30.099 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:30.068+0000 7f093aa6c700 1 -- 172.21.15.132:0/2657789777 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f09340f5610 msgr2=0x7f09340f5a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:18:30.099 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:30.068+0000 7f093aa6c700 1 --2- 172.21.15.132:0/2657789777 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f09340f5610 0x7f09340f5a30 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 rx=0x7f092c009960 tx=0x7f092c004d10).stop 2024-02-20T00:18:30.099 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:30.068+0000 7f093aa6c700 1 -- 172.21.15.132:0/2657789777 shutdown_connections 2024-02-20T00:18:30.099 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:30.068+0000 7f093aa6c700 1 --2- 172.21.15.132:0/2657789777 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f09340f5610 0x7f09340f5a30 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:18:30.100 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:30.068+0000 7f093aa6c700 1 -- 172.21.15.132:0/2657789777 >> 172.21.15.132:0/2657789777 conn(0x7f09340f0120 msgr2=0x7f09340f2540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:18:30.100 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:30.068+0000 7f093aa6c700 1 -- 172.21.15.132:0/2657789777 shutdown_connections 2024-02-20T00:18:30.100 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:30.068+0000 7f093aa6c700 1 -- 172.21.15.132:0/2657789777 wait complete. 2024-02-20T00:18:30.100 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:30.068+0000 7f093aa6c700 1 Processor -- start 2024-02-20T00:18:30.100 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:30.068+0000 7f093aa6c700 1 -- start start 2024-02-20T00:18:30.100 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:30.068+0000 7f093aa6c700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f09340f5610 0x7f093410cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:18:30.100 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:30.068+0000 7f093aa6c700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f093410d0a0 con 0x7f09340f5610 2024-02-20T00:18:30.100 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:30.069+0000 7f0939a6a700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f09340f5610 0x7f093410cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:18:30.100 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:30.069+0000 7f0939a6a700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f09340f5610 0x7f093410cbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:56256/0 (socket says 172.21.15.132:56256) 2024-02-20T00:18:30.100 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:30.069+0000 7f0939a6a700 1 -- 172.21.15.132:0/2717726172 learned_addr learned my addr 172.21.15.132:0/2717726172 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:18:30.100 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:30.069+0000 7f0939a6a700 1 -- 172.21.15.132:0/2717726172 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f092c009640 con 0x7f09340f5610 2024-02-20T00:18:30.100 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:30.069+0000 7f0939a6a700 1 --2- 172.21.15.132:0/2717726172 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f09340f5610 0x7f093410cbb0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0x7f092c007bd0 tx=0x7f092c004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:18:30.101 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:30.069+0000 7f092a7fc700 1 -- 172.21.15.132:0/2717726172 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f092c018070 con 0x7f09340f5610 2024-02-20T00:18:30.101 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:30.069+0000 7f093aa6c700 1 -- 172.21.15.132:0/2717726172 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f093410d2a0 con 0x7f09340f5610 2024-02-20T00:18:30.101 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:30.069+0000 7f092a7fc700 1 -- 172.21.15.132:0/2717726172 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f092c00b970 con 0x7f09340f5610 2024-02-20T00:18:30.101 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:30.069+0000 7f092a7fc700 1 -- 172.21.15.132:0/2717726172 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f092c01d600 con 0x7f09340f5610 2024-02-20T00:18:30.101 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:30.069+0000 7f093aa6c700 1 -- 172.21.15.132:0/2717726172 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f093410d730 con 0x7f09340f5610 2024-02-20T00:18:30.101 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:30.070+0000 7f093aa6c700 1 -- 172.21.15.132:0/2717726172 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0934045ad0 con 0x7f09340f5610 2024-02-20T00:18:30.101 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:30.070+0000 7f092a7fc700 1 -- 172.21.15.132:0/2717726172 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f092c024060 con 0x7f09340f5610 2024-02-20T00:18:30.101 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:30.070+0000 7f092a7fc700 1 --2- 172.21.15.132:0/2717726172 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f092002ff60 0x7f0920032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:18:30.101 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:30.070+0000 7f092a7fc700 1 -- 172.21.15.132:0/2717726172 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f092c041190 con 0x7f09340f5610 2024-02-20T00:18:30.101 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:30.070+0000 7f0939269700 1 --2- 172.21.15.132:0/2717726172 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f092002ff60 0x7f0920032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:18:30.101 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:30.071+0000 7f0939269700 1 --2- 172.21.15.132:0/2717726172 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f092002ff60 0x7f0920032420 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f09240096a0 tx=0x7f0924006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:18:30.101 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:30.075+0000 7f092a7fc700 1 -- 172.21.15.132:0/2717726172 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f092c01bb10 con 0x7f09340f5610 2024-02-20T00:18:30.280 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:30.279+0000 7f093aa6c700 1 -- 172.21.15.132:0/2717726172 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f093410ed50 con 0x7f09340f5610 2024-02-20T00:18:30.281 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:30.280+0000 7f092a7fc700 1 -- 172.21.15.132:0/2717726172 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f092c027e00 con 0x7f09340f5610 2024-02-20T00:18:30.281 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:18:30.281 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:18:30.283 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:30.282+0000 7f093aa6c700 1 -- 172.21.15.132:0/2717726172 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f092002ff60 msgr2=0x7f0920032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:18:30.283 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:30.282+0000 7f093aa6c700 1 --2- 172.21.15.132:0/2717726172 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f092002ff60 0x7f0920032420 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f09240096a0 tx=0x7f0924006b40).stop 2024-02-20T00:18:30.283 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:30.282+0000 7f093aa6c700 1 -- 172.21.15.132:0/2717726172 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f09340f5610 msgr2=0x7f093410cbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:18:30.283 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:30.282+0000 7f093aa6c700 1 --2- 172.21.15.132:0/2717726172 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f09340f5610 0x7f093410cbb0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0x7f092c007bd0 tx=0x7f092c004d10).stop 2024-02-20T00:18:30.283 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:30.282+0000 7f093aa6c700 1 -- 172.21.15.132:0/2717726172 shutdown_connections 2024-02-20T00:18:30.283 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:30.282+0000 7f093aa6c700 1 --2- 172.21.15.132:0/2717726172 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f092002ff60 0x7f0920032420 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:18:30.284 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:30.282+0000 7f093aa6c700 1 --2- 172.21.15.132:0/2717726172 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f09340f5610 0x7f093410cbb0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:18:30.284 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:30.282+0000 7f093aa6c700 1 -- 172.21.15.132:0/2717726172 >> 172.21.15.132:0/2717726172 conn(0x7f09340f0120 msgr2=0x7f09340f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:18:30.284 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:30.282+0000 7f093aa6c700 1 -- 172.21.15.132:0/2717726172 shutdown_connections 2024-02-20T00:18:30.284 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:30.282+0000 7f093aa6c700 1 -- 172.21.15.132:0/2717726172 wait complete. 2024-02-20T00:18:30.284 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:18:30.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:18:29 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:18:28.916488+0000 mgr.smithi084.bmqrnf (mgr.14162) 264 : cluster [DBG] pgmap v250: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:18:31.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:18:30 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:18:30.280135+0000 mon.smithi084 (mon.0) 269 : audit [DBG] from='client.? 172.21.15.132:0/2717726172' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:18:32.034 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:18:32.034 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:18:32.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:18:31 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:18:30.916813+0000 mgr.smithi084.bmqrnf (mgr.14162) 265 : cluster [DBG] pgmap v251: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:18:34.041 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:34.038+0000 7ff260673700 1 -- 172.21.15.132:0/4263259604 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff2580f6510 msgr2=0x7ff2580f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:18:34.041 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:34.038+0000 7ff260673700 1 --2- 172.21.15.132:0/4263259604 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff2580f6510 0x7ff2580f6930 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 rx=0x7ff248009fb0 tx=0x7ff248009b30).stop 2024-02-20T00:18:34.041 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:34.038+0000 7ff260673700 1 -- 172.21.15.132:0/4263259604 shutdown_connections 2024-02-20T00:18:34.041 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:34.038+0000 7ff260673700 1 --2- 172.21.15.132:0/4263259604 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff2580f6510 0x7ff2580f6930 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:18:34.041 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:34.038+0000 7ff260673700 1 -- 172.21.15.132:0/4263259604 >> 172.21.15.132:0/4263259604 conn(0x7ff2580f1c70 msgr2=0x7ff2580f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:18:34.042 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:34.038+0000 7ff260673700 1 -- 172.21.15.132:0/4263259604 shutdown_connections 2024-02-20T00:18:34.042 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:34.038+0000 7ff260673700 1 -- 172.21.15.132:0/4263259604 wait complete. 2024-02-20T00:18:34.042 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:34.038+0000 7ff260673700 1 Processor -- start 2024-02-20T00:18:34.042 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:34.039+0000 7ff260673700 1 -- start start 2024-02-20T00:18:34.042 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:34.039+0000 7ff260673700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff2580f6510 0x7ff258108a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:18:34.042 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:34.039+0000 7ff260673700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff258108ef0 con 0x7ff2580f6510 2024-02-20T00:18:34.042 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:34.039+0000 7ff25e40f700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff2580f6510 0x7ff258108a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:18:34.042 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:34.039+0000 7ff25e40f700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff2580f6510 0x7ff258108a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:56270/0 (socket says 172.21.15.132:56270) 2024-02-20T00:18:34.042 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:34.039+0000 7ff25e40f700 1 -- 172.21.15.132:0/2852032275 learned_addr learned my addr 172.21.15.132:0/2852032275 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:18:34.042 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:34.039+0000 7ff25e40f700 1 -- 172.21.15.132:0/2852032275 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff248005040 con 0x7ff2580f6510 2024-02-20T00:18:34.042 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:34.039+0000 7ff25e40f700 1 --2- 172.21.15.132:0/2852032275 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff2580f6510 0x7ff258108a00 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 rx=0x7ff248005010 tx=0x7ff24800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:18:34.042 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:34.039+0000 7ff24effd700 1 -- 172.21.15.132:0/2852032275 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff248009320 con 0x7ff2580f6510 2024-02-20T00:18:34.042 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:34.039+0000 7ff260673700 1 -- 172.21.15.132:0/2852032275 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2581090f0 con 0x7ff2580f6510 2024-02-20T00:18:34.042 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:34.040+0000 7ff24effd700 1 -- 172.21.15.132:0/2852032275 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff248013380 con 0x7ff2580f6510 2024-02-20T00:18:34.043 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:34.040+0000 7ff24effd700 1 -- 172.21.15.132:0/2852032275 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff248014e50 con 0x7ff2580f6510 2024-02-20T00:18:34.043 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:34.040+0000 7ff260673700 1 -- 172.21.15.132:0/2852032275 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff258109580 con 0x7ff2580f6510 2024-02-20T00:18:34.044 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:34.040+0000 7ff24effd700 1 -- 172.21.15.132:0/2852032275 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff2480095f0 con 0x7ff2580f6510 2024-02-20T00:18:34.044 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:34.040+0000 7ff260673700 1 -- 172.21.15.132:0/2852032275 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff258045ad0 con 0x7ff2580f6510 2024-02-20T00:18:34.044 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:34.040+0000 7ff24effd700 1 --2- 172.21.15.132:0/2852032275 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7ff24402ff10 0x7ff2440323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:18:34.044 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:34.040+0000 7ff24effd700 1 -- 172.21.15.132:0/2852032275 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff248040e70 con 0x7ff2580f6510 2024-02-20T00:18:34.044 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:34.041+0000 7ff25dc0e700 1 --2- 172.21.15.132:0/2852032275 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7ff24402ff10 0x7ff2440323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:18:34.044 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:34.041+0000 7ff25dc0e700 1 --2- 172.21.15.132:0/2852032275 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7ff24402ff10 0x7ff2440323d0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7ff2540096a0 tx=0x7ff254006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:18:34.046 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:34.045+0000 7ff24effd700 1 -- 172.21.15.132:0/2852032275 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff248014420 con 0x7ff2580f6510 2024-02-20T00:18:34.253 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:34.251+0000 7ff260673700 1 -- 172.21.15.132:0/2852032275 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff25810a3d0 con 0x7ff2580f6510 2024-02-20T00:18:34.253 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:34.252+0000 7ff24effd700 1 -- 172.21.15.132:0/2852032275 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff248010230 con 0x7ff2580f6510 2024-02-20T00:18:34.254 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:18:34.254 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:18:34.255 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:34.254+0000 7ff260673700 1 -- 172.21.15.132:0/2852032275 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7ff24402ff10 msgr2=0x7ff2440323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:18:34.255 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:34.254+0000 7ff260673700 1 --2- 172.21.15.132:0/2852032275 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7ff24402ff10 0x7ff2440323d0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7ff2540096a0 tx=0x7ff254006b40).stop 2024-02-20T00:18:34.256 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:34.254+0000 7ff260673700 1 -- 172.21.15.132:0/2852032275 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff2580f6510 msgr2=0x7ff258108a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:18:34.256 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:34.254+0000 7ff260673700 1 --2- 172.21.15.132:0/2852032275 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff2580f6510 0x7ff258108a00 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 rx=0x7ff248005010 tx=0x7ff24800b040).stop 2024-02-20T00:18:34.256 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:34.254+0000 7ff260673700 1 -- 172.21.15.132:0/2852032275 shutdown_connections 2024-02-20T00:18:34.256 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:34.254+0000 7ff260673700 1 --2- 172.21.15.132:0/2852032275 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7ff24402ff10 0x7ff2440323d0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:18:34.256 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:34.254+0000 7ff260673700 1 --2- 172.21.15.132:0/2852032275 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff2580f6510 0x7ff258108a00 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:18:34.256 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:34.254+0000 7ff260673700 1 -- 172.21.15.132:0/2852032275 >> 172.21.15.132:0/2852032275 conn(0x7ff2580f1c70 msgr2=0x7ff2580f26d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:18:34.256 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:34.254+0000 7ff260673700 1 -- 172.21.15.132:0/2852032275 shutdown_connections 2024-02-20T00:18:34.256 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:34.254+0000 7ff260673700 1 -- 172.21.15.132:0/2852032275 wait complete. 2024-02-20T00:18:34.256 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:18:34.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:18:33 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:18:32.917165+0000 mgr.smithi084.bmqrnf (mgr.14162) 266 : cluster [DBG] pgmap v252: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:18:35.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:18:34 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:18:34.252607+0000 mon.smithi084 (mon.0) 270 : audit [DBG] from='client.? 172.21.15.132:0/2852032275' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:18:35.984 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:18:35.984 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:18:36.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:18:35 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:18:34.917489+0000 mgr.smithi084.bmqrnf (mgr.14162) 267 : cluster [DBG] pgmap v253: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:18:37.899 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:37.877+0000 7f41dae72700 1 -- 172.21.15.132:0/3402677878 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f41d40f5410 msgr2=0x7f41d40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:18:37.900 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:37.877+0000 7f41dae72700 1 --2- 172.21.15.132:0/3402677878 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f41d40f5410 0x7f41d40f5830 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 rx=0x7f41c8009fb0 tx=0x7f41c8009b30).stop 2024-02-20T00:18:37.900 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:37.877+0000 7f41dae72700 1 -- 172.21.15.132:0/3402677878 shutdown_connections 2024-02-20T00:18:37.900 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:37.877+0000 7f41dae72700 1 --2- 172.21.15.132:0/3402677878 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f41d40f5410 0x7f41d40f5830 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:18:37.900 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:37.877+0000 7f41dae72700 1 -- 172.21.15.132:0/3402677878 >> 172.21.15.132:0/3402677878 conn(0x7f41d40f0b70 msgr2=0x7f41d40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:18:37.900 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:37.877+0000 7f41dae72700 1 -- 172.21.15.132:0/3402677878 shutdown_connections 2024-02-20T00:18:37.900 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:37.877+0000 7f41dae72700 1 -- 172.21.15.132:0/3402677878 wait complete. 2024-02-20T00:18:37.900 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:37.878+0000 7f41dae72700 1 Processor -- start 2024-02-20T00:18:37.900 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:37.878+0000 7f41dae72700 1 -- start start 2024-02-20T00:18:37.900 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:37.878+0000 7f41dae72700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f41d40f5410 0x7f41d41067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:18:37.900 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:37.878+0000 7f41dae72700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41d4106cd0 con 0x7f41d40f5410 2024-02-20T00:18:37.900 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:37.878+0000 7f41d8c0e700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f41d40f5410 0x7f41d41067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:18:37.901 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:37.879+0000 7f41d8c0e700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f41d40f5410 0x7f41d41067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:38304/0 (socket says 172.21.15.132:38304) 2024-02-20T00:18:37.901 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:37.879+0000 7f41d8c0e700 1 -- 172.21.15.132:0/2812517839 learned_addr learned my addr 172.21.15.132:0/2812517839 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:18:37.901 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:37.879+0000 7f41d8c0e700 1 -- 172.21.15.132:0/2812517839 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41c8005040 con 0x7f41d40f5410 2024-02-20T00:18:37.901 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:37.879+0000 7f41d8c0e700 1 --2- 172.21.15.132:0/2812517839 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f41d40f5410 0x7f41d41067e0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7f41c8005010 tx=0x7f41c800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:18:37.901 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:37.879+0000 7f41d17fa700 1 -- 172.21.15.132:0/2812517839 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f41c8009320 con 0x7f41d40f5410 2024-02-20T00:18:37.901 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:37.879+0000 7f41dae72700 1 -- 172.21.15.132:0/2812517839 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41d4106ed0 con 0x7f41d40f5410 2024-02-20T00:18:37.901 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:37.879+0000 7f41d17fa700 1 -- 172.21.15.132:0/2812517839 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f41c8013380 con 0x7f41d40f5410 2024-02-20T00:18:37.901 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:37.879+0000 7f41d17fa700 1 -- 172.21.15.132:0/2812517839 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f41c8014e50 con 0x7f41d40f5410 2024-02-20T00:18:37.901 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:37.879+0000 7f41dae72700 1 -- 172.21.15.132:0/2812517839 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41d4107360 con 0x7f41d40f5410 2024-02-20T00:18:37.901 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:37.880+0000 7f41dae72700 1 -- 172.21.15.132:0/2812517839 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41d4045ad0 con 0x7f41d40f5410 2024-02-20T00:18:37.901 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:37.880+0000 7f41d17fa700 1 -- 172.21.15.132:0/2812517839 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f41c80095f0 con 0x7f41d40f5410 2024-02-20T00:18:37.901 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:37.880+0000 7f41d17fa700 1 --2- 172.21.15.132:0/2812517839 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f41c402ff10 0x7f41c40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:18:37.901 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:37.880+0000 7f41d17fa700 1 -- 172.21.15.132:0/2812517839 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f41c8040e70 con 0x7f41d40f5410 2024-02-20T00:18:37.902 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:37.881+0000 7f41d3fff700 1 --2- 172.21.15.132:0/2812517839 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f41c402ff10 0x7f41c40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:18:37.902 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:37.881+0000 7f41d3fff700 1 --2- 172.21.15.132:0/2812517839 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f41c402ff10 0x7f41c40323d0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f41c00096a0 tx=0x7f41c0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:18:37.902 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:37.885+0000 7f41d17fa700 1 -- 172.21.15.132:0/2812517839 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f41c8014420 con 0x7f41d40f5410 2024-02-20T00:18:38.094 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:38.092+0000 7f41dae72700 1 -- 172.21.15.132:0/2812517839 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f41d4108210 con 0x7f41d40f5410 2024-02-20T00:18:38.095 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:38.092+0000 7f41d17fa700 1 -- 172.21.15.132:0/2812517839 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f41c8010230 con 0x7f41d40f5410 2024-02-20T00:18:38.095 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:18:38.095 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:18:38.096 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:38.094+0000 7f41dae72700 1 -- 172.21.15.132:0/2812517839 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f41c402ff10 msgr2=0x7f41c40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:18:38.096 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:38.094+0000 7f41dae72700 1 --2- 172.21.15.132:0/2812517839 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f41c402ff10 0x7f41c40323d0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f41c00096a0 tx=0x7f41c0006b40).stop 2024-02-20T00:18:38.096 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:38.094+0000 7f41dae72700 1 -- 172.21.15.132:0/2812517839 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f41d40f5410 msgr2=0x7f41d41067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:18:38.096 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:38.094+0000 7f41dae72700 1 --2- 172.21.15.132:0/2812517839 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f41d40f5410 0x7f41d41067e0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7f41c8005010 tx=0x7f41c800b040).stop 2024-02-20T00:18:38.096 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:38.094+0000 7f41dae72700 1 -- 172.21.15.132:0/2812517839 shutdown_connections 2024-02-20T00:18:38.097 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:38.094+0000 7f41dae72700 1 --2- 172.21.15.132:0/2812517839 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f41c402ff10 0x7f41c40323d0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:18:38.097 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:38.094+0000 7f41dae72700 1 --2- 172.21.15.132:0/2812517839 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f41d40f5410 0x7f41d41067e0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:18:38.097 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:38.094+0000 7f41dae72700 1 -- 172.21.15.132:0/2812517839 >> 172.21.15.132:0/2812517839 conn(0x7f41d40f0b70 msgr2=0x7f41d40f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:18:38.097 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:38.094+0000 7f41dae72700 1 -- 172.21.15.132:0/2812517839 shutdown_connections 2024-02-20T00:18:38.097 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:38.094+0000 7f41dae72700 1 -- 172.21.15.132:0/2812517839 wait complete. 2024-02-20T00:18:38.097 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:18:38.359 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:18:37 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:18:36.917800+0000 mgr.smithi084.bmqrnf (mgr.14162) 268 : cluster [DBG] pgmap v254: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:18:39.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:18:38 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:18:38.093001+0000 mon.smithi084 (mon.0) 271 : audit [DBG] from='client.? 172.21.15.132:0/2812517839' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:18:39.746 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:18:39.747 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:18:40.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:18:39 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:18:38.918157+0000 mgr.smithi084.bmqrnf (mgr.14162) 269 : cluster [DBG] pgmap v255: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:18:41.561 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:41.558+0000 7fc1ce63f700 1 -- 172.21.15.132:0/3049360033 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc1c80f62a0 msgr2=0x7fc1c80f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:18:41.561 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:41.558+0000 7fc1ce63f700 1 --2- 172.21.15.132:0/3049360033 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc1c80f62a0 0x7fc1c80f66c0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 rx=0x7fc1b8009fb0 tx=0x7fc1b8009b30).stop 2024-02-20T00:18:41.561 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:41.558+0000 7fc1ce63f700 1 -- 172.21.15.132:0/3049360033 shutdown_connections 2024-02-20T00:18:41.561 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:41.558+0000 7fc1ce63f700 1 --2- 172.21.15.132:0/3049360033 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc1c80f62a0 0x7fc1c80f66c0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:18:41.561 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:41.558+0000 7fc1ce63f700 1 -- 172.21.15.132:0/3049360033 >> 172.21.15.132:0/3049360033 conn(0x7fc1c80f1aa0 msgr2=0x7fc1c80f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:18:41.561 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:41.558+0000 7fc1ce63f700 1 -- 172.21.15.132:0/3049360033 shutdown_connections 2024-02-20T00:18:41.561 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:41.558+0000 7fc1ce63f700 1 -- 172.21.15.132:0/3049360033 wait complete. 2024-02-20T00:18:41.562 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:41.558+0000 7fc1ce63f700 1 Processor -- start 2024-02-20T00:18:41.562 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:41.559+0000 7fc1ce63f700 1 -- start start 2024-02-20T00:18:41.562 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:41.559+0000 7fc1ce63f700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc1c80f62a0 0x7fc1c81065d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:18:41.562 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:41.559+0000 7fc1ce63f700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1c8106ac0 con 0x7fc1c80f62a0 2024-02-20T00:18:41.562 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:41.559+0000 7fc1cd63d700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc1c80f62a0 0x7fc1c81065d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:18:41.562 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:41.559+0000 7fc1cd63d700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc1c80f62a0 0x7fc1c81065d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:38326/0 (socket says 172.21.15.132:38326) 2024-02-20T00:18:41.562 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:41.559+0000 7fc1cd63d700 1 -- 172.21.15.132:0/2435111166 learned_addr learned my addr 172.21.15.132:0/2435111166 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:18:41.562 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:41.559+0000 7fc1cd63d700 1 -- 172.21.15.132:0/2435111166 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1b8005040 con 0x7fc1c80f62a0 2024-02-20T00:18:41.562 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:41.559+0000 7fc1cd63d700 1 --2- 172.21.15.132:0/2435111166 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc1c80f62a0 0x7fc1c81065d0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0x7fc1b80089f0 tx=0x7fc1b800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:18:41.562 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:41.560+0000 7fc1bdffb700 1 -- 172.21.15.132:0/2435111166 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc1b8009320 con 0x7fc1c80f62a0 2024-02-20T00:18:41.562 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:41.560+0000 7fc1ce63f700 1 -- 172.21.15.132:0/2435111166 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1c8106cc0 con 0x7fc1c80f62a0 2024-02-20T00:18:41.562 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:41.560+0000 7fc1bdffb700 1 -- 172.21.15.132:0/2435111166 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc1b8013380 con 0x7fc1c80f62a0 2024-02-20T00:18:41.562 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:41.560+0000 7fc1bdffb700 1 -- 172.21.15.132:0/2435111166 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc1b8014e20 con 0x7fc1c80f62a0 2024-02-20T00:18:41.562 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:41.560+0000 7fc1ce63f700 1 -- 172.21.15.132:0/2435111166 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1c8107150 con 0x7fc1c80f62a0 2024-02-20T00:18:41.563 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:41.561+0000 7fc1bdffb700 1 -- 172.21.15.132:0/2435111166 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc1b80095f0 con 0x7fc1c80f62a0 2024-02-20T00:18:41.563 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:41.561+0000 7fc1ce63f700 1 -- 172.21.15.132:0/2435111166 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1c8045ad0 con 0x7fc1c80f62a0 2024-02-20T00:18:41.563 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:41.561+0000 7fc1bdffb700 1 --2- 172.21.15.132:0/2435111166 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fc1b402ff10 0x7fc1b40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:18:41.564 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:41.561+0000 7fc1bdffb700 1 -- 172.21.15.132:0/2435111166 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc1b8040f50 con 0x7fc1c80f62a0 2024-02-20T00:18:41.564 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:41.561+0000 7fc1cce3c700 1 --2- 172.21.15.132:0/2435111166 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fc1b402ff10 0x7fc1b40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:18:41.564 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:41.561+0000 7fc1cce3c700 1 --2- 172.21.15.132:0/2435111166 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fc1b402ff10 0x7fc1b40323d0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7fc1c40096a0 tx=0x7fc1c4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:18:41.566 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:41.565+0000 7fc1bdffb700 1 -- 172.21.15.132:0/2435111166 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc1b8014420 con 0x7fc1c80f62a0 2024-02-20T00:18:41.772 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:41.771+0000 7fc1ce63f700 1 -- 172.21.15.132:0/2435111166 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc1c8108000 con 0x7fc1c80f62a0 2024-02-20T00:18:41.773 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:41.772+0000 7fc1bdffb700 1 -- 172.21.15.132:0/2435111166 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc1b801d980 con 0x7fc1c80f62a0 2024-02-20T00:18:41.773 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:18:41.773 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:18:41.775 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:41.773+0000 7fc1ce63f700 1 -- 172.21.15.132:0/2435111166 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fc1b402ff10 msgr2=0x7fc1b40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:18:41.775 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:41.773+0000 7fc1ce63f700 1 --2- 172.21.15.132:0/2435111166 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fc1b402ff10 0x7fc1b40323d0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7fc1c40096a0 tx=0x7fc1c4006b40).stop 2024-02-20T00:18:41.775 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:41.773+0000 7fc1ce63f700 1 -- 172.21.15.132:0/2435111166 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc1c80f62a0 msgr2=0x7fc1c81065d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:18:41.775 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:41.773+0000 7fc1ce63f700 1 --2- 172.21.15.132:0/2435111166 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc1c80f62a0 0x7fc1c81065d0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0x7fc1b80089f0 tx=0x7fc1b800b040).stop 2024-02-20T00:18:41.775 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:41.773+0000 7fc1ce63f700 1 -- 172.21.15.132:0/2435111166 shutdown_connections 2024-02-20T00:18:41.775 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:41.773+0000 7fc1ce63f700 1 --2- 172.21.15.132:0/2435111166 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fc1b402ff10 0x7fc1b40323d0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:18:41.775 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:41.773+0000 7fc1ce63f700 1 --2- 172.21.15.132:0/2435111166 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc1c80f62a0 0x7fc1c81065d0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:18:41.775 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:41.773+0000 7fc1ce63f700 1 -- 172.21.15.132:0/2435111166 >> 172.21.15.132:0/2435111166 conn(0x7fc1c80f1aa0 msgr2=0x7fc1c80f2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:18:41.775 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:41.773+0000 7fc1ce63f700 1 -- 172.21.15.132:0/2435111166 shutdown_connections 2024-02-20T00:18:41.775 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:41.773+0000 7fc1ce63f700 1 -- 172.21.15.132:0/2435111166 wait complete. 2024-02-20T00:18:41.776 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:18:42.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:18:41 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:18:40.918558+0000 mgr.smithi084.bmqrnf (mgr.14162) 270 : cluster [DBG] pgmap v256: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:18:42.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:18:41 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:18:41.772396+0000 mon.smithi084 (mon.0) 272 : audit [DBG] from='client.? 172.21.15.132:0/2435111166' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:18:43.538 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:18:43.538 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:18:44.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:18:43 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:18:42.918873+0000 mgr.smithi084.bmqrnf (mgr.14162) 271 : cluster [DBG] pgmap v257: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:18:45.450 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:45.404+0000 7fe109ddf700 1 -- 172.21.15.132:0/1566632672 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe1040f5410 msgr2=0x7fe1040f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:18:45.450 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:45.404+0000 7fe109ddf700 1 --2- 172.21.15.132:0/1566632672 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe1040f5410 0x7fe1040f5830 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 rx=0x7fe0f4009fb0 tx=0x7fe0f4009b30).stop 2024-02-20T00:18:45.450 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:45.404+0000 7fe109ddf700 1 -- 172.21.15.132:0/1566632672 shutdown_connections 2024-02-20T00:18:45.450 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:45.404+0000 7fe109ddf700 1 --2- 172.21.15.132:0/1566632672 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe1040f5410 0x7fe1040f5830 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:18:45.450 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:45.404+0000 7fe109ddf700 1 -- 172.21.15.132:0/1566632672 >> 172.21.15.132:0/1566632672 conn(0x7fe1040f0b70 msgr2=0x7fe1040f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:18:45.451 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:45.404+0000 7fe109ddf700 1 -- 172.21.15.132:0/1566632672 shutdown_connections 2024-02-20T00:18:45.451 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:45.404+0000 7fe109ddf700 1 -- 172.21.15.132:0/1566632672 wait complete. 2024-02-20T00:18:45.451 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:45.404+0000 7fe109ddf700 1 Processor -- start 2024-02-20T00:18:45.451 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:45.405+0000 7fe109ddf700 1 -- start start 2024-02-20T00:18:45.451 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:45.405+0000 7fe109ddf700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe1040f5410 0x7fe1040fd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:18:45.451 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:45.405+0000 7fe109ddf700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1040fb960 con 0x7fe1040f5410 2024-02-20T00:18:45.451 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:45.405+0000 7fe1037fe700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe1040f5410 0x7fe1040fd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:18:45.451 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:45.405+0000 7fe1037fe700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe1040f5410 0x7fe1040fd4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:38358/0 (socket says 172.21.15.132:38358) 2024-02-20T00:18:45.451 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:45.405+0000 7fe1037fe700 1 -- 172.21.15.132:0/3336644083 learned_addr learned my addr 172.21.15.132:0/3336644083 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:18:45.451 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:45.405+0000 7fe1037fe700 1 -- 172.21.15.132:0/3336644083 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0f4005040 con 0x7fe1040f5410 2024-02-20T00:18:45.451 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:45.405+0000 7fe1037fe700 1 --2- 172.21.15.132:0/3336644083 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe1040f5410 0x7fe1040fd4e0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0x7fe0f40089f0 tx=0x7fe0f400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:18:45.452 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:45.406+0000 7fe0ebfff700 1 -- 172.21.15.132:0/3336644083 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe0f4009320 con 0x7fe1040f5410 2024-02-20T00:18:45.452 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:45.406+0000 7fe109ddf700 1 -- 172.21.15.132:0/3336644083 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1040fbb00 con 0x7fe1040f5410 2024-02-20T00:18:45.452 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:45.406+0000 7fe0ebfff700 1 -- 172.21.15.132:0/3336644083 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe0f4013380 con 0x7fe1040f5410 2024-02-20T00:18:45.452 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:45.406+0000 7fe0ebfff700 1 -- 172.21.15.132:0/3336644083 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe0f401ed80 con 0x7fe1040f5410 2024-02-20T00:18:45.452 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:45.406+0000 7fe109ddf700 1 -- 172.21.15.132:0/3336644083 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1040fbf90 con 0x7fe1040f5410 2024-02-20T00:18:45.452 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:45.406+0000 7fe0ebfff700 1 -- 172.21.15.132:0/3336644083 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe0f401e420 con 0x7fe1040f5410 2024-02-20T00:18:45.452 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:45.406+0000 7fe109ddf700 1 -- 172.21.15.132:0/3336644083 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe104059100 con 0x7fe1040f5410 2024-02-20T00:18:45.452 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:45.407+0000 7fe0ebfff700 1 --2- 172.21.15.132:0/3336644083 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fe0ec02ff60 0x7fe0ec032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:18:45.452 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:45.407+0000 7fe0ebfff700 1 -- 172.21.15.132:0/3336644083 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe0f4040f30 con 0x7fe1040f5410 2024-02-20T00:18:45.452 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:45.407+0000 7fe102ffd700 1 --2- 172.21.15.132:0/3336644083 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fe0ec02ff60 0x7fe0ec032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:18:45.452 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:45.407+0000 7fe102ffd700 1 --2- 172.21.15.132:0/3336644083 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fe0ec02ff60 0x7fe0ec032420 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7fe0f80096a0 tx=0x7fe0f8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:18:45.452 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:45.411+0000 7fe0ebfff700 1 -- 172.21.15.132:0/3336644083 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe0f400e2a0 con 0x7fe1040f5410 2024-02-20T00:18:45.617 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:45.615+0000 7fe109ddf700 1 -- 172.21.15.132:0/3336644083 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe1040fafa0 con 0x7fe1040f5410 2024-02-20T00:18:45.617 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:45.616+0000 7fe0ebfff700 1 -- 172.21.15.132:0/3336644083 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe0f400e2a0 con 0x7fe1040f5410 2024-02-20T00:18:45.617 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:18:45.618 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:18:45.620 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:45.618+0000 7fe109ddf700 1 -- 172.21.15.132:0/3336644083 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fe0ec02ff60 msgr2=0x7fe0ec032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:18:45.620 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:45.618+0000 7fe109ddf700 1 --2- 172.21.15.132:0/3336644083 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fe0ec02ff60 0x7fe0ec032420 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7fe0f80096a0 tx=0x7fe0f8006b40).stop 2024-02-20T00:18:45.620 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:45.618+0000 7fe109ddf700 1 -- 172.21.15.132:0/3336644083 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe1040f5410 msgr2=0x7fe1040fd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:18:45.621 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:45.618+0000 7fe109ddf700 1 --2- 172.21.15.132:0/3336644083 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe1040f5410 0x7fe1040fd4e0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0x7fe0f40089f0 tx=0x7fe0f400b040).stop 2024-02-20T00:18:45.621 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:45.618+0000 7fe109ddf700 1 -- 172.21.15.132:0/3336644083 shutdown_connections 2024-02-20T00:18:45.621 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:45.618+0000 7fe109ddf700 1 --2- 172.21.15.132:0/3336644083 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fe0ec02ff60 0x7fe0ec032420 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:18:45.621 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:45.618+0000 7fe109ddf700 1 --2- 172.21.15.132:0/3336644083 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe1040f5410 0x7fe1040fd4e0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:18:45.621 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:45.618+0000 7fe109ddf700 1 -- 172.21.15.132:0/3336644083 >> 172.21.15.132:0/3336644083 conn(0x7fe1040f0b70 msgr2=0x7fe10405fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:18:45.621 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:45.618+0000 7fe109ddf700 1 -- 172.21.15.132:0/3336644083 shutdown_connections 2024-02-20T00:18:45.622 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:45.618+0000 7fe109ddf700 1 -- 172.21.15.132:0/3336644083 wait complete. 2024-02-20T00:18:45.622 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:18:46.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:18:45 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:18:44.919262+0000 mgr.smithi084.bmqrnf (mgr.14162) 272 : cluster [DBG] pgmap v258: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:18:46.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:18:45 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:18:45.616716+0000 mon.smithi084 (mon.0) 273 : audit [DBG] from='client.? 172.21.15.132:0/3336644083' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:18:47.276 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:18:47.276 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:18:48.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:18:47 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:18:46.919650+0000 mgr.smithi084.bmqrnf (mgr.14162) 273 : cluster [DBG] pgmap v259: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:18:49.165 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:49.161+0000 7f6c64458700 1 -- 172.21.15.132:0/310526645 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6c5c0f6510 msgr2=0x7f6c5c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:18:49.165 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:49.161+0000 7f6c64458700 1 --2- 172.21.15.132:0/310526645 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6c5c0f6510 0x7f6c5c0f6930 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 rx=0x7f6c58009fb0 tx=0x7f6c58009b30).stop 2024-02-20T00:18:49.165 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:49.161+0000 7f6c64458700 1 -- 172.21.15.132:0/310526645 shutdown_connections 2024-02-20T00:18:49.165 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:49.161+0000 7f6c64458700 1 --2- 172.21.15.132:0/310526645 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6c5c0f6510 0x7f6c5c0f6930 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:18:49.165 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:49.161+0000 7f6c64458700 1 -- 172.21.15.132:0/310526645 >> 172.21.15.132:0/310526645 conn(0x7f6c5c0f1c70 msgr2=0x7f6c5c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:18:49.165 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:49.161+0000 7f6c64458700 1 -- 172.21.15.132:0/310526645 shutdown_connections 2024-02-20T00:18:49.165 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:49.161+0000 7f6c64458700 1 -- 172.21.15.132:0/310526645 wait complete. 2024-02-20T00:18:49.165 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:49.161+0000 7f6c64458700 1 Processor -- start 2024-02-20T00:18:49.165 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:49.161+0000 7f6c64458700 1 -- start start 2024-02-20T00:18:49.166 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:49.161+0000 7f6c64458700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6c5c0f6510 0x7f6c5c1067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:18:49.166 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:49.161+0000 7f6c64458700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c5c106cd0 con 0x7f6c5c0f6510 2024-02-20T00:18:49.166 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:49.162+0000 7f6c621f4700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6c5c0f6510 0x7f6c5c1067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:18:49.166 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:49.162+0000 7f6c621f4700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6c5c0f6510 0x7f6c5c1067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:55868/0 (socket says 172.21.15.132:55868) 2024-02-20T00:18:49.166 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:49.162+0000 7f6c621f4700 1 -- 172.21.15.132:0/2339285042 learned_addr learned my addr 172.21.15.132:0/2339285042 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:18:49.166 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:49.162+0000 7f6c621f4700 1 -- 172.21.15.132:0/2339285042 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c58005040 con 0x7f6c5c0f6510 2024-02-20T00:18:49.166 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:49.162+0000 7f6c621f4700 1 --2- 172.21.15.132:0/2339285042 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6c5c0f6510 0x7f6c5c1067e0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7f6c58005010 tx=0x7f6c5800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:18:49.166 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:49.162+0000 7f6c52ffd700 1 -- 172.21.15.132:0/2339285042 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6c58009320 con 0x7f6c5c0f6510 2024-02-20T00:18:49.166 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:49.162+0000 7f6c64458700 1 -- 172.21.15.132:0/2339285042 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c5c106ed0 con 0x7f6c5c0f6510 2024-02-20T00:18:49.166 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:49.162+0000 7f6c52ffd700 1 -- 172.21.15.132:0/2339285042 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6c58013380 con 0x7f6c5c0f6510 2024-02-20T00:18:49.166 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:49.162+0000 7f6c52ffd700 1 -- 172.21.15.132:0/2339285042 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6c58014e50 con 0x7f6c5c0f6510 2024-02-20T00:18:49.166 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:49.162+0000 7f6c64458700 1 -- 172.21.15.132:0/2339285042 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c5c107360 con 0x7f6c5c0f6510 2024-02-20T00:18:49.167 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:49.163+0000 7f6c52ffd700 1 -- 172.21.15.132:0/2339285042 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6c580095f0 con 0x7f6c5c0f6510 2024-02-20T00:18:49.167 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:49.163+0000 7f6c64458700 1 -- 172.21.15.132:0/2339285042 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c5c045ad0 con 0x7f6c5c0f6510 2024-02-20T00:18:49.167 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:49.163+0000 7f6c52ffd700 1 --2- 172.21.15.132:0/2339285042 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f6c4802ff10 0x7f6c480323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:18:49.167 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:49.163+0000 7f6c52ffd700 1 -- 172.21.15.132:0/2339285042 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6c58040fa0 con 0x7f6c5c0f6510 2024-02-20T00:18:49.167 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:49.163+0000 7f6c619f3700 1 --2- 172.21.15.132:0/2339285042 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f6c4802ff10 0x7f6c480323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:18:49.168 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:49.164+0000 7f6c619f3700 1 --2- 172.21.15.132:0/2339285042 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f6c4802ff10 0x7f6c480323d0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f6c4c0096a0 tx=0x7f6c4c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:18:49.170 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:49.168+0000 7f6c52ffd700 1 -- 172.21.15.132:0/2339285042 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6c58014420 con 0x7f6c5c0f6510 2024-02-20T00:18:49.375 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:49.372+0000 7f6c64458700 1 -- 172.21.15.132:0/2339285042 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6c5c108210 con 0x7f6c5c0f6510 2024-02-20T00:18:49.376 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:49.373+0000 7f6c52ffd700 1 -- 172.21.15.132:0/2339285042 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6c58010230 con 0x7f6c5c0f6510 2024-02-20T00:18:49.376 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:18:49.376 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:18:49.378 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:49.375+0000 7f6c64458700 1 -- 172.21.15.132:0/2339285042 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f6c4802ff10 msgr2=0x7f6c480323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:18:49.378 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:49.375+0000 7f6c64458700 1 --2- 172.21.15.132:0/2339285042 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f6c4802ff10 0x7f6c480323d0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f6c4c0096a0 tx=0x7f6c4c006b40).stop 2024-02-20T00:18:49.378 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:49.375+0000 7f6c64458700 1 -- 172.21.15.132:0/2339285042 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6c5c0f6510 msgr2=0x7f6c5c1067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:18:49.378 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:49.375+0000 7f6c64458700 1 --2- 172.21.15.132:0/2339285042 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6c5c0f6510 0x7f6c5c1067e0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7f6c58005010 tx=0x7f6c5800b040).stop 2024-02-20T00:18:49.378 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:49.375+0000 7f6c64458700 1 -- 172.21.15.132:0/2339285042 shutdown_connections 2024-02-20T00:18:49.378 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:49.375+0000 7f6c64458700 1 --2- 172.21.15.132:0/2339285042 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f6c4802ff10 0x7f6c480323d0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:18:49.378 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:49.375+0000 7f6c64458700 1 --2- 172.21.15.132:0/2339285042 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6c5c0f6510 0x7f6c5c1067e0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:18:49.378 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:49.375+0000 7f6c64458700 1 -- 172.21.15.132:0/2339285042 >> 172.21.15.132:0/2339285042 conn(0x7f6c5c0f1c70 msgr2=0x7f6c5c0f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:18:49.378 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:49.375+0000 7f6c64458700 1 -- 172.21.15.132:0/2339285042 shutdown_connections 2024-02-20T00:18:49.378 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:49.375+0000 7f6c64458700 1 -- 172.21.15.132:0/2339285042 wait complete. 2024-02-20T00:18:49.379 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:18:50.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:18:49 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:18:48.920006+0000 mgr.smithi084.bmqrnf (mgr.14162) 274 : cluster [DBG] pgmap v260: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:18:50.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:18:49 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:18:49.374564+0000 mon.smithi084 (mon.0) 274 : audit [DBG] from='client.? 172.21.15.132:0/2339285042' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:18:51.048 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:18:51.049 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:18:52.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:18:51 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:18:50.920352+0000 mgr.smithi084.bmqrnf (mgr.14162) 275 : cluster [DBG] pgmap v261: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:18:52.911 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:52.905+0000 7f2d78321700 1 -- 172.21.15.132:0/3717554645 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2d700f5410 msgr2=0x7f2d700f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:18:52.912 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:52.905+0000 7f2d78321700 1 --2- 172.21.15.132:0/3717554645 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2d700f5410 0x7f2d700f5830 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 rx=0x7f2d64009fb0 tx=0x7f2d64009b30).stop 2024-02-20T00:18:52.912 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:52.905+0000 7f2d78321700 1 -- 172.21.15.132:0/3717554645 shutdown_connections 2024-02-20T00:18:52.912 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:52.905+0000 7f2d78321700 1 --2- 172.21.15.132:0/3717554645 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2d700f5410 0x7f2d700f5830 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:18:52.912 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:52.905+0000 7f2d78321700 1 -- 172.21.15.132:0/3717554645 >> 172.21.15.132:0/3717554645 conn(0x7f2d700f0b70 msgr2=0x7f2d700f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:18:52.912 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:52.905+0000 7f2d78321700 1 -- 172.21.15.132:0/3717554645 shutdown_connections 2024-02-20T00:18:52.913 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:52.905+0000 7f2d78321700 1 -- 172.21.15.132:0/3717554645 wait complete. 2024-02-20T00:18:52.913 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:52.906+0000 7f2d78321700 1 Processor -- start 2024-02-20T00:18:52.913 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:52.906+0000 7f2d78321700 1 -- start start 2024-02-20T00:18:52.913 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:52.906+0000 7f2d78321700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2d700f5410 0x7f2d701056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:18:52.913 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:52.906+0000 7f2d78321700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d70105bd0 con 0x7f2d700f5410 2024-02-20T00:18:52.913 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:52.906+0000 7f2d760bd700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2d700f5410 0x7f2d701056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:18:52.914 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:52.906+0000 7f2d760bd700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2d700f5410 0x7f2d701056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:55892/0 (socket says 172.21.15.132:55892) 2024-02-20T00:18:52.914 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:52.906+0000 7f2d760bd700 1 -- 172.21.15.132:0/2595943 learned_addr learned my addr 172.21.15.132:0/2595943 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:18:52.914 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:52.907+0000 7f2d760bd700 1 -- 172.21.15.132:0/2595943 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d64005040 con 0x7f2d700f5410 2024-02-20T00:18:52.914 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:52.907+0000 7f2d760bd700 1 --2- 172.21.15.132:0/2595943 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2d700f5410 0x7f2d701056e0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7f2d64005010 tx=0x7f2d6400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:18:52.914 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:52.907+0000 7f2d62ffd700 1 -- 172.21.15.132:0/2595943 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2d64009320 con 0x7f2d700f5410 2024-02-20T00:18:52.914 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:52.907+0000 7f2d62ffd700 1 -- 172.21.15.132:0/2595943 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2d64013380 con 0x7f2d700f5410 2024-02-20T00:18:52.915 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:52.907+0000 7f2d78321700 1 -- 172.21.15.132:0/2595943 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d70105dd0 con 0x7f2d700f5410 2024-02-20T00:18:52.915 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:52.907+0000 7f2d62ffd700 1 -- 172.21.15.132:0/2595943 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2d64014e50 con 0x7f2d700f5410 2024-02-20T00:18:52.915 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:52.907+0000 7f2d78321700 1 -- 172.21.15.132:0/2595943 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d70106260 con 0x7f2d700f5410 2024-02-20T00:18:52.915 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:52.908+0000 7f2d62ffd700 1 -- 172.21.15.132:0/2595943 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2d640095f0 con 0x7f2d700f5410 2024-02-20T00:18:52.915 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:52.908+0000 7f2d78321700 1 -- 172.21.15.132:0/2595943 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d70045ad0 con 0x7f2d700f5410 2024-02-20T00:18:52.915 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:52.908+0000 7f2d62ffd700 1 --2- 172.21.15.132:0/2595943 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f2d5c02ff10 0x7f2d5c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:18:52.916 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:52.908+0000 7f2d62ffd700 1 -- 172.21.15.132:0/2595943 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2d64040e90 con 0x7f2d700f5410 2024-02-20T00:18:52.916 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:52.908+0000 7f2d758bc700 1 --2- 172.21.15.132:0/2595943 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f2d5c02ff10 0x7f2d5c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:18:52.916 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:52.908+0000 7f2d758bc700 1 --2- 172.21.15.132:0/2595943 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f2d5c02ff10 0x7f2d5c0323d0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f2d6c0096a0 tx=0x7f2d6c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:18:52.917 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:52.913+0000 7f2d62ffd700 1 -- 172.21.15.132:0/2595943 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2d64014420 con 0x7f2d700f5410 2024-02-20T00:18:53.117 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:53.115+0000 7f2d78321700 1 -- 172.21.15.132:0/2595943 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2d70107110 con 0x7f2d700f5410 2024-02-20T00:18:53.118 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:53.115+0000 7f2d62ffd700 1 -- 172.21.15.132:0/2595943 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2d64010230 con 0x7f2d700f5410 2024-02-20T00:18:53.118 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:18:53.119 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:18:53.120 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:53.117+0000 7f2d78321700 1 -- 172.21.15.132:0/2595943 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f2d5c02ff10 msgr2=0x7f2d5c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:18:53.120 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:53.117+0000 7f2d78321700 1 --2- 172.21.15.132:0/2595943 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f2d5c02ff10 0x7f2d5c0323d0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f2d6c0096a0 tx=0x7f2d6c006b40).stop 2024-02-20T00:18:53.120 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:53.117+0000 7f2d78321700 1 -- 172.21.15.132:0/2595943 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2d700f5410 msgr2=0x7f2d701056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:18:53.121 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:53.117+0000 7f2d78321700 1 --2- 172.21.15.132:0/2595943 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2d700f5410 0x7f2d701056e0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7f2d64005010 tx=0x7f2d6400b040).stop 2024-02-20T00:18:53.121 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:53.117+0000 7f2d78321700 1 -- 172.21.15.132:0/2595943 shutdown_connections 2024-02-20T00:18:53.121 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:53.117+0000 7f2d78321700 1 --2- 172.21.15.132:0/2595943 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f2d5c02ff10 0x7f2d5c0323d0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:18:53.121 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:53.117+0000 7f2d78321700 1 --2- 172.21.15.132:0/2595943 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2d700f5410 0x7f2d701056e0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:18:53.121 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:53.117+0000 7f2d78321700 1 -- 172.21.15.132:0/2595943 >> 172.21.15.132:0/2595943 conn(0x7f2d700f0b70 msgr2=0x7f2d700f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:18:53.121 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:53.117+0000 7f2d78321700 1 -- 172.21.15.132:0/2595943 shutdown_connections 2024-02-20T00:18:53.121 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:53.117+0000 7f2d78321700 1 -- 172.21.15.132:0/2595943 wait complete. 2024-02-20T00:18:53.122 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:18:54.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:18:53 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:18:52.920640+0000 mgr.smithi084.bmqrnf (mgr.14162) 276 : cluster [DBG] pgmap v262: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:18:54.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:18:53 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:18:53.117073+0000 mon.smithi084 (mon.0) 275 : audit [DBG] from='client.? 172.21.15.132:0/2595943' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:18:54.784 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:18:54.784 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:18:56.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:18:55 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:18:54.920958+0000 mgr.smithi084.bmqrnf (mgr.14162) 277 : cluster [DBG] pgmap v263: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:18:56.830 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:56.809+0000 7f9c7af53700 1 -- 172.21.15.132:0/389562208 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9c740f6510 msgr2=0x7f9c740f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:18:56.830 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:56.809+0000 7f9c7af53700 1 --2- 172.21.15.132:0/389562208 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9c740f6510 0x7f9c740f6930 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 rx=0x7f9c68009fb0 tx=0x7f9c68009b30).stop 2024-02-20T00:18:56.830 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:56.809+0000 7f9c7af53700 1 -- 172.21.15.132:0/389562208 shutdown_connections 2024-02-20T00:18:56.830 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:56.809+0000 7f9c7af53700 1 --2- 172.21.15.132:0/389562208 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9c740f6510 0x7f9c740f6930 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:18:56.830 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:56.809+0000 7f9c7af53700 1 -- 172.21.15.132:0/389562208 >> 172.21.15.132:0/389562208 conn(0x7f9c740f1c70 msgr2=0x7f9c740f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:18:56.830 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:56.809+0000 7f9c7af53700 1 -- 172.21.15.132:0/389562208 shutdown_connections 2024-02-20T00:18:56.830 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:56.809+0000 7f9c7af53700 1 -- 172.21.15.132:0/389562208 wait complete. 2024-02-20T00:18:56.830 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:56.809+0000 7f9c7af53700 1 Processor -- start 2024-02-20T00:18:56.830 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:56.809+0000 7f9c7af53700 1 -- start start 2024-02-20T00:18:56.830 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:56.810+0000 7f9c7af53700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9c740f6510 0x7f9c741067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:18:56.830 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:56.810+0000 7f9c7af53700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c74106cd0 con 0x7f9c740f6510 2024-02-20T00:18:56.831 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:56.810+0000 7f9c78cef700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9c740f6510 0x7f9c741067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:18:56.831 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:56.810+0000 7f9c78cef700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9c740f6510 0x7f9c741067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:55908/0 (socket says 172.21.15.132:55908) 2024-02-20T00:18:56.831 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:56.810+0000 7f9c78cef700 1 -- 172.21.15.132:0/2533253127 learned_addr learned my addr 172.21.15.132:0/2533253127 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:18:56.831 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:56.810+0000 7f9c78cef700 1 -- 172.21.15.132:0/2533253127 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c68005040 con 0x7f9c740f6510 2024-02-20T00:18:56.831 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:56.811+0000 7f9c78cef700 1 --2- 172.21.15.132:0/2533253127 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9c740f6510 0x7f9c741067e0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 rx=0x7f9c68005010 tx=0x7f9c6800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:18:56.831 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:56.811+0000 7f9c717fa700 1 -- 172.21.15.132:0/2533253127 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9c68009320 con 0x7f9c740f6510 2024-02-20T00:18:56.831 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:56.811+0000 7f9c7af53700 1 -- 172.21.15.132:0/2533253127 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c74106ed0 con 0x7f9c740f6510 2024-02-20T00:18:56.831 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:56.811+0000 7f9c717fa700 1 -- 172.21.15.132:0/2533253127 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9c68013380 con 0x7f9c740f6510 2024-02-20T00:18:56.831 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:56.811+0000 7f9c717fa700 1 -- 172.21.15.132:0/2533253127 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9c68014e50 con 0x7f9c740f6510 2024-02-20T00:18:56.831 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:56.811+0000 7f9c7af53700 1 -- 172.21.15.132:0/2533253127 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c74107360 con 0x7f9c740f6510 2024-02-20T00:18:56.831 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:56.812+0000 7f9c717fa700 1 -- 172.21.15.132:0/2533253127 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9c680095f0 con 0x7f9c740f6510 2024-02-20T00:18:56.831 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:56.812+0000 7f9c7af53700 1 -- 172.21.15.132:0/2533253127 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c74045ad0 con 0x7f9c740f6510 2024-02-20T00:18:56.831 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:56.812+0000 7f9c717fa700 1 --2- 172.21.15.132:0/2533253127 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f9c6402ff10 0x7f9c640323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:18:56.831 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:56.812+0000 7f9c717fa700 1 -- 172.21.15.132:0/2533253127 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9c68040e90 con 0x7f9c740f6510 2024-02-20T00:18:56.832 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:56.812+0000 7f9c73fff700 1 --2- 172.21.15.132:0/2533253127 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f9c6402ff10 0x7f9c640323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:18:56.832 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:56.812+0000 7f9c73fff700 1 --2- 172.21.15.132:0/2533253127 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f9c6402ff10 0x7f9c640323d0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f9c600096a0 tx=0x7f9c60006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:18:56.832 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:56.817+0000 7f9c717fa700 1 -- 172.21.15.132:0/2533253127 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9c68014420 con 0x7f9c740f6510 2024-02-20T00:18:57.024 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:57.022+0000 7f9c7af53700 1 -- 172.21.15.132:0/2533253127 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9c74108210 con 0x7f9c740f6510 2024-02-20T00:18:57.024 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:57.022+0000 7f9c717fa700 1 -- 172.21.15.132:0/2533253127 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9c68010230 con 0x7f9c740f6510 2024-02-20T00:18:57.025 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:18:57.025 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:18:57.026 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:57.024+0000 7f9c7af53700 1 -- 172.21.15.132:0/2533253127 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f9c6402ff10 msgr2=0x7f9c640323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:18:57.026 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:57.024+0000 7f9c7af53700 1 --2- 172.21.15.132:0/2533253127 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f9c6402ff10 0x7f9c640323d0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f9c600096a0 tx=0x7f9c60006b40).stop 2024-02-20T00:18:57.026 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:57.024+0000 7f9c7af53700 1 -- 172.21.15.132:0/2533253127 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9c740f6510 msgr2=0x7f9c741067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:18:57.026 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:57.024+0000 7f9c7af53700 1 --2- 172.21.15.132:0/2533253127 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9c740f6510 0x7f9c741067e0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 rx=0x7f9c68005010 tx=0x7f9c6800b040).stop 2024-02-20T00:18:57.027 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:57.024+0000 7f9c7af53700 1 -- 172.21.15.132:0/2533253127 shutdown_connections 2024-02-20T00:18:57.027 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:57.024+0000 7f9c7af53700 1 --2- 172.21.15.132:0/2533253127 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f9c6402ff10 0x7f9c640323d0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:18:57.027 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:57.024+0000 7f9c7af53700 1 --2- 172.21.15.132:0/2533253127 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9c740f6510 0x7f9c741067e0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:18:57.027 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:57.024+0000 7f9c7af53700 1 -- 172.21.15.132:0/2533253127 >> 172.21.15.132:0/2533253127 conn(0x7f9c740f1c70 msgr2=0x7f9c740f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:18:57.027 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:57.024+0000 7f9c7af53700 1 -- 172.21.15.132:0/2533253127 shutdown_connections 2024-02-20T00:18:57.027 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:18:57.024+0000 7f9c7af53700 1 -- 172.21.15.132:0/2533253127 wait complete. 2024-02-20T00:18:57.027 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:18:58.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:18:57 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:18:56.921243+0000 mgr.smithi084.bmqrnf (mgr.14162) 278 : cluster [DBG] pgmap v264: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:18:58.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:18:57 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:18:56.981042+0000 mon.smithi084 (mon.0) 276 : audit [INF] from='mgr.14162 172.21.15.84:0/2152933216' entity='mgr.smithi084.bmqrnf' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi084.bmqrnf/mirror_snapshot_schedule"}]: dispatch 2024-02-20T00:18:58.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:18:57 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:18:56.984591+0000 mon.smithi084 (mon.0) 277 : audit [INF] from='mgr.14162 172.21.15.84:0/2152933216' entity='mgr.smithi084.bmqrnf' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi084.bmqrnf/trash_purge_schedule"}]: dispatch 2024-02-20T00:18:58.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:18:57 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:18:57.023816+0000 mon.smithi084 (mon.0) 278 : audit [DBG] from='client.? 172.21.15.132:0/2533253127' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:18:58.595 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:18:58.596 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:19:00.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:18:59 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:18:58.921476+0000 mgr.smithi084.bmqrnf (mgr.14162) 279 : cluster [DBG] pgmap v265: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:19:00.442 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:00.397+0000 7fde34a14700 1 -- 172.21.15.132:0/1345815767 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fde300f51a0 msgr2=0x7fde300f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:19:00.442 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:00.397+0000 7fde34a14700 1 --2- 172.21.15.132:0/1345815767 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fde300f51a0 0x7fde300f55c0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 rx=0x7fde20009fb0 tx=0x7fde20009b30).stop 2024-02-20T00:19:00.442 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:00.398+0000 7fde34a14700 1 -- 172.21.15.132:0/1345815767 shutdown_connections 2024-02-20T00:19:00.442 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:00.398+0000 7fde34a14700 1 --2- 172.21.15.132:0/1345815767 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fde300f51a0 0x7fde300f55c0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:19:00.442 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:00.398+0000 7fde34a14700 1 -- 172.21.15.132:0/1345815767 >> 172.21.15.132:0/1345815767 conn(0x7fde300f09a0 msgr2=0x7fde300f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:19:00.442 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:00.398+0000 7fde34a14700 1 -- 172.21.15.132:0/1345815767 shutdown_connections 2024-02-20T00:19:00.442 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:00.398+0000 7fde34a14700 1 -- 172.21.15.132:0/1345815767 wait complete. 2024-02-20T00:19:00.443 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:00.398+0000 7fde34a14700 1 Processor -- start 2024-02-20T00:19:00.443 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:00.398+0000 7fde34a14700 1 -- start start 2024-02-20T00:19:00.443 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:00.398+0000 7fde34a14700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fde300f51a0 0x7fde30104390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:19:00.443 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:00.398+0000 7fde34a14700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde30104880 con 0x7fde300f51a0 2024-02-20T00:19:00.443 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:00.399+0000 7fde2ed9d700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fde300f51a0 0x7fde30104390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:19:00.443 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:00.399+0000 7fde2ed9d700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fde300f51a0 0x7fde30104390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:34946/0 (socket says 172.21.15.132:34946) 2024-02-20T00:19:00.443 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:00.399+0000 7fde2ed9d700 1 -- 172.21.15.132:0/3847406564 learned_addr learned my addr 172.21.15.132:0/3847406564 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:19:00.443 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:00.399+0000 7fde2ed9d700 1 -- 172.21.15.132:0/3847406564 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde20005040 con 0x7fde300f51a0 2024-02-20T00:19:00.443 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:00.399+0000 7fde2ed9d700 1 --2- 172.21.15.132:0/3847406564 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fde300f51a0 0x7fde30104390 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 rx=0x7fde200089f0 tx=0x7fde2000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:19:00.443 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:00.399+0000 7fde177fe700 1 -- 172.21.15.132:0/3847406564 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fde20009320 con 0x7fde300f51a0 2024-02-20T00:19:00.443 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:00.399+0000 7fde34a14700 1 -- 172.21.15.132:0/3847406564 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde30104a80 con 0x7fde300f51a0 2024-02-20T00:19:00.443 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:00.399+0000 7fde177fe700 1 -- 172.21.15.132:0/3847406564 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fde20013380 con 0x7fde300f51a0 2024-02-20T00:19:00.443 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:00.399+0000 7fde177fe700 1 -- 172.21.15.132:0/3847406564 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fde20014e20 con 0x7fde300f51a0 2024-02-20T00:19:00.444 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:00.399+0000 7fde34a14700 1 -- 172.21.15.132:0/3847406564 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde30104f10 con 0x7fde300f51a0 2024-02-20T00:19:00.444 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:00.400+0000 7fde177fe700 1 -- 172.21.15.132:0/3847406564 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fde20014420 con 0x7fde300f51a0 2024-02-20T00:19:00.444 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:00.400+0000 7fde34a14700 1 -- 172.21.15.132:0/3847406564 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde30045ad0 con 0x7fde300f51a0 2024-02-20T00:19:00.444 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:00.400+0000 7fde177fe700 1 --2- 172.21.15.132:0/3847406564 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fde180321b0 0x7fde18034670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:19:00.444 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:00.400+0000 7fde177fe700 1 -- 172.21.15.132:0/3847406564 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fde20040ec0 con 0x7fde300f51a0 2024-02-20T00:19:00.444 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:00.400+0000 7fde2e59c700 1 --2- 172.21.15.132:0/3847406564 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fde180321b0 0x7fde18034670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:19:00.444 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:00.401+0000 7fde2e59c700 1 --2- 172.21.15.132:0/3847406564 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fde180321b0 0x7fde18034670 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7fde240096a0 tx=0x7fde24006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:19:00.444 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:00.405+0000 7fde177fe700 1 -- 172.21.15.132:0/3847406564 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fde20014790 con 0x7fde300f51a0 2024-02-20T00:19:00.614 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:00.611+0000 7fde34a14700 1 -- 172.21.15.132:0/3847406564 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fde30105dc0 con 0x7fde300f51a0 2024-02-20T00:19:00.615 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:00.612+0000 7fde177fe700 1 -- 172.21.15.132:0/3847406564 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fde20004490 con 0x7fde300f51a0 2024-02-20T00:19:00.615 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:19:00.615 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:19:00.616 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:00.614+0000 7fde34a14700 1 -- 172.21.15.132:0/3847406564 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fde180321b0 msgr2=0x7fde18034670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:19:00.617 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:00.614+0000 7fde34a14700 1 --2- 172.21.15.132:0/3847406564 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fde180321b0 0x7fde18034670 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7fde240096a0 tx=0x7fde24006b40).stop 2024-02-20T00:19:00.617 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:00.614+0000 7fde34a14700 1 -- 172.21.15.132:0/3847406564 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fde300f51a0 msgr2=0x7fde30104390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:19:00.617 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:00.614+0000 7fde34a14700 1 --2- 172.21.15.132:0/3847406564 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fde300f51a0 0x7fde30104390 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 rx=0x7fde200089f0 tx=0x7fde2000b040).stop 2024-02-20T00:19:00.617 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:00.614+0000 7fde34a14700 1 -- 172.21.15.132:0/3847406564 shutdown_connections 2024-02-20T00:19:00.617 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:00.614+0000 7fde34a14700 1 --2- 172.21.15.132:0/3847406564 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fde180321b0 0x7fde18034670 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:19:00.617 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:00.614+0000 7fde34a14700 1 --2- 172.21.15.132:0/3847406564 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fde300f51a0 0x7fde30104390 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:19:00.617 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:00.614+0000 7fde34a14700 1 -- 172.21.15.132:0/3847406564 >> 172.21.15.132:0/3847406564 conn(0x7fde300f09a0 msgr2=0x7fde300f13c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:19:00.617 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:00.614+0000 7fde34a14700 1 -- 172.21.15.132:0/3847406564 shutdown_connections 2024-02-20T00:19:00.617 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:00.614+0000 7fde34a14700 1 -- 172.21.15.132:0/3847406564 wait complete. 2024-02-20T00:19:00.617 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:19:01.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:19:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:19:00.613834+0000 mon.smithi084 (mon.0) 279 : audit [DBG] from='client.? 172.21.15.132:0/3847406564' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:19:02.189 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:19:02.189 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:19:02.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:19:01 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:19:00.921827+0000 mgr.smithi084.bmqrnf (mgr.14162) 280 : cluster [DBG] pgmap v266: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:19:04.063 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:04.050+0000 7f19ae6cc700 1 -- 172.21.15.132:0/1858230582 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f19a80f6510 msgr2=0x7f19a80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:19:04.063 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:04.050+0000 7f19ae6cc700 1 --2- 172.21.15.132:0/1858230582 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f19a80f6510 0x7f19a80f6930 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 rx=0x7f1998009fb0 tx=0x7f1998009b30).stop 2024-02-20T00:19:04.063 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:04.050+0000 7f19ae6cc700 1 -- 172.21.15.132:0/1858230582 shutdown_connections 2024-02-20T00:19:04.063 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:04.050+0000 7f19ae6cc700 1 --2- 172.21.15.132:0/1858230582 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f19a80f6510 0x7f19a80f6930 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:19:04.063 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:04.050+0000 7f19ae6cc700 1 -- 172.21.15.132:0/1858230582 >> 172.21.15.132:0/1858230582 conn(0x7f19a80f1c70 msgr2=0x7f19a80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:19:04.063 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:04.050+0000 7f19ae6cc700 1 -- 172.21.15.132:0/1858230582 shutdown_connections 2024-02-20T00:19:04.063 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:04.050+0000 7f19ae6cc700 1 -- 172.21.15.132:0/1858230582 wait complete. 2024-02-20T00:19:04.063 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:04.050+0000 7f19ae6cc700 1 Processor -- start 2024-02-20T00:19:04.063 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:04.051+0000 7f19ae6cc700 1 -- start start 2024-02-20T00:19:04.064 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:04.051+0000 7f19ae6cc700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f19a80f6510 0x7f19a81067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:19:04.064 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:04.051+0000 7f19ae6cc700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19a8106cd0 con 0x7f19a80f6510 2024-02-20T00:19:04.064 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:04.051+0000 7f19a7fff700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f19a80f6510 0x7f19a81067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:19:04.064 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:04.051+0000 7f19a7fff700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f19a80f6510 0x7f19a81067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:34952/0 (socket says 172.21.15.132:34952) 2024-02-20T00:19:04.064 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:04.051+0000 7f19a7fff700 1 -- 172.21.15.132:0/2771744862 learned_addr learned my addr 172.21.15.132:0/2771744862 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:19:04.064 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:04.052+0000 7f19a7fff700 1 -- 172.21.15.132:0/2771744862 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1998005040 con 0x7f19a80f6510 2024-02-20T00:19:04.064 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:04.052+0000 7f19a7fff700 1 --2- 172.21.15.132:0/2771744862 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f19a80f6510 0x7f19a81067e0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0x7f19980089f0 tx=0x7f199800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:19:04.064 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:04.052+0000 7f19a4ff9700 1 -- 172.21.15.132:0/2771744862 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1998009320 con 0x7f19a80f6510 2024-02-20T00:19:04.064 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:04.052+0000 7f19ae6cc700 1 -- 172.21.15.132:0/2771744862 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19a8106ed0 con 0x7f19a80f6510 2024-02-20T00:19:04.064 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:04.052+0000 7f19a4ff9700 1 -- 172.21.15.132:0/2771744862 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1998013380 con 0x7f19a80f6510 2024-02-20T00:19:04.064 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:04.052+0000 7f19a4ff9700 1 -- 172.21.15.132:0/2771744862 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f199801ed80 con 0x7f19a80f6510 2024-02-20T00:19:04.064 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:04.052+0000 7f19ae6cc700 1 -- 172.21.15.132:0/2771744862 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19a8107360 con 0x7f19a80f6510 2024-02-20T00:19:04.064 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:04.053+0000 7f19a4ff9700 1 -- 172.21.15.132:0/2771744862 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1998014420 con 0x7f19a80f6510 2024-02-20T00:19:04.064 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:04.053+0000 7f19a4ff9700 1 --2- 172.21.15.132:0/2771744862 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f199402ff60 0x7f1994032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:19:04.065 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:04.053+0000 7f19a4ff9700 1 -- 172.21.15.132:0/2771744862 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1998040f00 con 0x7f19a80f6510 2024-02-20T00:19:04.065 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:04.053+0000 7f19ae6cc700 1 -- 172.21.15.132:0/2771744862 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19a8059100 con 0x7f19a80f6510 2024-02-20T00:19:04.065 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:04.053+0000 7f19a77fe700 1 --2- 172.21.15.132:0/2771744862 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f199402ff60 0x7f1994032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:19:04.065 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:04.053+0000 7f19a77fe700 1 --2- 172.21.15.132:0/2771744862 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f199402ff60 0x7f1994032420 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f19900096a0 tx=0x7f1990006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:19:04.065 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:04.058+0000 7f19a4ff9700 1 -- 172.21.15.132:0/2771744862 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f199801c910 con 0x7f19a80f6510 2024-02-20T00:19:04.261 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:04.259+0000 7f19ae6cc700 1 -- 172.21.15.132:0/2771744862 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f19a80fc0a0 con 0x7f19a80f6510 2024-02-20T00:19:04.262 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:04.260+0000 7f19a4ff9700 1 -- 172.21.15.132:0/2771744862 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f199801c320 con 0x7f19a80f6510 2024-02-20T00:19:04.262 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:19:04.263 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:19:04.264 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:04.261+0000 7f19ae6cc700 1 -- 172.21.15.132:0/2771744862 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f199402ff60 msgr2=0x7f1994032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:19:04.264 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:04.261+0000 7f19ae6cc700 1 --2- 172.21.15.132:0/2771744862 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f199402ff60 0x7f1994032420 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f19900096a0 tx=0x7f1990006b40).stop 2024-02-20T00:19:04.264 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:04.261+0000 7f19ae6cc700 1 -- 172.21.15.132:0/2771744862 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f19a80f6510 msgr2=0x7f19a81067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:19:04.264 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:04.261+0000 7f19ae6cc700 1 --2- 172.21.15.132:0/2771744862 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f19a80f6510 0x7f19a81067e0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0x7f19980089f0 tx=0x7f199800b040).stop 2024-02-20T00:19:04.264 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:04.262+0000 7f19ae6cc700 1 -- 172.21.15.132:0/2771744862 shutdown_connections 2024-02-20T00:19:04.265 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:04.262+0000 7f19ae6cc700 1 --2- 172.21.15.132:0/2771744862 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f199402ff60 0x7f1994032420 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:19:04.265 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:04.262+0000 7f19ae6cc700 1 --2- 172.21.15.132:0/2771744862 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f19a80f6510 0x7f19a81067e0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:19:04.265 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:04.262+0000 7f19ae6cc700 1 -- 172.21.15.132:0/2771744862 >> 172.21.15.132:0/2771744862 conn(0x7f19a80f1c70 msgr2=0x7f19a80f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:19:04.265 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:04.262+0000 7f19ae6cc700 1 -- 172.21.15.132:0/2771744862 shutdown_connections 2024-02-20T00:19:04.265 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:04.262+0000 7f19ae6cc700 1 -- 172.21.15.132:0/2771744862 wait complete. 2024-02-20T00:19:04.265 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:19:04.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:19:03 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:19:02.922208+0000 mgr.smithi084.bmqrnf (mgr.14162) 281 : cluster [DBG] pgmap v267: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:19:05.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:19:04 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:19:04.261326+0000 mon.smithi084 (mon.0) 280 : audit [DBG] from='client.? 172.21.15.132:0/2771744862' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:19:05.888 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:19:05.888 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:19:06.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:19:05 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:19:04.922446+0000 mgr.smithi084.bmqrnf (mgr.14162) 282 : cluster [DBG] pgmap v268: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:19:07.720 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:07.718+0000 7f9176e3f700 1 -- 172.21.15.132:0/2829133437 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f91700f5410 msgr2=0x7f91700f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:19:07.720 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:07.718+0000 7f9176e3f700 1 --2- 172.21.15.132:0/2829133437 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f91700f5410 0x7f91700f5830 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 rx=0x7f9164009fb0 tx=0x7f9164009b30).stop 2024-02-20T00:19:07.720 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:07.718+0000 7f9176e3f700 1 -- 172.21.15.132:0/2829133437 shutdown_connections 2024-02-20T00:19:07.720 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:07.718+0000 7f9176e3f700 1 --2- 172.21.15.132:0/2829133437 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f91700f5410 0x7f91700f5830 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:19:07.720 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:07.718+0000 7f9176e3f700 1 -- 172.21.15.132:0/2829133437 >> 172.21.15.132:0/2829133437 conn(0x7f91700f0b70 msgr2=0x7f91700f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:19:07.720 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:07.718+0000 7f9176e3f700 1 -- 172.21.15.132:0/2829133437 shutdown_connections 2024-02-20T00:19:07.720 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:07.718+0000 7f9176e3f700 1 -- 172.21.15.132:0/2829133437 wait complete. 2024-02-20T00:19:07.722 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:07.719+0000 7f9176e3f700 1 Processor -- start 2024-02-20T00:19:07.722 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:07.719+0000 7f9176e3f700 1 -- start start 2024-02-20T00:19:07.722 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:07.719+0000 7f9176e3f700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f91700f5410 0x7f91701094a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:19:07.722 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:07.719+0000 7f9176e3f700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9170060280 con 0x7f91700f5410 2024-02-20T00:19:07.722 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:07.720+0000 7f9174bdb700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f91700f5410 0x7f91701094a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:19:07.722 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:07.720+0000 7f9174bdb700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f91700f5410 0x7f91701094a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:39048/0 (socket says 172.21.15.132:39048) 2024-02-20T00:19:07.722 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:07.720+0000 7f9174bdb700 1 -- 172.21.15.132:0/1328545545 learned_addr learned my addr 172.21.15.132:0/1328545545 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:19:07.722 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:07.720+0000 7f9174bdb700 1 -- 172.21.15.132:0/1328545545 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9164005040 con 0x7f91700f5410 2024-02-20T00:19:07.722 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:07.720+0000 7f9174bdb700 1 --2- 172.21.15.132:0/1328545545 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f91700f5410 0x7f91701094a0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 rx=0x7f9164005010 tx=0x7f916400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:19:07.722 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:07.720+0000 7f916d7fa700 1 -- 172.21.15.132:0/1328545545 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9164009320 con 0x7f91700f5410 2024-02-20T00:19:07.722 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:07.720+0000 7f9176e3f700 1 -- 172.21.15.132:0/1328545545 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9170060480 con 0x7f91700f5410 2024-02-20T00:19:07.723 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:07.720+0000 7f916d7fa700 1 -- 172.21.15.132:0/1328545545 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9164013380 con 0x7f91700f5410 2024-02-20T00:19:07.723 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:07.720+0000 7f916d7fa700 1 -- 172.21.15.132:0/1328545545 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9164014e50 con 0x7f91700f5410 2024-02-20T00:19:07.723 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:07.720+0000 7f9176e3f700 1 -- 172.21.15.132:0/1328545545 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9170109bd0 con 0x7f91700f5410 2024-02-20T00:19:07.724 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:07.721+0000 7f916d7fa700 1 -- 172.21.15.132:0/1328545545 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f91640263a0 con 0x7f91700f5410 2024-02-20T00:19:07.724 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:07.721+0000 7f916d7fa700 1 --2- 172.21.15.132:0/1328545545 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f916002ff10 0x7f91600323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:19:07.724 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:07.721+0000 7f9176e3f700 1 -- 172.21.15.132:0/1328545545 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9170045ad0 con 0x7f91700f5410 2024-02-20T00:19:07.724 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:07.721+0000 7f916d7fa700 1 -- 172.21.15.132:0/1328545545 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9164041260 con 0x7f91700f5410 2024-02-20T00:19:07.724 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:07.721+0000 7f916ffff700 1 --2- 172.21.15.132:0/1328545545 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f916002ff10 0x7f91600323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:19:07.724 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:07.722+0000 7f916ffff700 1 --2- 172.21.15.132:0/1328545545 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f916002ff10 0x7f91600323d0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7f915c0096a0 tx=0x7f915c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:19:07.728 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:07.726+0000 7f916d7fa700 1 -- 172.21.15.132:0/1328545545 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f91640126f0 con 0x7f91700f5410 2024-02-20T00:19:07.933 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:07.931+0000 7f9176e3f700 1 -- 172.21.15.132:0/1328545545 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f91700fafa0 con 0x7f91700f5410 2024-02-20T00:19:07.933 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:07.932+0000 7f916d7fa700 1 -- 172.21.15.132:0/1328545545 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9164012330 con 0x7f91700f5410 2024-02-20T00:19:07.934 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:19:07.934 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:19:07.935 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:07.933+0000 7f9176e3f700 1 -- 172.21.15.132:0/1328545545 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f916002ff10 msgr2=0x7f91600323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:19:07.935 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:07.933+0000 7f9176e3f700 1 --2- 172.21.15.132:0/1328545545 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f916002ff10 0x7f91600323d0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7f915c0096a0 tx=0x7f915c006b40).stop 2024-02-20T00:19:07.935 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:07.933+0000 7f9176e3f700 1 -- 172.21.15.132:0/1328545545 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f91700f5410 msgr2=0x7f91701094a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:19:07.935 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:07.933+0000 7f9176e3f700 1 --2- 172.21.15.132:0/1328545545 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f91700f5410 0x7f91701094a0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 rx=0x7f9164005010 tx=0x7f916400b040).stop 2024-02-20T00:19:07.936 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:07.933+0000 7f9176e3f700 1 -- 172.21.15.132:0/1328545545 shutdown_connections 2024-02-20T00:19:07.936 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:07.934+0000 7f9176e3f700 1 --2- 172.21.15.132:0/1328545545 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f916002ff10 0x7f91600323d0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:19:07.936 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:07.934+0000 7f9176e3f700 1 --2- 172.21.15.132:0/1328545545 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f91700f5410 0x7f91701094a0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:19:07.936 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:07.934+0000 7f9176e3f700 1 -- 172.21.15.132:0/1328545545 >> 172.21.15.132:0/1328545545 conn(0x7f91700f0b70 msgr2=0x7f91700f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:19:07.936 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:07.934+0000 7f9176e3f700 1 -- 172.21.15.132:0/1328545545 shutdown_connections 2024-02-20T00:19:07.936 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:07.934+0000 7f9176e3f700 1 -- 172.21.15.132:0/1328545545 wait complete. 2024-02-20T00:19:07.936 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:19:08.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:19:07 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:19:06.922803+0000 mgr.smithi084.bmqrnf (mgr.14162) 283 : cluster [DBG] pgmap v269: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:19:08.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:19:07 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:19:07.932636+0000 mon.smithi084 (mon.0) 281 : audit [DBG] from='client.? 172.21.15.132:0/1328545545' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:19:09.564 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:19:09.564 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:19:10.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:19:09 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:19:08.923139+0000 mgr.smithi084.bmqrnf (mgr.14162) 284 : cluster [DBG] pgmap v270: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:19:11.443 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:11.441+0000 7f6cdcf3f700 1 -- 172.21.15.132:0/1063815991 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6cd80f62a0 msgr2=0x7f6cd80f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:19:11.443 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:11.441+0000 7f6cdcf3f700 1 --2- 172.21.15.132:0/1063815991 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6cd80f62a0 0x7f6cd80f66c0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 rx=0x7f6cc0009fb0 tx=0x7f6cc0009b30).stop 2024-02-20T00:19:11.443 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:11.441+0000 7f6cdcf3f700 1 -- 172.21.15.132:0/1063815991 shutdown_connections 2024-02-20T00:19:11.443 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:11.441+0000 7f6cdcf3f700 1 --2- 172.21.15.132:0/1063815991 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6cd80f62a0 0x7f6cd80f66c0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:19:11.443 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:11.441+0000 7f6cdcf3f700 1 -- 172.21.15.132:0/1063815991 >> 172.21.15.132:0/1063815991 conn(0x7f6cd80f1aa0 msgr2=0x7f6cd80f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:19:11.443 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:11.441+0000 7f6cdcf3f700 1 -- 172.21.15.132:0/1063815991 shutdown_connections 2024-02-20T00:19:11.443 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:11.441+0000 7f6cdcf3f700 1 -- 172.21.15.132:0/1063815991 wait complete. 2024-02-20T00:19:11.443 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:11.441+0000 7f6cdcf3f700 1 Processor -- start 2024-02-20T00:19:11.443 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:11.441+0000 7f6cdcf3f700 1 -- start start 2024-02-20T00:19:11.445 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:11.442+0000 7f6cdcf3f700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6cd80f62a0 0x7f6cd81065d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:19:11.445 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:11.442+0000 7f6cdcf3f700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cd8106ac0 con 0x7f6cd80f62a0 2024-02-20T00:19:11.445 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:11.442+0000 7f6cd77fe700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6cd80f62a0 0x7f6cd81065d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:19:11.445 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:11.442+0000 7f6cd77fe700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6cd80f62a0 0x7f6cd81065d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:39078/0 (socket says 172.21.15.132:39078) 2024-02-20T00:19:11.445 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:11.442+0000 7f6cd77fe700 1 -- 172.21.15.132:0/1533271101 learned_addr learned my addr 172.21.15.132:0/1533271101 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:19:11.445 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:11.442+0000 7f6cd77fe700 1 -- 172.21.15.132:0/1533271101 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cc0005040 con 0x7f6cd80f62a0 2024-02-20T00:19:11.446 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:11.442+0000 7f6cd77fe700 1 --2- 172.21.15.132:0/1533271101 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6cd80f62a0 0x7f6cd81065d0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0x7f6cc00089f0 tx=0x7f6cc000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:19:11.446 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:11.442+0000 7f6ccffff700 1 -- 172.21.15.132:0/1533271101 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6cc0009320 con 0x7f6cd80f62a0 2024-02-20T00:19:11.446 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:11.442+0000 7f6cdcf3f700 1 -- 172.21.15.132:0/1533271101 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6cd8106cc0 con 0x7f6cd80f62a0 2024-02-20T00:19:11.446 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:11.442+0000 7f6ccffff700 1 -- 172.21.15.132:0/1533271101 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6cc0013380 con 0x7f6cd80f62a0 2024-02-20T00:19:11.446 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:11.442+0000 7f6ccffff700 1 -- 172.21.15.132:0/1533271101 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6cc0014e20 con 0x7f6cd80f62a0 2024-02-20T00:19:11.446 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:11.443+0000 7f6cdcf3f700 1 -- 172.21.15.132:0/1533271101 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cd8107150 con 0x7f6cd80f62a0 2024-02-20T00:19:11.446 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:11.443+0000 7f6ccffff700 1 -- 172.21.15.132:0/1533271101 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6cc0014420 con 0x7f6cd80f62a0 2024-02-20T00:19:11.446 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:11.443+0000 7f6cdcf3f700 1 -- 172.21.15.132:0/1533271101 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6cd8045ad0 con 0x7f6cd80f62a0 2024-02-20T00:19:11.446 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:11.443+0000 7f6ccffff700 1 --2- 172.21.15.132:0/1533271101 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f6cc402ff10 0x7f6cc40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:19:11.446 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:11.444+0000 7f6ccffff700 1 -- 172.21.15.132:0/1533271101 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6cc0040ff0 con 0x7f6cd80f62a0 2024-02-20T00:19:11.446 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:11.444+0000 7f6cd6ffd700 1 --2- 172.21.15.132:0/1533271101 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f6cc402ff10 0x7f6cc40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:19:11.446 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:11.444+0000 7f6cd6ffd700 1 --2- 172.21.15.132:0/1533271101 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f6cc402ff10 0x7f6cc40323d0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f6cc80096a0 tx=0x7f6cc8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:19:11.449 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:11.448+0000 7f6ccffff700 1 -- 172.21.15.132:0/1533271101 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6cc00095f0 con 0x7f6cd80f62a0 2024-02-20T00:19:11.652 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:11.651+0000 7f6cdcf3f700 1 -- 172.21.15.132:0/1533271101 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6cd8108000 con 0x7f6cd80f62a0 2024-02-20T00:19:11.653 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:11.652+0000 7f6ccffff700 1 -- 172.21.15.132:0/1533271101 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6cc001db70 con 0x7f6cd80f62a0 2024-02-20T00:19:11.653 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:19:11.653 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:19:11.655 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:11.653+0000 7f6cdcf3f700 1 -- 172.21.15.132:0/1533271101 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f6cc402ff10 msgr2=0x7f6cc40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:19:11.655 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:11.653+0000 7f6cdcf3f700 1 --2- 172.21.15.132:0/1533271101 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f6cc402ff10 0x7f6cc40323d0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f6cc80096a0 tx=0x7f6cc8006b40).stop 2024-02-20T00:19:11.655 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:11.653+0000 7f6cdcf3f700 1 -- 172.21.15.132:0/1533271101 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6cd80f62a0 msgr2=0x7f6cd81065d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:19:11.655 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:11.653+0000 7f6cdcf3f700 1 --2- 172.21.15.132:0/1533271101 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6cd80f62a0 0x7f6cd81065d0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0x7f6cc00089f0 tx=0x7f6cc000b040).stop 2024-02-20T00:19:11.655 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:11.653+0000 7f6cdcf3f700 1 -- 172.21.15.132:0/1533271101 shutdown_connections 2024-02-20T00:19:11.655 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:11.653+0000 7f6cdcf3f700 1 --2- 172.21.15.132:0/1533271101 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f6cc402ff10 0x7f6cc40323d0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:19:11.655 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:11.653+0000 7f6cdcf3f700 1 --2- 172.21.15.132:0/1533271101 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6cd80f62a0 0x7f6cd81065d0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:19:11.656 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:11.653+0000 7f6cdcf3f700 1 -- 172.21.15.132:0/1533271101 >> 172.21.15.132:0/1533271101 conn(0x7f6cd80f1aa0 msgr2=0x7f6cd80f2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:19:11.656 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:11.654+0000 7f6cdcf3f700 1 -- 172.21.15.132:0/1533271101 shutdown_connections 2024-02-20T00:19:11.656 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:11.654+0000 7f6cdcf3f700 1 -- 172.21.15.132:0/1533271101 wait complete. 2024-02-20T00:19:11.656 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:19:12.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:19:11 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:19:10.923487+0000 mgr.smithi084.bmqrnf (mgr.14162) 285 : cluster [DBG] pgmap v271: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:19:12.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:19:11 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:19:11.652350+0000 mon.smithi084 (mon.0) 282 : audit [DBG] from='client.? 172.21.15.132:0/1533271101' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:19:13.245 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:19:13.245 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:19:14.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:19:13 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:19:12.923866+0000 mgr.smithi084.bmqrnf (mgr.14162) 286 : cluster [DBG] pgmap v272: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:19:15.193 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:15.191+0000 7f3d59bf6700 1 -- 172.21.15.132:0/1423769357 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3d540f62a0 msgr2=0x7f3d540f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:19:15.194 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:15.191+0000 7f3d59bf6700 1 --2- 172.21.15.132:0/1423769357 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3d540f62a0 0x7f3d540f66c0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 rx=0x7f3d48009fb0 tx=0x7f3d48009b30).stop 2024-02-20T00:19:15.194 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:15.191+0000 7f3d59bf6700 1 -- 172.21.15.132:0/1423769357 shutdown_connections 2024-02-20T00:19:15.194 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:15.191+0000 7f3d59bf6700 1 --2- 172.21.15.132:0/1423769357 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3d540f62a0 0x7f3d540f66c0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:19:15.194 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:15.191+0000 7f3d59bf6700 1 -- 172.21.15.132:0/1423769357 >> 172.21.15.132:0/1423769357 conn(0x7f3d540f1aa0 msgr2=0x7f3d540f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:19:15.194 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:15.191+0000 7f3d59bf6700 1 -- 172.21.15.132:0/1423769357 shutdown_connections 2024-02-20T00:19:15.194 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:15.191+0000 7f3d59bf6700 1 -- 172.21.15.132:0/1423769357 wait complete. 2024-02-20T00:19:15.194 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:15.192+0000 7f3d59bf6700 1 Processor -- start 2024-02-20T00:19:15.194 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:15.192+0000 7f3d59bf6700 1 -- start start 2024-02-20T00:19:15.194 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:15.192+0000 7f3d59bf6700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3d540f62a0 0x7f3d5406b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:19:15.194 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:15.192+0000 7f3d59bf6700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d54069e20 con 0x7f3d540f62a0 2024-02-20T00:19:15.196 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:15.192+0000 7f3d58bf4700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3d540f62a0 0x7f3d5406b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:19:15.196 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:15.192+0000 7f3d58bf4700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3d540f62a0 0x7f3d5406b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:39096/0 (socket says 172.21.15.132:39096) 2024-02-20T00:19:15.196 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:15.192+0000 7f3d58bf4700 1 -- 172.21.15.132:0/2563598025 learned_addr learned my addr 172.21.15.132:0/2563598025 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:19:15.196 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:15.193+0000 7f3d58bf4700 1 -- 172.21.15.132:0/2563598025 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d48005040 con 0x7f3d540f62a0 2024-02-20T00:19:15.196 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:15.193+0000 7f3d58bf4700 1 --2- 172.21.15.132:0/2563598025 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3d540f62a0 0x7f3d5406b9a0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0x7f3d480089f0 tx=0x7f3d4800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:19:15.196 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:15.193+0000 7f3d517fa700 1 -- 172.21.15.132:0/2563598025 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3d48009320 con 0x7f3d540f62a0 2024-02-20T00:19:15.196 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:15.193+0000 7f3d59bf6700 1 -- 172.21.15.132:0/2563598025 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d54069fc0 con 0x7f3d540f62a0 2024-02-20T00:19:15.196 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:15.193+0000 7f3d517fa700 1 -- 172.21.15.132:0/2563598025 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3d48013380 con 0x7f3d540f62a0 2024-02-20T00:19:15.196 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:15.193+0000 7f3d517fa700 1 -- 172.21.15.132:0/2563598025 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3d48014e20 con 0x7f3d540f62a0 2024-02-20T00:19:15.196 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:15.193+0000 7f3d59bf6700 1 -- 172.21.15.132:0/2563598025 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d5406a450 con 0x7f3d540f62a0 2024-02-20T00:19:15.197 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:15.194+0000 7f3d517fa700 1 -- 172.21.15.132:0/2563598025 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3d480263a0 con 0x7f3d540f62a0 2024-02-20T00:19:15.197 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:15.194+0000 7f3d517fa700 1 --2- 172.21.15.132:0/2563598025 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f3d4402ff60 0x7f3d44032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:19:15.197 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:15.194+0000 7f3d517fa700 1 -- 172.21.15.132:0/2563598025 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3d48040f30 con 0x7f3d540f62a0 2024-02-20T00:19:15.197 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:15.194+0000 7f3d59bf6700 1 -- 172.21.15.132:0/2563598025 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d54059100 con 0x7f3d540f62a0 2024-02-20T00:19:15.197 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:15.194+0000 7f3d53fff700 1 --2- 172.21.15.132:0/2563598025 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f3d4402ff60 0x7f3d44032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:19:15.197 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:15.195+0000 7f3d53fff700 1 --2- 172.21.15.132:0/2563598025 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f3d4402ff60 0x7f3d44032420 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7f3d400096a0 tx=0x7f3d40006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:19:15.200 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:15.199+0000 7f3d517fa700 1 -- 172.21.15.132:0/2563598025 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3d48012920 con 0x7f3d540f62a0 2024-02-20T00:19:15.405 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:15.404+0000 7f3d59bf6700 1 -- 172.21.15.132:0/2563598025 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3d5406aab0 con 0x7f3d540f62a0 2024-02-20T00:19:15.406 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:15.405+0000 7f3d517fa700 1 -- 172.21.15.132:0/2563598025 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3d48012330 con 0x7f3d540f62a0 2024-02-20T00:19:15.406 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:19:15.406 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:19:15.408 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:15.406+0000 7f3d59bf6700 1 -- 172.21.15.132:0/2563598025 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f3d4402ff60 msgr2=0x7f3d44032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:19:15.408 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:15.406+0000 7f3d59bf6700 1 --2- 172.21.15.132:0/2563598025 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f3d4402ff60 0x7f3d44032420 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7f3d400096a0 tx=0x7f3d40006b40).stop 2024-02-20T00:19:15.408 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:15.406+0000 7f3d59bf6700 1 -- 172.21.15.132:0/2563598025 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3d540f62a0 msgr2=0x7f3d5406b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:19:15.408 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:15.406+0000 7f3d59bf6700 1 --2- 172.21.15.132:0/2563598025 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3d540f62a0 0x7f3d5406b9a0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0x7f3d480089f0 tx=0x7f3d4800b040).stop 2024-02-20T00:19:15.408 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:15.406+0000 7f3d59bf6700 1 -- 172.21.15.132:0/2563598025 shutdown_connections 2024-02-20T00:19:15.408 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:15.406+0000 7f3d59bf6700 1 --2- 172.21.15.132:0/2563598025 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f3d4402ff60 0x7f3d44032420 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:19:15.408 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:15.406+0000 7f3d59bf6700 1 --2- 172.21.15.132:0/2563598025 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3d540f62a0 0x7f3d5406b9a0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:19:15.408 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:15.406+0000 7f3d59bf6700 1 -- 172.21.15.132:0/2563598025 >> 172.21.15.132:0/2563598025 conn(0x7f3d540f1aa0 msgr2=0x7f3d540f2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:19:15.408 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:15.406+0000 7f3d59bf6700 1 -- 172.21.15.132:0/2563598025 shutdown_connections 2024-02-20T00:19:15.408 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:15.406+0000 7f3d59bf6700 1 -- 172.21.15.132:0/2563598025 wait complete. 2024-02-20T00:19:15.409 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:19:16.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:19:15 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:19:14.924239+0000 mgr.smithi084.bmqrnf (mgr.14162) 287 : cluster [DBG] pgmap v273: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:19:16.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:19:15 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:19:15.405448+0000 mon.smithi084 (mon.0) 283 : audit [DBG] from='client.? 172.21.15.132:0/2563598025' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:19:17.042 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:19:17.043 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:19:18.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:19:17 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:19:16.924602+0000 mgr.smithi084.bmqrnf (mgr.14162) 288 : cluster [DBG] pgmap v274: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:19:19.005 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:19.002+0000 7f6b2f59e700 1 -- 172.21.15.132:0/1921852804 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6b300f51a0 msgr2=0x7f6b300f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:19:19.005 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:19.002+0000 7f6b2f59e700 1 --2- 172.21.15.132:0/1921852804 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6b300f51a0 0x7f6b300f55c0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 rx=0x7f6b20009fb0 tx=0x7f6b20009b30).stop 2024-02-20T00:19:19.005 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:19.002+0000 7f6b2f59e700 1 -- 172.21.15.132:0/1921852804 shutdown_connections 2024-02-20T00:19:19.005 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:19.002+0000 7f6b2f59e700 1 --2- 172.21.15.132:0/1921852804 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6b300f51a0 0x7f6b300f55c0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:19:19.005 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:19.002+0000 7f6b2f59e700 1 -- 172.21.15.132:0/1921852804 >> 172.21.15.132:0/1921852804 conn(0x7f6b300f09a0 msgr2=0x7f6b300f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:19:19.006 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:19.002+0000 7f6b2f59e700 1 -- 172.21.15.132:0/1921852804 shutdown_connections 2024-02-20T00:19:19.006 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:19.002+0000 7f6b2f59e700 1 -- 172.21.15.132:0/1921852804 wait complete. 2024-02-20T00:19:19.006 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:19.002+0000 7f6b2f59e700 1 Processor -- start 2024-02-20T00:19:19.006 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:19.002+0000 7f6b2f59e700 1 -- start start 2024-02-20T00:19:19.006 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:19.003+0000 7f6b2f59e700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6b300f51a0 0x7f6b30106600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:19:19.006 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:19.003+0000 7f6b2f59e700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b30106af0 con 0x7f6b300f51a0 2024-02-20T00:19:19.006 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:19.003+0000 7f6b2e59c700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6b300f51a0 0x7f6b30106600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:19:19.006 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:19.003+0000 7f6b2e59c700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6b300f51a0 0x7f6b30106600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:56488/0 (socket says 172.21.15.132:56488) 2024-02-20T00:19:19.006 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:19.003+0000 7f6b2e59c700 1 -- 172.21.15.132:0/1287144872 learned_addr learned my addr 172.21.15.132:0/1287144872 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:19:19.006 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:19.003+0000 7f6b2e59c700 1 -- 172.21.15.132:0/1287144872 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b20005040 con 0x7f6b300f51a0 2024-02-20T00:19:19.006 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:19.003+0000 7f6b2e59c700 1 --2- 172.21.15.132:0/1287144872 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6b300f51a0 0x7f6b30106600 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7f6b200089f0 tx=0x7f6b2000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:19:19.006 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:19.003+0000 7f6b1effd700 1 -- 172.21.15.132:0/1287144872 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6b20009320 con 0x7f6b300f51a0 2024-02-20T00:19:19.007 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:19.003+0000 7f6b2f59e700 1 -- 172.21.15.132:0/1287144872 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b30106cf0 con 0x7f6b300f51a0 2024-02-20T00:19:19.007 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:19.003+0000 7f6b1effd700 1 -- 172.21.15.132:0/1287144872 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6b20013380 con 0x7f6b300f51a0 2024-02-20T00:19:19.007 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:19.003+0000 7f6b1effd700 1 -- 172.21.15.132:0/1287144872 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6b20014e20 con 0x7f6b300f51a0 2024-02-20T00:19:19.007 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:19.004+0000 7f6b2f59e700 1 -- 172.21.15.132:0/1287144872 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b30107180 con 0x7f6b300f51a0 2024-02-20T00:19:19.008 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:19.004+0000 7f6b1effd700 1 -- 172.21.15.132:0/1287144872 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6b200263a0 con 0x7f6b300f51a0 2024-02-20T00:19:19.008 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:19.004+0000 7f6b1effd700 1 --2- 172.21.15.132:0/1287144872 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f6b1802ff10 0x7f6b180323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:19:19.008 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:19.004+0000 7f6b2f59e700 1 -- 172.21.15.132:0/1287144872 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b30045ad0 con 0x7f6b300f51a0 2024-02-20T00:19:19.008 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:19.004+0000 7f6b1effd700 1 -- 172.21.15.132:0/1287144872 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6b20041210 con 0x7f6b300f51a0 2024-02-20T00:19:19.008 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:19.005+0000 7f6b2dd9b700 1 --2- 172.21.15.132:0/1287144872 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f6b1802ff10 0x7f6b180323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:19:19.008 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:19.005+0000 7f6b2dd9b700 1 --2- 172.21.15.132:0/1287144872 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f6b1802ff10 0x7f6b180323d0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f6b240096a0 tx=0x7f6b24006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:19:19.010 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:19.009+0000 7f6b1effd700 1 -- 172.21.15.132:0/1287144872 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6b20012920 con 0x7f6b300f51a0 2024-02-20T00:19:19.214 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:19.213+0000 7f6b2f59e700 1 -- 172.21.15.132:0/1287144872 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6b30108030 con 0x7f6b300f51a0 2024-02-20T00:19:19.214 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:19.213+0000 7f6b1effd700 1 -- 172.21.15.132:0/1287144872 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6b200126f0 con 0x7f6b300f51a0 2024-02-20T00:19:19.215 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:19:19.215 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:19:19.216 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:19.215+0000 7f6b2f59e700 1 -- 172.21.15.132:0/1287144872 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f6b1802ff10 msgr2=0x7f6b180323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:19:19.216 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:19.215+0000 7f6b2f59e700 1 --2- 172.21.15.132:0/1287144872 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f6b1802ff10 0x7f6b180323d0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f6b240096a0 tx=0x7f6b24006b40).stop 2024-02-20T00:19:19.216 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:19.215+0000 7f6b2f59e700 1 -- 172.21.15.132:0/1287144872 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6b300f51a0 msgr2=0x7f6b30106600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:19:19.216 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:19.215+0000 7f6b2f59e700 1 --2- 172.21.15.132:0/1287144872 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6b300f51a0 0x7f6b30106600 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7f6b200089f0 tx=0x7f6b2000b040).stop 2024-02-20T00:19:19.217 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:19.215+0000 7f6b2f59e700 1 -- 172.21.15.132:0/1287144872 shutdown_connections 2024-02-20T00:19:19.217 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:19.215+0000 7f6b2f59e700 1 --2- 172.21.15.132:0/1287144872 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f6b1802ff10 0x7f6b180323d0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:19:19.217 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:19.215+0000 7f6b2f59e700 1 --2- 172.21.15.132:0/1287144872 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6b300f51a0 0x7f6b30106600 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:19:19.217 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:19.215+0000 7f6b2f59e700 1 -- 172.21.15.132:0/1287144872 >> 172.21.15.132:0/1287144872 conn(0x7f6b300f09a0 msgr2=0x7f6b300f13e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:19:19.217 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:19.215+0000 7f6b2f59e700 1 -- 172.21.15.132:0/1287144872 shutdown_connections 2024-02-20T00:19:19.217 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:19.215+0000 7f6b2f59e700 1 -- 172.21.15.132:0/1287144872 wait complete. 2024-02-20T00:19:19.217 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:19:20.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:19:19 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:19:18.924936+0000 mgr.smithi084.bmqrnf (mgr.14162) 289 : cluster [DBG] pgmap v275: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:19:20.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:19:19 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:19:19.213876+0000 mon.smithi084 (mon.0) 284 : audit [DBG] from='client.? 172.21.15.132:0/1287144872' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:19:20.788 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:19:20.789 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:19:22.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:19:21 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:19:20.925306+0000 mgr.smithi084.bmqrnf (mgr.14162) 290 : cluster [DBG] pgmap v276: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:19:22.643 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:22.641+0000 7fb0c5bd0700 1 -- 172.21.15.132:0/1661133720 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb0c00f51a0 msgr2=0x7fb0c00f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:19:22.643 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:22.641+0000 7fb0c5bd0700 1 --2- 172.21.15.132:0/1661133720 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb0c00f51a0 0x7fb0c00f55c0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 rx=0x7fb0b4009fb0 tx=0x7fb0b4009b30).stop 2024-02-20T00:19:22.643 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:22.641+0000 7fb0c5bd0700 1 -- 172.21.15.132:0/1661133720 shutdown_connections 2024-02-20T00:19:22.643 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:22.641+0000 7fb0c5bd0700 1 --2- 172.21.15.132:0/1661133720 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb0c00f51a0 0x7fb0c00f55c0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:19:22.643 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:22.641+0000 7fb0c5bd0700 1 -- 172.21.15.132:0/1661133720 >> 172.21.15.132:0/1661133720 conn(0x7fb0c00f09a0 msgr2=0x7fb0c00f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:19:22.643 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:22.641+0000 7fb0c5bd0700 1 -- 172.21.15.132:0/1661133720 shutdown_connections 2024-02-20T00:19:22.643 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:22.641+0000 7fb0c5bd0700 1 -- 172.21.15.132:0/1661133720 wait complete. 2024-02-20T00:19:22.643 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:22.641+0000 7fb0c5bd0700 1 Processor -- start 2024-02-20T00:19:22.644 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:22.641+0000 7fb0c5bd0700 1 -- start start 2024-02-20T00:19:22.645 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:22.642+0000 7fb0c5bd0700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb0c00f51a0 0x7fb0c01065d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:19:22.645 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:22.642+0000 7fb0c5bd0700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0c0106ac0 con 0x7fb0c00f51a0 2024-02-20T00:19:22.645 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:22.642+0000 7fb0c4bce700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb0c00f51a0 0x7fb0c01065d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:19:22.645 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:22.642+0000 7fb0c4bce700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb0c00f51a0 0x7fb0c01065d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:56510/0 (socket says 172.21.15.132:56510) 2024-02-20T00:19:22.645 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:22.642+0000 7fb0c4bce700 1 -- 172.21.15.132:0/329812452 learned_addr learned my addr 172.21.15.132:0/329812452 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:19:22.646 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:22.642+0000 7fb0c4bce700 1 -- 172.21.15.132:0/329812452 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0b4005040 con 0x7fb0c00f51a0 2024-02-20T00:19:22.646 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:22.643+0000 7fb0c4bce700 1 --2- 172.21.15.132:0/329812452 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb0c00f51a0 0x7fb0c01065d0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0x7fb0b40089f0 tx=0x7fb0b400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:19:22.646 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:22.643+0000 7fb0bd7fa700 1 -- 172.21.15.132:0/329812452 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb0b4009320 con 0x7fb0c00f51a0 2024-02-20T00:19:22.646 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:22.643+0000 7fb0c5bd0700 1 -- 172.21.15.132:0/329812452 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0c0106cc0 con 0x7fb0c00f51a0 2024-02-20T00:19:22.646 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:22.643+0000 7fb0bd7fa700 1 -- 172.21.15.132:0/329812452 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb0b4013380 con 0x7fb0c00f51a0 2024-02-20T00:19:22.646 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:22.643+0000 7fb0bd7fa700 1 -- 172.21.15.132:0/329812452 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb0b4014e20 con 0x7fb0c00f51a0 2024-02-20T00:19:22.646 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:22.643+0000 7fb0c5bd0700 1 -- 172.21.15.132:0/329812452 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0c0107150 con 0x7fb0c00f51a0 2024-02-20T00:19:22.646 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:22.644+0000 7fb0bd7fa700 1 -- 172.21.15.132:0/329812452 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb0b4014420 con 0x7fb0c00f51a0 2024-02-20T00:19:22.646 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:22.644+0000 7fb0c5bd0700 1 -- 172.21.15.132:0/329812452 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0c0045ad0 con 0x7fb0c00f51a0 2024-02-20T00:19:22.646 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:22.644+0000 7fb0bd7fa700 1 --2- 172.21.15.132:0/329812452 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fb0b002ff60 0x7fb0b0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:19:22.646 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:22.644+0000 7fb0bd7fa700 1 -- 172.21.15.132:0/329812452 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb0b4040fc0 con 0x7fb0c00f51a0 2024-02-20T00:19:22.647 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:22.644+0000 7fb0bffff700 1 --2- 172.21.15.132:0/329812452 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fb0b002ff60 0x7fb0b0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:19:22.647 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:22.645+0000 7fb0bffff700 1 --2- 172.21.15.132:0/329812452 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fb0b002ff60 0x7fb0b0032420 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7fb0ac0096a0 tx=0x7fb0ac006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:19:22.650 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:22.649+0000 7fb0bd7fa700 1 -- 172.21.15.132:0/329812452 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb0b40095f0 con 0x7fb0c00f51a0 2024-02-20T00:19:22.854 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:22.853+0000 7fb0c5bd0700 1 -- 172.21.15.132:0/329812452 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb0c00fad30 con 0x7fb0c00f51a0 2024-02-20T00:19:22.855 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:22.853+0000 7fb0bd7fa700 1 -- 172.21.15.132:0/329812452 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb0b401db70 con 0x7fb0c00f51a0 2024-02-20T00:19:22.855 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:19:22.855 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:19:22.857 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:22.855+0000 7fb0c5bd0700 1 -- 172.21.15.132:0/329812452 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fb0b002ff60 msgr2=0x7fb0b0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:19:22.857 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:22.855+0000 7fb0c5bd0700 1 --2- 172.21.15.132:0/329812452 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fb0b002ff60 0x7fb0b0032420 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7fb0ac0096a0 tx=0x7fb0ac006b40).stop 2024-02-20T00:19:22.857 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:22.855+0000 7fb0c5bd0700 1 -- 172.21.15.132:0/329812452 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb0c00f51a0 msgr2=0x7fb0c01065d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:19:22.857 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:22.855+0000 7fb0c5bd0700 1 --2- 172.21.15.132:0/329812452 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb0c00f51a0 0x7fb0c01065d0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0x7fb0b40089f0 tx=0x7fb0b400b040).stop 2024-02-20T00:19:22.857 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:22.855+0000 7fb0c5bd0700 1 -- 172.21.15.132:0/329812452 shutdown_connections 2024-02-20T00:19:22.857 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:22.855+0000 7fb0c5bd0700 1 --2- 172.21.15.132:0/329812452 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fb0b002ff60 0x7fb0b0032420 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:19:22.857 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:22.855+0000 7fb0c5bd0700 1 --2- 172.21.15.132:0/329812452 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb0c00f51a0 0x7fb0c01065d0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:19:22.857 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:22.855+0000 7fb0c5bd0700 1 -- 172.21.15.132:0/329812452 >> 172.21.15.132:0/329812452 conn(0x7fb0c00f09a0 msgr2=0x7fb0c005fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:19:22.858 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:22.855+0000 7fb0c5bd0700 1 -- 172.21.15.132:0/329812452 shutdown_connections 2024-02-20T00:19:22.858 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:22.855+0000 7fb0c5bd0700 1 -- 172.21.15.132:0/329812452 wait complete. 2024-02-20T00:19:22.858 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:19:23.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:19:22 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:19:22.854242+0000 mon.smithi084 (mon.0) 285 : audit [DBG] from='client.? 172.21.15.132:0/329812452' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:19:24.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:19:23 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:19:22.925613+0000 mgr.smithi084.bmqrnf (mgr.14162) 291 : cluster [DBG] pgmap v277: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:19:24.591 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:19:24.592 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:19:26.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:19:25 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:19:24.925863+0000 mgr.smithi084.bmqrnf (mgr.14162) 292 : cluster [DBG] pgmap v278: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:19:26.668 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:26.627+0000 7fdfba29e700 1 -- 172.21.15.132:0/3522558684 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdfb4060280 msgr2=0x7fdfb40f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:19:26.668 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:26.627+0000 7fdfba29e700 1 --2- 172.21.15.132:0/3522558684 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdfb4060280 0x7fdfb40f90f0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 rx=0x7fdfa4009fb0 tx=0x7fdfa4009b30).stop 2024-02-20T00:19:26.668 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:26.627+0000 7fdfba29e700 1 -- 172.21.15.132:0/3522558684 shutdown_connections 2024-02-20T00:19:26.668 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:26.627+0000 7fdfba29e700 1 --2- 172.21.15.132:0/3522558684 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdfb4060280 0x7fdfb40f90f0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:19:26.668 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:26.627+0000 7fdfba29e700 1 -- 172.21.15.132:0/3522558684 >> 172.21.15.132:0/3522558684 conn(0x7fdfb40f0b70 msgr2=0x7fdfb40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:19:26.669 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:26.627+0000 7fdfba29e700 1 -- 172.21.15.132:0/3522558684 shutdown_connections 2024-02-20T00:19:26.669 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:26.627+0000 7fdfba29e700 1 -- 172.21.15.132:0/3522558684 wait complete. 2024-02-20T00:19:26.669 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:26.628+0000 7fdfba29e700 1 Processor -- start 2024-02-20T00:19:26.669 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:26.628+0000 7fdfba29e700 1 -- start start 2024-02-20T00:19:26.669 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:26.628+0000 7fdfba29e700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdfb4060280 0x7fdfb4106780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:19:26.669 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:26.628+0000 7fdfba29e700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfb4106c70 con 0x7fdfb4060280 2024-02-20T00:19:26.669 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:26.628+0000 7fdfb3fff700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdfb4060280 0x7fdfb4106780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:19:26.669 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:26.628+0000 7fdfb3fff700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdfb4060280 0x7fdfb4106780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:56528/0 (socket says 172.21.15.132:56528) 2024-02-20T00:19:26.669 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:26.628+0000 7fdfb3fff700 1 -- 172.21.15.132:0/2282933372 learned_addr learned my addr 172.21.15.132:0/2282933372 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:19:26.669 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:26.629+0000 7fdfb3fff700 1 -- 172.21.15.132:0/2282933372 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfa4005040 con 0x7fdfb4060280 2024-02-20T00:19:26.669 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:26.629+0000 7fdfb3fff700 1 --2- 172.21.15.132:0/2282933372 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdfb4060280 0x7fdfb4106780 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7fdfa40089f0 tx=0x7fdfa400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:19:26.669 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:26.629+0000 7fdfb0ff9700 1 -- 172.21.15.132:0/2282933372 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdfa4009320 con 0x7fdfb4060280 2024-02-20T00:19:26.669 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:26.629+0000 7fdfba29e700 1 -- 172.21.15.132:0/2282933372 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfb4106e70 con 0x7fdfb4060280 2024-02-20T00:19:26.670 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:26.629+0000 7fdfb0ff9700 1 -- 172.21.15.132:0/2282933372 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdfa4013380 con 0x7fdfb4060280 2024-02-20T00:19:26.670 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:26.629+0000 7fdfb0ff9700 1 -- 172.21.15.132:0/2282933372 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdfa4014e20 con 0x7fdfb4060280 2024-02-20T00:19:26.670 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:26.629+0000 7fdfba29e700 1 -- 172.21.15.132:0/2282933372 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfb4107300 con 0x7fdfb4060280 2024-02-20T00:19:26.670 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:26.630+0000 7fdfba29e700 1 -- 172.21.15.132:0/2282933372 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfb4045ad0 con 0x7fdfb4060280 2024-02-20T00:19:26.670 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:26.630+0000 7fdfb0ff9700 1 -- 172.21.15.132:0/2282933372 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdfa40095f0 con 0x7fdfb4060280 2024-02-20T00:19:26.670 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:26.630+0000 7fdfb0ff9700 1 --2- 172.21.15.132:0/2282933372 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fdfa002ff10 0x7fdfa00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:19:26.670 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:26.630+0000 7fdfb0ff9700 1 -- 172.21.15.132:0/2282933372 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdfa4040fb0 con 0x7fdfb4060280 2024-02-20T00:19:26.670 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:26.630+0000 7fdfb37fe700 1 --2- 172.21.15.132:0/2282933372 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fdfa002ff10 0x7fdfa00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:19:26.670 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:26.631+0000 7fdfb37fe700 1 --2- 172.21.15.132:0/2282933372 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fdfa002ff10 0x7fdfa00323d0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7fdf9c0096a0 tx=0x7fdf9c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:19:26.670 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:26.635+0000 7fdfb0ff9700 1 -- 172.21.15.132:0/2282933372 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdfa406d2b0 con 0x7fdfb4060280 2024-02-20T00:19:26.845 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:26.842+0000 7fdfba29e700 1 -- 172.21.15.132:0/2282933372 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdfb4108150 con 0x7fdfb4060280 2024-02-20T00:19:26.845 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:26.843+0000 7fdfb0ff9700 1 -- 172.21.15.132:0/2282933372 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fdfa4014420 con 0x7fdfb4060280 2024-02-20T00:19:26.846 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:19:26.846 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:19:26.847 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:26.845+0000 7fdfba29e700 1 -- 172.21.15.132:0/2282933372 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fdfa002ff10 msgr2=0x7fdfa00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:19:26.847 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:26.845+0000 7fdfba29e700 1 --2- 172.21.15.132:0/2282933372 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fdfa002ff10 0x7fdfa00323d0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7fdf9c0096a0 tx=0x7fdf9c006b40).stop 2024-02-20T00:19:26.847 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:26.845+0000 7fdfba29e700 1 -- 172.21.15.132:0/2282933372 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdfb4060280 msgr2=0x7fdfb4106780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:19:26.847 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:26.845+0000 7fdfba29e700 1 --2- 172.21.15.132:0/2282933372 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdfb4060280 0x7fdfb4106780 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7fdfa40089f0 tx=0x7fdfa400b040).stop 2024-02-20T00:19:26.847 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:26.845+0000 7fdfba29e700 1 -- 172.21.15.132:0/2282933372 shutdown_connections 2024-02-20T00:19:26.847 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:26.845+0000 7fdfba29e700 1 --2- 172.21.15.132:0/2282933372 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fdfa002ff10 0x7fdfa00323d0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:19:26.847 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:26.845+0000 7fdfba29e700 1 --2- 172.21.15.132:0/2282933372 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdfb4060280 0x7fdfb4106780 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:19:26.848 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:26.845+0000 7fdfba29e700 1 -- 172.21.15.132:0/2282933372 >> 172.21.15.132:0/2282933372 conn(0x7fdfb40f0b70 msgr2=0x7fdfb40f15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:19:26.848 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:26.845+0000 7fdfba29e700 1 -- 172.21.15.132:0/2282933372 shutdown_connections 2024-02-20T00:19:26.848 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:26.845+0000 7fdfba29e700 1 -- 172.21.15.132:0/2282933372 wait complete. 2024-02-20T00:19:26.848 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:19:27.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:19:26 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:19:26.844720+0000 mon.smithi084 (mon.0) 286 : audit [DBG] from='client.? 172.21.15.132:0/2282933372' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:19:28.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:19:27 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:19:26.926208+0000 mgr.smithi084.bmqrnf (mgr.14162) 293 : cluster [DBG] pgmap v279: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:19:28.537 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:19:28.538 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:19:30.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:19:29 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:19:28.926580+0000 mgr.smithi084.bmqrnf (mgr.14162) 294 : cluster [DBG] pgmap v280: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:19:30.561 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:30.554+0000 7ff746558700 1 -- 172.21.15.132:0/1805705938 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff740060220 msgr2=0x7ff7400f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:19:30.561 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:30.554+0000 7ff746558700 1 --2- 172.21.15.132:0/1805705938 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff740060220 0x7ff7400f9150 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 rx=0x7ff730009fb0 tx=0x7ff730009b30).stop 2024-02-20T00:19:30.562 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:30.554+0000 7ff746558700 1 -- 172.21.15.132:0/1805705938 shutdown_connections 2024-02-20T00:19:30.562 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:30.554+0000 7ff746558700 1 --2- 172.21.15.132:0/1805705938 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff740060220 0x7ff7400f9150 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:19:30.562 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:30.554+0000 7ff746558700 1 -- 172.21.15.132:0/1805705938 >> 172.21.15.132:0/1805705938 conn(0x7ff7400f0b70 msgr2=0x7ff7400f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:19:30.562 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:30.554+0000 7ff746558700 1 -- 172.21.15.132:0/1805705938 shutdown_connections 2024-02-20T00:19:30.562 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:30.554+0000 7ff746558700 1 -- 172.21.15.132:0/1805705938 wait complete. 2024-02-20T00:19:30.563 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:30.555+0000 7ff746558700 1 Processor -- start 2024-02-20T00:19:30.563 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:30.555+0000 7ff746558700 1 -- start start 2024-02-20T00:19:30.563 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:30.555+0000 7ff746558700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff740060220 0x7ff7400f6bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:19:30.563 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:30.555+0000 7ff746558700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7400f5050 con 0x7ff740060220 2024-02-20T00:19:30.563 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:30.555+0000 7ff73ffff700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff740060220 0x7ff7400f6bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:19:30.563 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:30.555+0000 7ff73ffff700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff740060220 0x7ff7400f6bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:50466/0 (socket says 172.21.15.132:50466) 2024-02-20T00:19:30.564 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:30.555+0000 7ff73ffff700 1 -- 172.21.15.132:0/3876548859 learned_addr learned my addr 172.21.15.132:0/3876548859 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:19:30.564 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:30.556+0000 7ff73ffff700 1 -- 172.21.15.132:0/3876548859 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff730005040 con 0x7ff740060220 2024-02-20T00:19:30.564 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:30.556+0000 7ff73ffff700 1 --2- 172.21.15.132:0/3876548859 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff740060220 0x7ff7400f6bd0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0x7ff7300089f0 tx=0x7ff73000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:19:30.564 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:30.556+0000 7ff73cff9700 1 -- 172.21.15.132:0/3876548859 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff730009320 con 0x7ff740060220 2024-02-20T00:19:30.564 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:30.556+0000 7ff73cff9700 1 -- 172.21.15.132:0/3876548859 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff730013380 con 0x7ff740060220 2024-02-20T00:19:30.565 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:30.556+0000 7ff746558700 1 -- 172.21.15.132:0/3876548859 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7400f51f0 con 0x7ff740060220 2024-02-20T00:19:30.565 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:30.556+0000 7ff73cff9700 1 -- 172.21.15.132:0/3876548859 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff730014e20 con 0x7ff740060220 2024-02-20T00:19:30.565 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:30.556+0000 7ff746558700 1 -- 172.21.15.132:0/3876548859 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7400f5680 con 0x7ff740060220 2024-02-20T00:19:30.565 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:30.557+0000 7ff73cff9700 1 -- 172.21.15.132:0/3876548859 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff730014420 con 0x7ff740060220 2024-02-20T00:19:30.565 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:30.557+0000 7ff746558700 1 -- 172.21.15.132:0/3876548859 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff740059100 con 0x7ff740060220 2024-02-20T00:19:30.565 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:30.557+0000 7ff73cff9700 1 --2- 172.21.15.132:0/3876548859 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7ff72c02ff10 0x7ff72c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:19:30.566 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:30.557+0000 7ff73cff9700 1 -- 172.21.15.132:0/3876548859 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff730040f80 con 0x7ff740060220 2024-02-20T00:19:30.566 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:30.557+0000 7ff73f7fe700 1 --2- 172.21.15.132:0/3876548859 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7ff72c02ff10 0x7ff72c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:19:30.566 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:30.558+0000 7ff73f7fe700 1 --2- 172.21.15.132:0/3876548859 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7ff72c02ff10 0x7ff72c0323d0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7ff7280096a0 tx=0x7ff728006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:19:30.566 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:30.562+0000 7ff73cff9700 1 -- 172.21.15.132:0/3876548859 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff73006d2b0 con 0x7ff740060220 2024-02-20T00:19:30.769 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:30.767+0000 7ff746558700 1 -- 172.21.15.132:0/3876548859 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff7400fc090 con 0x7ff740060220 2024-02-20T00:19:30.770 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:30.768+0000 7ff73cff9700 1 -- 172.21.15.132:0/3876548859 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff7300095f0 con 0x7ff740060220 2024-02-20T00:19:30.770 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:19:30.770 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:19:30.772 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:30.769+0000 7ff746558700 1 -- 172.21.15.132:0/3876548859 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7ff72c02ff10 msgr2=0x7ff72c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:19:30.772 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:30.769+0000 7ff746558700 1 --2- 172.21.15.132:0/3876548859 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7ff72c02ff10 0x7ff72c0323d0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7ff7280096a0 tx=0x7ff728006b40).stop 2024-02-20T00:19:30.772 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:30.769+0000 7ff746558700 1 -- 172.21.15.132:0/3876548859 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff740060220 msgr2=0x7ff7400f6bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:19:30.772 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:30.769+0000 7ff746558700 1 --2- 172.21.15.132:0/3876548859 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff740060220 0x7ff7400f6bd0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0x7ff7300089f0 tx=0x7ff73000b040).stop 2024-02-20T00:19:30.772 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:30.769+0000 7ff746558700 1 -- 172.21.15.132:0/3876548859 shutdown_connections 2024-02-20T00:19:30.772 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:30.769+0000 7ff746558700 1 --2- 172.21.15.132:0/3876548859 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7ff72c02ff10 0x7ff72c0323d0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:19:30.772 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:30.769+0000 7ff746558700 1 --2- 172.21.15.132:0/3876548859 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff740060220 0x7ff7400f6bd0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:19:30.772 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:30.769+0000 7ff746558700 1 -- 172.21.15.132:0/3876548859 >> 172.21.15.132:0/3876548859 conn(0x7ff7400f0b70 msgr2=0x7ff7400f1620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:19:30.772 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:30.770+0000 7ff746558700 1 -- 172.21.15.132:0/3876548859 shutdown_connections 2024-02-20T00:19:30.772 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:30.770+0000 7ff746558700 1 -- 172.21.15.132:0/3876548859 wait complete. 2024-02-20T00:19:30.773 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:19:31.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:19:30 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:19:30.769430+0000 mon.smithi084 (mon.0) 287 : audit [DBG] from='client.? 172.21.15.132:0/3876548859' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:19:32.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:19:31 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:19:30.926941+0000 mgr.smithi084.bmqrnf (mgr.14162) 295 : cluster [DBG] pgmap v281: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:19:32.507 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:19:32.508 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:19:34.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:19:33 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:19:32.927310+0000 mgr.smithi084.bmqrnf (mgr.14162) 296 : cluster [DBG] pgmap v282: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:19:34.471 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:34.467+0000 7f8692249700 1 -- 172.21.15.132:0/1888019338 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f868c0f51a0 msgr2=0x7f868c0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:19:34.471 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:34.467+0000 7f8692249700 1 --2- 172.21.15.132:0/1888019338 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f868c0f51a0 0x7f868c0f55c0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 rx=0x7f867c009fb0 tx=0x7f867c009b30).stop 2024-02-20T00:19:34.471 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:34.467+0000 7f8692249700 1 -- 172.21.15.132:0/1888019338 shutdown_connections 2024-02-20T00:19:34.471 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:34.467+0000 7f8692249700 1 --2- 172.21.15.132:0/1888019338 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f868c0f51a0 0x7f868c0f55c0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:19:34.471 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:34.467+0000 7f8692249700 1 -- 172.21.15.132:0/1888019338 >> 172.21.15.132:0/1888019338 conn(0x7f868c0f09a0 msgr2=0x7f868c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:19:34.471 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:34.467+0000 7f8692249700 1 -- 172.21.15.132:0/1888019338 shutdown_connections 2024-02-20T00:19:34.471 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:34.467+0000 7f8692249700 1 -- 172.21.15.132:0/1888019338 wait complete. 2024-02-20T00:19:34.471 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:34.468+0000 7f8692249700 1 Processor -- start 2024-02-20T00:19:34.471 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:34.468+0000 7f8692249700 1 -- start start 2024-02-20T00:19:34.471 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:34.468+0000 7f8692249700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f868c0f51a0 0x7f868c0f2f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:19:34.471 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:34.468+0000 7f8692249700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f868c0f13a0 con 0x7f868c0f51a0 2024-02-20T00:19:34.471 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:34.468+0000 7f8691247700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f868c0f51a0 0x7f868c0f2f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:19:34.472 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:34.468+0000 7f8691247700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f868c0f51a0 0x7f868c0f2f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:50498/0 (socket says 172.21.15.132:50498) 2024-02-20T00:19:34.472 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:34.468+0000 7f8691247700 1 -- 172.21.15.132:0/2003875778 learned_addr learned my addr 172.21.15.132:0/2003875778 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:19:34.473 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:34.469+0000 7f8691247700 1 -- 172.21.15.132:0/2003875778 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f867c005040 con 0x7f868c0f51a0 2024-02-20T00:19:34.473 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:34.469+0000 7f8691247700 1 --2- 172.21.15.132:0/2003875778 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f868c0f51a0 0x7f868c0f2f20 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0x7f867c0089f0 tx=0x7f867c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:19:34.473 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:34.469+0000 7f8681ffb700 1 -- 172.21.15.132:0/2003875778 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f867c009320 con 0x7f868c0f51a0 2024-02-20T00:19:34.474 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:34.469+0000 7f8681ffb700 1 -- 172.21.15.132:0/2003875778 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f867c013380 con 0x7f868c0f51a0 2024-02-20T00:19:34.474 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:34.469+0000 7f8692249700 1 -- 172.21.15.132:0/2003875778 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f868c0f1540 con 0x7f868c0f51a0 2024-02-20T00:19:34.474 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:34.469+0000 7f8681ffb700 1 -- 172.21.15.132:0/2003875778 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f867c014e20 con 0x7f868c0f51a0 2024-02-20T00:19:34.474 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:34.469+0000 7f8692249700 1 -- 172.21.15.132:0/2003875778 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f868c0f19d0 con 0x7f868c0f51a0 2024-02-20T00:19:34.474 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:34.470+0000 7f8681ffb700 1 -- 172.21.15.132:0/2003875778 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f867c0095f0 con 0x7f868c0f51a0 2024-02-20T00:19:34.474 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:34.470+0000 7f8692249700 1 -- 172.21.15.132:0/2003875778 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f868c059100 con 0x7f868c0f51a0 2024-02-20T00:19:34.474 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:34.470+0000 7f8681ffb700 1 --2- 172.21.15.132:0/2003875778 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f867802ff10 0x7f86780323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:19:34.474 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:34.470+0000 7f8681ffb700 1 -- 172.21.15.132:0/2003875778 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f867c040f50 con 0x7f868c0f51a0 2024-02-20T00:19:34.474 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:34.471+0000 7f8690a46700 1 --2- 172.21.15.132:0/2003875778 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f867802ff10 0x7f86780323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:19:34.474 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:34.471+0000 7f8690a46700 1 --2- 172.21.15.132:0/2003875778 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f867802ff10 0x7f86780323d0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f86880096a0 tx=0x7f8688006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:19:34.478 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:34.475+0000 7f8681ffb700 1 -- 172.21.15.132:0/2003875778 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f867c014420 con 0x7f868c0f51a0 2024-02-20T00:19:34.682 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:34.680+0000 7f8692249700 1 -- 172.21.15.132:0/2003875778 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f868c045ad0 con 0x7f868c0f51a0 2024-02-20T00:19:34.683 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:34.681+0000 7f8681ffb700 1 -- 172.21.15.132:0/2003875778 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f867c01d980 con 0x7f868c0f51a0 2024-02-20T00:19:34.683 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:19:34.683 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:19:34.685 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:34.682+0000 7f8692249700 1 -- 172.21.15.132:0/2003875778 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f867802ff10 msgr2=0x7f86780323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:19:34.685 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:34.682+0000 7f8692249700 1 --2- 172.21.15.132:0/2003875778 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f867802ff10 0x7f86780323d0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f86880096a0 tx=0x7f8688006b40).stop 2024-02-20T00:19:34.685 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:34.682+0000 7f8692249700 1 -- 172.21.15.132:0/2003875778 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f868c0f51a0 msgr2=0x7f868c0f2f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:19:34.685 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:34.682+0000 7f8692249700 1 --2- 172.21.15.132:0/2003875778 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f868c0f51a0 0x7f868c0f2f20 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0x7f867c0089f0 tx=0x7f867c00b040).stop 2024-02-20T00:19:34.685 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:34.682+0000 7f8692249700 1 -- 172.21.15.132:0/2003875778 shutdown_connections 2024-02-20T00:19:34.685 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:34.682+0000 7f8692249700 1 --2- 172.21.15.132:0/2003875778 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f867802ff10 0x7f86780323d0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:19:34.685 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:34.682+0000 7f8692249700 1 --2- 172.21.15.132:0/2003875778 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f868c0f51a0 0x7f868c0f2f20 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:19:34.685 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:34.682+0000 7f8692249700 1 -- 172.21.15.132:0/2003875778 >> 172.21.15.132:0/2003875778 conn(0x7f868c0f09a0 msgr2=0x7f868c05fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:19:34.685 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:34.682+0000 7f8692249700 1 -- 172.21.15.132:0/2003875778 shutdown_connections 2024-02-20T00:19:34.686 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:34.682+0000 7f8692249700 1 -- 172.21.15.132:0/2003875778 wait complete. 2024-02-20T00:19:34.686 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:19:35.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:19:34 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:19:34.682264+0000 mon.smithi084 (mon.0) 288 : audit [DBG] from='client.? 172.21.15.132:0/2003875778' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:19:36.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:19:35 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:19:34.927640+0000 mgr.smithi084.bmqrnf (mgr.14162) 297 : cluster [DBG] pgmap v283: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:19:36.400 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:19:36.400 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:19:38.283 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:38.279+0000 7f763acc8700 1 -- 172.21.15.132:0/3903174949 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f76340f51a0 msgr2=0x7f76340f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:19:38.283 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:38.279+0000 7f763acc8700 1 --2- 172.21.15.132:0/3903174949 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f76340f51a0 0x7f76340f55c0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 rx=0x7f7630009fb0 tx=0x7f7630009b30).stop 2024-02-20T00:19:38.283 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:38.279+0000 7f763acc8700 1 -- 172.21.15.132:0/3903174949 shutdown_connections 2024-02-20T00:19:38.283 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:38.279+0000 7f763acc8700 1 --2- 172.21.15.132:0/3903174949 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f76340f51a0 0x7f76340f55c0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:19:38.283 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:38.279+0000 7f763acc8700 1 -- 172.21.15.132:0/3903174949 >> 172.21.15.132:0/3903174949 conn(0x7f76340f09a0 msgr2=0x7f76340f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:19:38.283 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:38.279+0000 7f763acc8700 1 -- 172.21.15.132:0/3903174949 shutdown_connections 2024-02-20T00:19:38.283 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:38.279+0000 7f763acc8700 1 -- 172.21.15.132:0/3903174949 wait complete. 2024-02-20T00:19:38.283 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:38.279+0000 7f763acc8700 1 Processor -- start 2024-02-20T00:19:38.283 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:38.279+0000 7f763acc8700 1 -- start start 2024-02-20T00:19:38.284 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:38.279+0000 7f763acc8700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f76340f51a0 0x7f76341065d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:19:38.284 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:38.279+0000 7f763acc8700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7634106ac0 con 0x7f76340f51a0 2024-02-20T00:19:38.284 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:38.280+0000 7f7639cc6700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f76340f51a0 0x7f76341065d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:19:38.284 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:38.280+0000 7f7639cc6700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f76340f51a0 0x7f76341065d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:50884/0 (socket says 172.21.15.132:50884) 2024-02-20T00:19:38.284 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:38.280+0000 7f7639cc6700 1 -- 172.21.15.132:0/1885243270 learned_addr learned my addr 172.21.15.132:0/1885243270 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:19:38.284 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:38.280+0000 7f7639cc6700 1 -- 172.21.15.132:0/1885243270 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7630005040 con 0x7f76340f51a0 2024-02-20T00:19:38.284 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:38.280+0000 7f7639cc6700 1 --2- 172.21.15.132:0/1885243270 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f76340f51a0 0x7f76341065d0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0x7f76300089f0 tx=0x7f763000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:19:38.284 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:38.280+0000 7f762a7fc700 1 -- 172.21.15.132:0/1885243270 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7630009320 con 0x7f76340f51a0 2024-02-20T00:19:38.284 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:38.280+0000 7f763acc8700 1 -- 172.21.15.132:0/1885243270 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7634106cc0 con 0x7f76340f51a0 2024-02-20T00:19:38.284 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:38.280+0000 7f762a7fc700 1 -- 172.21.15.132:0/1885243270 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7630013380 con 0x7f76340f51a0 2024-02-20T00:19:38.284 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:38.280+0000 7f762a7fc700 1 -- 172.21.15.132:0/1885243270 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7630014e20 con 0x7f76340f51a0 2024-02-20T00:19:38.284 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:38.280+0000 7f763acc8700 1 -- 172.21.15.132:0/1885243270 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7634107150 con 0x7f76340f51a0 2024-02-20T00:19:38.285 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:38.281+0000 7f762a7fc700 1 -- 172.21.15.132:0/1885243270 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f76300095f0 con 0x7f76340f51a0 2024-02-20T00:19:38.285 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:38.281+0000 7f763acc8700 1 -- 172.21.15.132:0/1885243270 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7634045ad0 con 0x7f76340f51a0 2024-02-20T00:19:38.285 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:38.281+0000 7f762a7fc700 1 --2- 172.21.15.132:0/1885243270 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f762002ff60 0x7f7620032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:19:38.286 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:38.281+0000 7f762a7fc700 1 -- 172.21.15.132:0/1885243270 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7630040f00 con 0x7f76340f51a0 2024-02-20T00:19:38.286 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:38.281+0000 7f76394c5700 1 --2- 172.21.15.132:0/1885243270 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f762002ff60 0x7f7620032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:19:38.286 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:38.282+0000 7f76394c5700 1 --2- 172.21.15.132:0/1885243270 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f762002ff60 0x7f7620032420 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7f76240096a0 tx=0x7f7624006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:19:38.288 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:38.286+0000 7f762a7fc700 1 -- 172.21.15.132:0/1885243270 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7630014420 con 0x7f76340f51a0 2024-02-20T00:19:38.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:19:37 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:19:36.927975+0000 mgr.smithi084.bmqrnf (mgr.14162) 298 : cluster [DBG] pgmap v284: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:19:38.493 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:38.490+0000 7f763acc8700 1 -- 172.21.15.132:0/1885243270 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7634108000 con 0x7f76340f51a0 2024-02-20T00:19:38.493 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:38.491+0000 7f762a7fc700 1 -- 172.21.15.132:0/1885243270 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7630010230 con 0x7f76340f51a0 2024-02-20T00:19:38.494 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:19:38.494 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:19:38.495 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:38.493+0000 7f763acc8700 1 -- 172.21.15.132:0/1885243270 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f762002ff60 msgr2=0x7f7620032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:19:38.495 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:38.493+0000 7f763acc8700 1 --2- 172.21.15.132:0/1885243270 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f762002ff60 0x7f7620032420 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7f76240096a0 tx=0x7f7624006b40).stop 2024-02-20T00:19:38.496 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:38.493+0000 7f763acc8700 1 -- 172.21.15.132:0/1885243270 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f76340f51a0 msgr2=0x7f76341065d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:19:38.496 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:38.493+0000 7f763acc8700 1 --2- 172.21.15.132:0/1885243270 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f76340f51a0 0x7f76341065d0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0x7f76300089f0 tx=0x7f763000b040).stop 2024-02-20T00:19:38.496 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:38.493+0000 7f763acc8700 1 -- 172.21.15.132:0/1885243270 shutdown_connections 2024-02-20T00:19:38.496 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:38.493+0000 7f763acc8700 1 --2- 172.21.15.132:0/1885243270 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f762002ff60 0x7f7620032420 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:19:38.496 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:38.493+0000 7f763acc8700 1 --2- 172.21.15.132:0/1885243270 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f76340f51a0 0x7f76341065d0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:19:38.496 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:38.493+0000 7f763acc8700 1 -- 172.21.15.132:0/1885243270 >> 172.21.15.132:0/1885243270 conn(0x7f76340f09a0 msgr2=0x7f763405fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:19:38.496 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:38.493+0000 7f763acc8700 1 -- 172.21.15.132:0/1885243270 shutdown_connections 2024-02-20T00:19:38.496 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:38.493+0000 7f763acc8700 1 -- 172.21.15.132:0/1885243270 wait complete. 2024-02-20T00:19:38.496 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:19:39.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:19:38 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:19:38.492625+0000 mon.smithi084 (mon.0) 289 : audit [DBG] from='client.? 172.21.15.132:0/1885243270' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:19:40.130 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:19:40.130 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:19:40.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:19:39 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:19:38.928301+0000 mgr.smithi084.bmqrnf (mgr.14162) 299 : cluster [DBG] pgmap v285: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:19:42.030 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:42.012+0000 7ff81415a700 1 -- 172.21.15.132:0/1040710273 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff80c0f4740 msgr2=0x7ff80c0f4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:19:42.030 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:42.012+0000 7ff81415a700 1 --2- 172.21.15.132:0/1040710273 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff80c0f4740 0x7ff80c0f4b60 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 rx=0x7ff800009fb0 tx=0x7ff800009b30).stop 2024-02-20T00:19:42.031 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:42.012+0000 7ff81415a700 1 -- 172.21.15.132:0/1040710273 shutdown_connections 2024-02-20T00:19:42.031 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:42.012+0000 7ff81415a700 1 --2- 172.21.15.132:0/1040710273 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff80c0f4740 0x7ff80c0f4b60 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:19:42.031 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:42.012+0000 7ff81415a700 1 -- 172.21.15.132:0/1040710273 >> 172.21.15.132:0/1040710273 conn(0x7ff80c0f0260 msgr2=0x7ff80c0f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:19:42.031 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:42.012+0000 7ff81415a700 1 -- 172.21.15.132:0/1040710273 shutdown_connections 2024-02-20T00:19:42.031 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:42.012+0000 7ff81415a700 1 -- 172.21.15.132:0/1040710273 wait complete. 2024-02-20T00:19:42.031 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:42.013+0000 7ff81415a700 1 Processor -- start 2024-02-20T00:19:42.031 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:42.013+0000 7ff81415a700 1 -- start start 2024-02-20T00:19:42.031 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:42.013+0000 7ff81415a700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff80c0f4740 0x7ff80c1067b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:19:42.031 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:42.013+0000 7ff81415a700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff80c106ca0 con 0x7ff80c0f4740 2024-02-20T00:19:42.031 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:42.013+0000 7ff811ef6700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff80c0f4740 0x7ff80c1067b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:19:42.031 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:42.013+0000 7ff811ef6700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff80c0f4740 0x7ff80c1067b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:50908/0 (socket says 172.21.15.132:50908) 2024-02-20T00:19:42.031 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:42.013+0000 7ff811ef6700 1 -- 172.21.15.132:0/2832481112 learned_addr learned my addr 172.21.15.132:0/2832481112 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:19:42.031 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:42.013+0000 7ff811ef6700 1 -- 172.21.15.132:0/2832481112 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff800005040 con 0x7ff80c0f4740 2024-02-20T00:19:42.032 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:42.013+0000 7ff811ef6700 1 --2- 172.21.15.132:0/2832481112 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff80c0f4740 0x7ff80c1067b0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 rx=0x7ff800005010 tx=0x7ff80000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:19:42.032 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:42.014+0000 7ff7fe7fc700 1 -- 172.21.15.132:0/2832481112 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff800009320 con 0x7ff80c0f4740 2024-02-20T00:19:42.032 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:42.014+0000 7ff81415a700 1 -- 172.21.15.132:0/2832481112 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff80c106ea0 con 0x7ff80c0f4740 2024-02-20T00:19:42.032 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:42.014+0000 7ff7fe7fc700 1 -- 172.21.15.132:0/2832481112 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff800013380 con 0x7ff80c0f4740 2024-02-20T00:19:42.032 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:42.014+0000 7ff7fe7fc700 1 -- 172.21.15.132:0/2832481112 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff800014e50 con 0x7ff80c0f4740 2024-02-20T00:19:42.032 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:42.014+0000 7ff81415a700 1 -- 172.21.15.132:0/2832481112 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff80c107330 con 0x7ff80c0f4740 2024-02-20T00:19:42.032 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:42.015+0000 7ff7fe7fc700 1 -- 172.21.15.132:0/2832481112 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff8000263a0 con 0x7ff80c0f4740 2024-02-20T00:19:42.032 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:42.015+0000 7ff7fe7fc700 1 --2- 172.21.15.132:0/2832481112 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7ff7f802ff10 0x7ff7f80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:19:42.032 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:42.015+0000 7ff81415a700 1 -- 172.21.15.132:0/2832481112 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff80c045ad0 con 0x7ff80c0f4740 2024-02-20T00:19:42.032 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:42.015+0000 7ff7fe7fc700 1 -- 172.21.15.132:0/2832481112 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff8000412a0 con 0x7ff80c0f4740 2024-02-20T00:19:42.032 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:42.015+0000 7ff8116f5700 1 --2- 172.21.15.132:0/2832481112 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7ff7f802ff10 0x7ff7f80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:19:42.032 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:42.015+0000 7ff8116f5700 1 --2- 172.21.15.132:0/2832481112 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7ff7f802ff10 0x7ff7f80323d0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7ff8080096a0 tx=0x7ff808006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:19:42.032 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:42.020+0000 7ff7fe7fc700 1 -- 172.21.15.132:0/2832481112 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff8000126f0 con 0x7ff80c0f4740 2024-02-20T00:19:42.227 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:42.225+0000 7ff81415a700 1 -- 172.21.15.132:0/2832481112 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff80c1081e0 con 0x7ff80c0f4740 2024-02-20T00:19:42.228 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:42.226+0000 7ff7fe7fc700 1 -- 172.21.15.132:0/2832481112 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff80000b0b0 con 0x7ff80c0f4740 2024-02-20T00:19:42.228 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:19:42.228 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:19:42.230 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:42.227+0000 7ff81415a700 1 -- 172.21.15.132:0/2832481112 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7ff7f802ff10 msgr2=0x7ff7f80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:19:42.230 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:42.227+0000 7ff81415a700 1 --2- 172.21.15.132:0/2832481112 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7ff7f802ff10 0x7ff7f80323d0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7ff8080096a0 tx=0x7ff808006b40).stop 2024-02-20T00:19:42.230 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:42.227+0000 7ff81415a700 1 -- 172.21.15.132:0/2832481112 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff80c0f4740 msgr2=0x7ff80c1067b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:19:42.230 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:42.227+0000 7ff81415a700 1 --2- 172.21.15.132:0/2832481112 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff80c0f4740 0x7ff80c1067b0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 rx=0x7ff800005010 tx=0x7ff80000b040).stop 2024-02-20T00:19:42.230 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:42.227+0000 7ff81415a700 1 -- 172.21.15.132:0/2832481112 shutdown_connections 2024-02-20T00:19:42.230 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:42.227+0000 7ff81415a700 1 --2- 172.21.15.132:0/2832481112 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7ff7f802ff10 0x7ff7f80323d0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:19:42.230 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:42.227+0000 7ff81415a700 1 --2- 172.21.15.132:0/2832481112 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff80c0f4740 0x7ff80c1067b0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:19:42.230 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:42.227+0000 7ff81415a700 1 -- 172.21.15.132:0/2832481112 >> 172.21.15.132:0/2832481112 conn(0x7ff80c0f0260 msgr2=0x7ff80c0f0cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:19:42.231 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:42.227+0000 7ff81415a700 1 -- 172.21.15.132:0/2832481112 shutdown_connections 2024-02-20T00:19:42.231 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:42.228+0000 7ff81415a700 1 -- 172.21.15.132:0/2832481112 wait complete. 2024-02-20T00:19:42.231 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:19:42.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:19:41 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:19:40.928610+0000 mgr.smithi084.bmqrnf (mgr.14162) 300 : cluster [DBG] pgmap v286: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:19:43.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:19:42 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:19:42.227329+0000 mon.smithi084 (mon.0) 290 : audit [DBG] from='client.? 172.21.15.132:0/2832481112' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:19:43.930 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:19:43.930 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:19:44.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:19:43 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:19:42.928877+0000 mgr.smithi084.bmqrnf (mgr.14162) 301 : cluster [DBG] pgmap v287: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:19:45.818 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:45.812+0000 7fec3406d700 1 -- 172.21.15.132:0/1813560959 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fec2c0f6510 msgr2=0x7fec2c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:19:45.818 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:45.812+0000 7fec3406d700 1 --2- 172.21.15.132:0/1813560959 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fec2c0f6510 0x7fec2c0f6930 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 rx=0x7fec20009fb0 tx=0x7fec20009b30).stop 2024-02-20T00:19:45.818 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:45.812+0000 7fec3406d700 1 -- 172.21.15.132:0/1813560959 shutdown_connections 2024-02-20T00:19:45.818 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:45.812+0000 7fec3406d700 1 --2- 172.21.15.132:0/1813560959 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fec2c0f6510 0x7fec2c0f6930 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:19:45.819 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:45.812+0000 7fec3406d700 1 -- 172.21.15.132:0/1813560959 >> 172.21.15.132:0/1813560959 conn(0x7fec2c0f1c70 msgr2=0x7fec2c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:19:45.819 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:45.812+0000 7fec3406d700 1 -- 172.21.15.132:0/1813560959 shutdown_connections 2024-02-20T00:19:45.819 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:45.812+0000 7fec3406d700 1 -- 172.21.15.132:0/1813560959 wait complete. 2024-02-20T00:19:45.819 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:45.813+0000 7fec3406d700 1 Processor -- start 2024-02-20T00:19:45.819 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:45.813+0000 7fec3406d700 1 -- start start 2024-02-20T00:19:45.819 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:45.813+0000 7fec3406d700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fec2c0f6510 0x7fec2c1067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:19:45.819 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:45.813+0000 7fec3406d700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec2c106cd0 con 0x7fec2c0f6510 2024-02-20T00:19:45.819 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:45.813+0000 7fec31e09700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fec2c0f6510 0x7fec2c1067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:19:45.819 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:45.813+0000 7fec31e09700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fec2c0f6510 0x7fec2c1067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:50916/0 (socket says 172.21.15.132:50916) 2024-02-20T00:19:45.819 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:45.813+0000 7fec31e09700 1 -- 172.21.15.132:0/1028027219 learned_addr learned my addr 172.21.15.132:0/1028027219 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:19:45.819 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:45.814+0000 7fec31e09700 1 -- 172.21.15.132:0/1028027219 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec20005040 con 0x7fec2c0f6510 2024-02-20T00:19:45.819 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:45.814+0000 7fec31e09700 1 --2- 172.21.15.132:0/1028027219 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fec2c0f6510 0x7fec2c1067e0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0x7fec20005010 tx=0x7fec2000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:19:45.819 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:45.814+0000 7fec1e7fc700 1 -- 172.21.15.132:0/1028027219 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fec20009320 con 0x7fec2c0f6510 2024-02-20T00:19:45.820 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:45.814+0000 7fec3406d700 1 -- 172.21.15.132:0/1028027219 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec2c106ed0 con 0x7fec2c0f6510 2024-02-20T00:19:45.820 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:45.814+0000 7fec1e7fc700 1 -- 172.21.15.132:0/1028027219 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fec20013380 con 0x7fec2c0f6510 2024-02-20T00:19:45.820 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:45.814+0000 7fec1e7fc700 1 -- 172.21.15.132:0/1028027219 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fec20014e50 con 0x7fec2c0f6510 2024-02-20T00:19:45.820 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:45.814+0000 7fec3406d700 1 -- 172.21.15.132:0/1028027219 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec2c107360 con 0x7fec2c0f6510 2024-02-20T00:19:45.820 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:45.815+0000 7fec1e7fc700 1 -- 172.21.15.132:0/1028027219 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fec20014420 con 0x7fec2c0f6510 2024-02-20T00:19:45.820 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:45.815+0000 7fec3406d700 1 -- 172.21.15.132:0/1028027219 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec2c045ad0 con 0x7fec2c0f6510 2024-02-20T00:19:45.820 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:45.815+0000 7fec1e7fc700 1 --2- 172.21.15.132:0/1028027219 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fec1802ff10 0x7fec180323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:19:45.820 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:45.815+0000 7fec1e7fc700 1 -- 172.21.15.132:0/1028027219 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fec20040eb0 con 0x7fec2c0f6510 2024-02-20T00:19:45.820 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:45.815+0000 7fec31608700 1 --2- 172.21.15.132:0/1028027219 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fec1802ff10 0x7fec180323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:19:45.820 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:45.815+0000 7fec31608700 1 --2- 172.21.15.132:0/1028027219 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fec1802ff10 0x7fec180323d0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7fec280096a0 tx=0x7fec28006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:19:45.821 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:45.820+0000 7fec1e7fc700 1 -- 172.21.15.132:0/1028027219 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fec200095f0 con 0x7fec2c0f6510 2024-02-20T00:19:46.024 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:46.022+0000 7fec3406d700 1 -- 172.21.15.132:0/1028027219 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fec2c108210 con 0x7fec2c0f6510 2024-02-20T00:19:46.024 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:46.023+0000 7fec1e7fc700 1 -- 172.21.15.132:0/1028027219 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fec2001db70 con 0x7fec2c0f6510 2024-02-20T00:19:46.025 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:19:46.025 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:19:46.026 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:46.025+0000 7fec3406d700 1 -- 172.21.15.132:0/1028027219 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fec1802ff10 msgr2=0x7fec180323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:19:46.026 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:46.025+0000 7fec3406d700 1 --2- 172.21.15.132:0/1028027219 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fec1802ff10 0x7fec180323d0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7fec280096a0 tx=0x7fec28006b40).stop 2024-02-20T00:19:46.026 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:46.025+0000 7fec3406d700 1 -- 172.21.15.132:0/1028027219 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fec2c0f6510 msgr2=0x7fec2c1067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:19:46.026 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:46.025+0000 7fec3406d700 1 --2- 172.21.15.132:0/1028027219 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fec2c0f6510 0x7fec2c1067e0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0x7fec20005010 tx=0x7fec2000b040).stop 2024-02-20T00:19:46.026 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:46.025+0000 7fec3406d700 1 -- 172.21.15.132:0/1028027219 shutdown_connections 2024-02-20T00:19:46.026 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:46.025+0000 7fec3406d700 1 --2- 172.21.15.132:0/1028027219 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fec1802ff10 0x7fec180323d0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:19:46.027 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:46.025+0000 7fec3406d700 1 --2- 172.21.15.132:0/1028027219 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fec2c0f6510 0x7fec2c1067e0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:19:46.027 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:46.025+0000 7fec3406d700 1 -- 172.21.15.132:0/1028027219 >> 172.21.15.132:0/1028027219 conn(0x7fec2c0f1c70 msgr2=0x7fec2c0f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:19:46.027 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:46.025+0000 7fec3406d700 1 -- 172.21.15.132:0/1028027219 shutdown_connections 2024-02-20T00:19:46.027 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:46.025+0000 7fec3406d700 1 -- 172.21.15.132:0/1028027219 wait complete. 2024-02-20T00:19:46.027 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:19:46.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:19:45 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:19:44.929249+0000 mgr.smithi084.bmqrnf (mgr.14162) 302 : cluster [DBG] pgmap v288: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:19:47.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:19:46 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:19:46.023656+0000 mon.smithi084 (mon.0) 291 : audit [DBG] from='client.? 172.21.15.132:0/1028027219' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:19:47.716 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:19:47.717 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:19:48.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:19:47 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:19:46.929635+0000 mgr.smithi084.bmqrnf (mgr.14162) 303 : cluster [DBG] pgmap v289: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:19:49.516 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:49.515+0000 7fd0e9159700 1 -- 172.21.15.132:0/4019518141 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd0e40f6510 msgr2=0x7fd0e40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:19:49.517 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:49.515+0000 7fd0e9159700 1 --2- 172.21.15.132:0/4019518141 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd0e40f6510 0x7fd0e40f6930 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 rx=0x7fd0d8009fb0 tx=0x7fd0d8009b30).stop 2024-02-20T00:19:49.517 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:49.515+0000 7fd0e9159700 1 -- 172.21.15.132:0/4019518141 shutdown_connections 2024-02-20T00:19:49.517 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:49.515+0000 7fd0e9159700 1 --2- 172.21.15.132:0/4019518141 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd0e40f6510 0x7fd0e40f6930 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:19:49.517 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:49.515+0000 7fd0e9159700 1 -- 172.21.15.132:0/4019518141 >> 172.21.15.132:0/4019518141 conn(0x7fd0e40f1c70 msgr2=0x7fd0e40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:19:49.517 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:49.515+0000 7fd0e9159700 1 -- 172.21.15.132:0/4019518141 shutdown_connections 2024-02-20T00:19:49.517 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:49.515+0000 7fd0e9159700 1 -- 172.21.15.132:0/4019518141 wait complete. 2024-02-20T00:19:49.518 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:49.515+0000 7fd0e9159700 1 Processor -- start 2024-02-20T00:19:49.518 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:49.516+0000 7fd0e9159700 1 -- start start 2024-02-20T00:19:49.518 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:49.516+0000 7fd0e9159700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd0e40f6510 0x7fd0e41067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:19:49.519 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:49.516+0000 7fd0e9159700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0e4106cd0 con 0x7fd0e40f6510 2024-02-20T00:19:49.519 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:49.516+0000 7fd0e2d9d700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd0e40f6510 0x7fd0e41067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:19:49.519 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:49.516+0000 7fd0e2d9d700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd0e40f6510 0x7fd0e41067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:52884/0 (socket says 172.21.15.132:52884) 2024-02-20T00:19:49.519 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:49.516+0000 7fd0e2d9d700 1 -- 172.21.15.132:0/3850478894 learned_addr learned my addr 172.21.15.132:0/3850478894 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:19:49.519 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:49.516+0000 7fd0e2d9d700 1 -- 172.21.15.132:0/3850478894 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0d8005040 con 0x7fd0e40f6510 2024-02-20T00:19:49.519 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:49.517+0000 7fd0e2d9d700 1 --2- 172.21.15.132:0/3850478894 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd0e40f6510 0x7fd0e41067e0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 rx=0x7fd0d8005010 tx=0x7fd0d800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:19:49.519 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:49.517+0000 7fd0cb7fe700 1 -- 172.21.15.132:0/3850478894 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd0d8009320 con 0x7fd0e40f6510 2024-02-20T00:19:49.519 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:49.517+0000 7fd0e9159700 1 -- 172.21.15.132:0/3850478894 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0e4106ed0 con 0x7fd0e40f6510 2024-02-20T00:19:49.519 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:49.517+0000 7fd0cb7fe700 1 -- 172.21.15.132:0/3850478894 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd0d8013380 con 0x7fd0e40f6510 2024-02-20T00:19:49.519 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:49.517+0000 7fd0cb7fe700 1 -- 172.21.15.132:0/3850478894 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd0d8014e50 con 0x7fd0e40f6510 2024-02-20T00:19:49.519 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:49.517+0000 7fd0e9159700 1 -- 172.21.15.132:0/3850478894 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0e4107360 con 0x7fd0e40f6510 2024-02-20T00:19:49.520 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:49.518+0000 7fd0e9159700 1 -- 172.21.15.132:0/3850478894 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0e4045ad0 con 0x7fd0e40f6510 2024-02-20T00:19:49.520 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:49.518+0000 7fd0cb7fe700 1 -- 172.21.15.132:0/3850478894 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd0d80095f0 con 0x7fd0e40f6510 2024-02-20T00:19:49.520 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:49.518+0000 7fd0cb7fe700 1 --2- 172.21.15.132:0/3850478894 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fd0cc02ff10 0x7fd0cc0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:19:49.520 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:49.518+0000 7fd0cb7fe700 1 -- 172.21.15.132:0/3850478894 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd0d8040e90 con 0x7fd0e40f6510 2024-02-20T00:19:49.520 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:49.518+0000 7fd0e259c700 1 --2- 172.21.15.132:0/3850478894 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fd0cc02ff10 0x7fd0cc0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:19:49.520 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:49.518+0000 7fd0e259c700 1 --2- 172.21.15.132:0/3850478894 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fd0cc02ff10 0x7fd0cc0323d0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7fd0d40096a0 tx=0x7fd0d4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:19:49.524 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:49.522+0000 7fd0cb7fe700 1 -- 172.21.15.132:0/3850478894 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd0d8014420 con 0x7fd0e40f6510 2024-02-20T00:19:49.728 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:49.726+0000 7fd0e9159700 1 -- 172.21.15.132:0/3850478894 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd0e4108210 con 0x7fd0e40f6510 2024-02-20T00:19:49.728 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:49.727+0000 7fd0cb7fe700 1 -- 172.21.15.132:0/3850478894 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd0d8010230 con 0x7fd0e40f6510 2024-02-20T00:19:49.729 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:19:49.729 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:19:49.730 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:49.729+0000 7fd0e9159700 1 -- 172.21.15.132:0/3850478894 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fd0cc02ff10 msgr2=0x7fd0cc0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:19:49.731 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:49.729+0000 7fd0e9159700 1 --2- 172.21.15.132:0/3850478894 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fd0cc02ff10 0x7fd0cc0323d0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7fd0d40096a0 tx=0x7fd0d4006b40).stop 2024-02-20T00:19:49.731 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:49.729+0000 7fd0e9159700 1 -- 172.21.15.132:0/3850478894 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd0e40f6510 msgr2=0x7fd0e41067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:19:49.731 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:49.729+0000 7fd0e9159700 1 --2- 172.21.15.132:0/3850478894 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd0e40f6510 0x7fd0e41067e0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 rx=0x7fd0d8005010 tx=0x7fd0d800b040).stop 2024-02-20T00:19:49.731 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:49.729+0000 7fd0e9159700 1 -- 172.21.15.132:0/3850478894 shutdown_connections 2024-02-20T00:19:49.731 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:49.729+0000 7fd0e9159700 1 --2- 172.21.15.132:0/3850478894 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fd0cc02ff10 0x7fd0cc0323d0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:19:49.731 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:49.729+0000 7fd0e9159700 1 --2- 172.21.15.132:0/3850478894 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd0e40f6510 0x7fd0e41067e0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:19:49.731 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:49.729+0000 7fd0e9159700 1 -- 172.21.15.132:0/3850478894 >> 172.21.15.132:0/3850478894 conn(0x7fd0e40f1c70 msgr2=0x7fd0e40f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:19:49.731 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:49.729+0000 7fd0e9159700 1 -- 172.21.15.132:0/3850478894 shutdown_connections 2024-02-20T00:19:49.731 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:49.729+0000 7fd0e9159700 1 -- 172.21.15.132:0/3850478894 wait complete. 2024-02-20T00:19:49.731 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:19:50.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:19:49 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:19:48.930005+0000 mgr.smithi084.bmqrnf (mgr.14162) 304 : cluster [DBG] pgmap v290: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:19:50.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:19:49 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:19:49.727699+0000 mon.smithi084 (mon.0) 292 : audit [DBG] from='client.? 172.21.15.132:0/3850478894' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:19:51.437 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:19:51.437 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:19:52.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:19:51 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:19:50.930323+0000 mgr.smithi084.bmqrnf (mgr.14162) 305 : cluster [DBG] pgmap v291: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:19:53.509 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:53.506+0000 7f12d87ce700 1 -- 172.21.15.132:0/397013432 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f12d00f5410 msgr2=0x7f12d00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:19:53.509 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:53.506+0000 7f12d87ce700 1 --2- 172.21.15.132:0/397013432 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f12d00f5410 0x7f12d00f5830 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 rx=0x7f12c4009fb0 tx=0x7f12c4009b30).stop 2024-02-20T00:19:53.510 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:53.506+0000 7f12d87ce700 1 -- 172.21.15.132:0/397013432 shutdown_connections 2024-02-20T00:19:53.510 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:53.506+0000 7f12d87ce700 1 --2- 172.21.15.132:0/397013432 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f12d00f5410 0x7f12d00f5830 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:19:53.510 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:53.506+0000 7f12d87ce700 1 -- 172.21.15.132:0/397013432 >> 172.21.15.132:0/397013432 conn(0x7f12d00f0b70 msgr2=0x7f12d00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:19:53.510 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:53.506+0000 7f12d87ce700 1 -- 172.21.15.132:0/397013432 shutdown_connections 2024-02-20T00:19:53.510 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:53.506+0000 7f12d87ce700 1 -- 172.21.15.132:0/397013432 wait complete. 2024-02-20T00:19:53.510 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:53.507+0000 7f12d87ce700 1 Processor -- start 2024-02-20T00:19:53.510 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:53.507+0000 7f12d87ce700 1 -- start start 2024-02-20T00:19:53.510 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:53.507+0000 7f12d87ce700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f12d00f5410 0x7f12d010b930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:19:53.510 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:53.507+0000 7f12d87ce700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12d0107320 con 0x7f12d00f5410 2024-02-20T00:19:53.510 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:53.507+0000 7f12d656a700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f12d00f5410 0x7f12d010b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:19:53.510 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:53.507+0000 7f12d656a700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f12d00f5410 0x7f12d010b930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:52906/0 (socket says 172.21.15.132:52906) 2024-02-20T00:19:53.510 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:53.507+0000 7f12d656a700 1 -- 172.21.15.132:0/3127144250 learned_addr learned my addr 172.21.15.132:0/3127144250 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:19:53.510 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:53.507+0000 7f12d656a700 1 -- 172.21.15.132:0/3127144250 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12c4005040 con 0x7f12d00f5410 2024-02-20T00:19:53.511 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:53.508+0000 7f12d656a700 1 --2- 172.21.15.132:0/3127144250 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f12d00f5410 0x7f12d010b930 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0x7f12c4005010 tx=0x7f12c400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:19:53.511 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:53.508+0000 7f12c2ffd700 1 -- 172.21.15.132:0/3127144250 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f12c4009320 con 0x7f12d00f5410 2024-02-20T00:19:53.511 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:53.508+0000 7f12d87ce700 1 -- 172.21.15.132:0/3127144250 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12d0060220 con 0x7f12d00f5410 2024-02-20T00:19:53.511 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:53.508+0000 7f12c2ffd700 1 -- 172.21.15.132:0/3127144250 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f12c4013380 con 0x7f12d00f5410 2024-02-20T00:19:53.511 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:53.508+0000 7f12c2ffd700 1 -- 172.21.15.132:0/3127144250 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f12c4014e50 con 0x7f12d00f5410 2024-02-20T00:19:53.511 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:53.508+0000 7f12d87ce700 1 -- 172.21.15.132:0/3127144250 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12d010be20 con 0x7f12d00f5410 2024-02-20T00:19:53.512 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:53.509+0000 7f12d87ce700 1 -- 172.21.15.132:0/3127144250 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12d0045ad0 con 0x7f12d00f5410 2024-02-20T00:19:53.512 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:53.509+0000 7f12c2ffd700 1 -- 172.21.15.132:0/3127144250 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f12c4014420 con 0x7f12d00f5410 2024-02-20T00:19:53.512 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:53.509+0000 7f12c2ffd700 1 --2- 172.21.15.132:0/3127144250 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f12bc02ff10 0x7f12bc0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:19:53.512 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:53.509+0000 7f12c2ffd700 1 -- 172.21.15.132:0/3127144250 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f12c4040e30 con 0x7f12d00f5410 2024-02-20T00:19:53.512 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:53.509+0000 7f12d5d69700 1 --2- 172.21.15.132:0/3127144250 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f12bc02ff10 0x7f12bc0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:19:53.512 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:53.509+0000 7f12d5d69700 1 --2- 172.21.15.132:0/3127144250 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f12bc02ff10 0x7f12bc0323d0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7f12cc0096a0 tx=0x7f12cc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:19:53.515 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:53.514+0000 7f12c2ffd700 1 -- 172.21.15.132:0/3127144250 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f12c40095f0 con 0x7f12d00f5410 2024-02-20T00:19:53.725 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:53.724+0000 7f12d87ce700 1 -- 172.21.15.132:0/3127144250 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f12d010c5e0 con 0x7f12d00f5410 2024-02-20T00:19:53.726 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:53.725+0000 7f12c2ffd700 1 -- 172.21.15.132:0/3127144250 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f12c401db70 con 0x7f12d00f5410 2024-02-20T00:19:53.726 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:19:53.726 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:19:53.728 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:53.726+0000 7f12d87ce700 1 -- 172.21.15.132:0/3127144250 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f12bc02ff10 msgr2=0x7f12bc0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:19:53.728 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:53.726+0000 7f12d87ce700 1 --2- 172.21.15.132:0/3127144250 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f12bc02ff10 0x7f12bc0323d0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7f12cc0096a0 tx=0x7f12cc006b40).stop 2024-02-20T00:19:53.728 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:53.726+0000 7f12d87ce700 1 -- 172.21.15.132:0/3127144250 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f12d00f5410 msgr2=0x7f12d010b930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:19:53.728 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:53.726+0000 7f12d87ce700 1 --2- 172.21.15.132:0/3127144250 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f12d00f5410 0x7f12d010b930 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0x7f12c4005010 tx=0x7f12c400b040).stop 2024-02-20T00:19:53.728 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:53.726+0000 7f12d87ce700 1 -- 172.21.15.132:0/3127144250 shutdown_connections 2024-02-20T00:19:53.728 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:53.726+0000 7f12d87ce700 1 --2- 172.21.15.132:0/3127144250 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f12bc02ff10 0x7f12bc0323d0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:19:53.729 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:53.726+0000 7f12d87ce700 1 --2- 172.21.15.132:0/3127144250 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f12d00f5410 0x7f12d010b930 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:19:53.729 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:53.726+0000 7f12d87ce700 1 -- 172.21.15.132:0/3127144250 >> 172.21.15.132:0/3127144250 conn(0x7f12d00f0b70 msgr2=0x7f12d00f15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:19:53.729 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:53.726+0000 7f12d87ce700 1 -- 172.21.15.132:0/3127144250 shutdown_connections 2024-02-20T00:19:53.729 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:53.726+0000 7f12d87ce700 1 -- 172.21.15.132:0/3127144250 wait complete. 2024-02-20T00:19:53.729 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:19:54.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:19:53 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:19:52.930677+0000 mgr.smithi084.bmqrnf (mgr.14162) 306 : cluster [DBG] pgmap v292: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:19:54.359 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:19:53 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:19:53.725272+0000 mon.smithi084 (mon.0) 293 : audit [DBG] from='client.? 172.21.15.132:0/3127144250' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:19:55.437 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:19:55.437 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:19:56.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:19:55 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:19:54.931045+0000 mgr.smithi084.bmqrnf (mgr.14162) 307 : cluster [DBG] pgmap v293: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:19:57.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:19:56 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:19:56.982121+0000 mon.smithi084 (mon.0) 294 : audit [INF] from='mgr.14162 172.21.15.84:0/2152933216' entity='mgr.smithi084.bmqrnf' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi084.bmqrnf/mirror_snapshot_schedule"}]: dispatch 2024-02-20T00:19:57.426 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:57.423+0000 7f0f2a438700 1 -- 172.21.15.132:0/1078973348 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0f24060220 msgr2=0x7f0f240f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:19:57.426 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:57.423+0000 7f0f2a438700 1 --2- 172.21.15.132:0/1078973348 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0f24060220 0x7f0f240f9150 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 rx=0x7f0f14009fb0 tx=0x7f0f14009b30).stop 2024-02-20T00:19:57.426 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:57.423+0000 7f0f2a438700 1 -- 172.21.15.132:0/1078973348 shutdown_connections 2024-02-20T00:19:57.426 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:57.423+0000 7f0f2a438700 1 --2- 172.21.15.132:0/1078973348 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0f24060220 0x7f0f240f9150 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:19:57.426 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:57.423+0000 7f0f2a438700 1 -- 172.21.15.132:0/1078973348 >> 172.21.15.132:0/1078973348 conn(0x7f0f240f0b70 msgr2=0x7f0f240f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:19:57.426 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:57.423+0000 7f0f2a438700 1 -- 172.21.15.132:0/1078973348 shutdown_connections 2024-02-20T00:19:57.427 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:57.423+0000 7f0f2a438700 1 -- 172.21.15.132:0/1078973348 wait complete. 2024-02-20T00:19:57.427 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:57.424+0000 7f0f2a438700 1 Processor -- start 2024-02-20T00:19:57.427 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:57.424+0000 7f0f2a438700 1 -- start start 2024-02-20T00:19:57.427 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:57.424+0000 7f0f2a438700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0f24060220 0x7f0f241067a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:19:57.427 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:57.424+0000 7f0f2a438700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f24106c90 con 0x7f0f24060220 2024-02-20T00:19:57.430 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:57.425+0000 7f0f23fff700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0f24060220 0x7f0f241067a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:19:57.430 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:57.425+0000 7f0f23fff700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0f24060220 0x7f0f241067a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:46658/0 (socket says 172.21.15.132:46658) 2024-02-20T00:19:57.430 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:57.425+0000 7f0f23fff700 1 -- 172.21.15.132:0/2074805193 learned_addr learned my addr 172.21.15.132:0/2074805193 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:19:57.430 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:57.425+0000 7f0f23fff700 1 -- 172.21.15.132:0/2074805193 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f14005040 con 0x7f0f24060220 2024-02-20T00:19:57.430 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:57.425+0000 7f0f23fff700 1 --2- 172.21.15.132:0/2074805193 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0f24060220 0x7f0f241067a0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0x7f0f140089f0 tx=0x7f0f1400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:19:57.430 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:57.425+0000 7f0f20ff9700 1 -- 172.21.15.132:0/2074805193 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0f14009320 con 0x7f0f24060220 2024-02-20T00:19:57.430 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:57.425+0000 7f0f2a438700 1 -- 172.21.15.132:0/2074805193 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f24106e90 con 0x7f0f24060220 2024-02-20T00:19:57.430 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:57.425+0000 7f0f2a438700 1 -- 172.21.15.132:0/2074805193 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f24107320 con 0x7f0f24060220 2024-02-20T00:19:57.430 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:57.426+0000 7f0f20ff9700 1 -- 172.21.15.132:0/2074805193 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0f14013380 con 0x7f0f24060220 2024-02-20T00:19:57.431 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:57.426+0000 7f0f20ff9700 1 -- 172.21.15.132:0/2074805193 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0f14014db0 con 0x7f0f24060220 2024-02-20T00:19:57.431 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:57.426+0000 7f0f20ff9700 1 -- 172.21.15.132:0/2074805193 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0f140263a0 con 0x7f0f24060220 2024-02-20T00:19:57.431 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:57.426+0000 7f0f2a438700 1 -- 172.21.15.132:0/2074805193 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f04004fa0 con 0x7f0f24060220 2024-02-20T00:19:57.431 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:57.427+0000 7f0f20ff9700 1 --2- 172.21.15.132:0/2074805193 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f0f0c02ff60 0x7f0f0c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:19:57.431 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:57.427+0000 7f0f20ff9700 1 -- 172.21.15.132:0/2074805193 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0f14041160 con 0x7f0f24060220 2024-02-20T00:19:57.431 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:57.427+0000 7f0f237fe700 1 --2- 172.21.15.132:0/2074805193 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f0f0c02ff60 0x7f0f0c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:19:57.431 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:57.427+0000 7f0f237fe700 1 --2- 172.21.15.132:0/2074805193 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f0f0c02ff60 0x7f0f0c032420 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f0f180096a0 tx=0x7f0f18006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:19:57.432 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:57.431+0000 7f0f20ff9700 1 -- 172.21.15.132:0/2074805193 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0f14012920 con 0x7f0f24060220 2024-02-20T00:19:57.639 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:57.637+0000 7f0f2a438700 1 -- 172.21.15.132:0/2074805193 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0f04005670 con 0x7f0f24060220 2024-02-20T00:19:57.639 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:57.638+0000 7f0f20ff9700 1 -- 172.21.15.132:0/2074805193 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0f14004490 con 0x7f0f24060220 2024-02-20T00:19:57.639 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:19:57.640 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:19:57.641 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:57.639+0000 7f0f2a438700 1 -- 172.21.15.132:0/2074805193 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f0f0c02ff60 msgr2=0x7f0f0c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:19:57.641 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:57.639+0000 7f0f2a438700 1 --2- 172.21.15.132:0/2074805193 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f0f0c02ff60 0x7f0f0c032420 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f0f180096a0 tx=0x7f0f18006b40).stop 2024-02-20T00:19:57.641 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:57.639+0000 7f0f2a438700 1 -- 172.21.15.132:0/2074805193 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0f24060220 msgr2=0x7f0f241067a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:19:57.641 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:57.639+0000 7f0f2a438700 1 --2- 172.21.15.132:0/2074805193 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0f24060220 0x7f0f241067a0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0x7f0f140089f0 tx=0x7f0f1400b040).stop 2024-02-20T00:19:57.641 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:57.639+0000 7f0f2a438700 1 -- 172.21.15.132:0/2074805193 shutdown_connections 2024-02-20T00:19:57.641 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:57.639+0000 7f0f2a438700 1 --2- 172.21.15.132:0/2074805193 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f0f0c02ff60 0x7f0f0c032420 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:19:57.641 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:57.640+0000 7f0f2a438700 1 --2- 172.21.15.132:0/2074805193 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0f24060220 0x7f0f241067a0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:19:57.642 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:57.640+0000 7f0f2a438700 1 -- 172.21.15.132:0/2074805193 >> 172.21.15.132:0/2074805193 conn(0x7f0f240f0b70 msgr2=0x7f0f240f15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:19:57.642 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:57.640+0000 7f0f2a438700 1 -- 172.21.15.132:0/2074805193 shutdown_connections 2024-02-20T00:19:57.642 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:19:57.640+0000 7f0f2a438700 1 -- 172.21.15.132:0/2074805193 wait complete. 2024-02-20T00:19:57.642 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:19:58.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:19:57 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:19:56.931308+0000 mgr.smithi084.bmqrnf (mgr.14162) 308 : cluster [DBG] pgmap v294: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:19:58.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:19:57 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:19:56.987142+0000 mon.smithi084 (mon.0) 295 : audit [INF] from='mgr.14162 172.21.15.84:0/2152933216' entity='mgr.smithi084.bmqrnf' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi084.bmqrnf/trash_purge_schedule"}]: dispatch 2024-02-20T00:19:58.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:19:57 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:19:57.638576+0000 mon.smithi084 (mon.0) 296 : audit [DBG] from='client.? 172.21.15.132:0/2074805193' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:19:59.380 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:19:59.381 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:20:00.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:19:59 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:19:58.931626+0000 mgr.smithi084.bmqrnf (mgr.14162) 309 : cluster [DBG] pgmap v295: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:20:01.263 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:01.261+0000 7f09a2afe700 1 -- 172.21.15.132:0/868198093 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f099c0f6510 msgr2=0x7f099c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:20:01.263 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:01.261+0000 7f09a2afe700 1 --2- 172.21.15.132:0/868198093 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f099c0f6510 0x7f099c0f6930 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 rx=0x7f0990009fb0 tx=0x7f0990009b30).stop 2024-02-20T00:20:01.263 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:01.261+0000 7f09a2afe700 1 -- 172.21.15.132:0/868198093 shutdown_connections 2024-02-20T00:20:01.263 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:01.261+0000 7f09a2afe700 1 --2- 172.21.15.132:0/868198093 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f099c0f6510 0x7f099c0f6930 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:20:01.263 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:01.261+0000 7f09a2afe700 1 -- 172.21.15.132:0/868198093 >> 172.21.15.132:0/868198093 conn(0x7f099c0f1c70 msgr2=0x7f099c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:20:01.263 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:01.261+0000 7f09a2afe700 1 -- 172.21.15.132:0/868198093 shutdown_connections 2024-02-20T00:20:01.264 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:01.261+0000 7f09a2afe700 1 -- 172.21.15.132:0/868198093 wait complete. 2024-02-20T00:20:01.264 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:01.262+0000 7f09a2afe700 1 Processor -- start 2024-02-20T00:20:01.264 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:01.262+0000 7f09a2afe700 1 -- start start 2024-02-20T00:20:01.266 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:01.262+0000 7f09a2afe700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f099c0f6510 0x7f099c1067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:20:01.266 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:01.262+0000 7f09a2afe700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f099c106cd0 con 0x7f099c0f6510 2024-02-20T00:20:01.266 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:01.262+0000 7f09a089a700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f099c0f6510 0x7f099c1067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:20:01.266 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:01.263+0000 7f09a089a700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f099c0f6510 0x7f099c1067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:46670/0 (socket says 172.21.15.132:46670) 2024-02-20T00:20:01.266 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:01.263+0000 7f09a089a700 1 -- 172.21.15.132:0/2698842540 learned_addr learned my addr 172.21.15.132:0/2698842540 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:20:01.266 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:01.263+0000 7f09a089a700 1 -- 172.21.15.132:0/2698842540 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0990005040 con 0x7f099c0f6510 2024-02-20T00:20:01.266 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:01.263+0000 7f09a089a700 1 --2- 172.21.15.132:0/2698842540 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f099c0f6510 0x7f099c1067e0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 rx=0x7f09900089f0 tx=0x7f099000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:20:01.266 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:01.263+0000 7f09997fa700 1 -- 172.21.15.132:0/2698842540 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0990009320 con 0x7f099c0f6510 2024-02-20T00:20:01.266 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:01.263+0000 7f09a2afe700 1 -- 172.21.15.132:0/2698842540 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f099c106ed0 con 0x7f099c0f6510 2024-02-20T00:20:01.266 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:01.263+0000 7f09997fa700 1 -- 172.21.15.132:0/2698842540 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0990013380 con 0x7f099c0f6510 2024-02-20T00:20:01.266 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:01.263+0000 7f09997fa700 1 -- 172.21.15.132:0/2698842540 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f099001ed80 con 0x7f099c0f6510 2024-02-20T00:20:01.266 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:01.263+0000 7f09a2afe700 1 -- 172.21.15.132:0/2698842540 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f099c107360 con 0x7f099c0f6510 2024-02-20T00:20:01.266 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:01.264+0000 7f09997fa700 1 -- 172.21.15.132:0/2698842540 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0990014420 con 0x7f099c0f6510 2024-02-20T00:20:01.266 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:01.264+0000 7f09997fa700 1 --2- 172.21.15.132:0/2698842540 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f098c02ff60 0x7f098c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:20:01.267 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:01.264+0000 7f09997fa700 1 -- 172.21.15.132:0/2698842540 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0990041ef0 con 0x7f099c0f6510 2024-02-20T00:20:01.267 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:01.264+0000 7f09a2afe700 1 -- 172.21.15.132:0/2698842540 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f099c045ad0 con 0x7f099c0f6510 2024-02-20T00:20:01.267 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:01.264+0000 7f099bfff700 1 --2- 172.21.15.132:0/2698842540 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f098c02ff60 0x7f098c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:20:01.267 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:01.265+0000 7f099bfff700 1 --2- 172.21.15.132:0/2698842540 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f098c02ff60 0x7f098c032420 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f09880096a0 tx=0x7f0988006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:20:01.270 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:01.269+0000 7f09997fa700 1 -- 172.21.15.132:0/2698842540 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f099001c910 con 0x7f099c0f6510 2024-02-20T00:20:01.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:20:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:20:00.607265+0000 mon.smithi084 (mon.0) 297 : audit [DBG] from='mgr.14162 172.21.15.84:0/2152933216' entity='mgr.smithi084.bmqrnf' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-20T00:20:01.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:20:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:20:00.833661+0000 mon.smithi084 (mon.0) 298 : audit [INF] from='mgr.14162 172.21.15.84:0/2152933216' entity='mgr.smithi084.bmqrnf' 2024-02-20T00:20:01.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:20:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:20:00.846040+0000 mon.smithi084 (mon.0) 299 : audit [INF] from='mgr.14162 172.21.15.84:0/2152933216' entity='mgr.smithi084.bmqrnf' 2024-02-20T00:20:01.513 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:01.511+0000 7f09a2afe700 1 -- 172.21.15.132:0/2698842540 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f099c1080a0 con 0x7f099c0f6510 2024-02-20T00:20:01.514 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:01.512+0000 7f09997fa700 1 -- 172.21.15.132:0/2698842540 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f099001c320 con 0x7f099c0f6510 2024-02-20T00:20:01.514 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:20:01.514 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:20:01.516 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:01.514+0000 7f09a2afe700 1 -- 172.21.15.132:0/2698842540 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f098c02ff60 msgr2=0x7f098c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:20:01.516 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:01.514+0000 7f09a2afe700 1 --2- 172.21.15.132:0/2698842540 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f098c02ff60 0x7f098c032420 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f09880096a0 tx=0x7f0988006b40).stop 2024-02-20T00:20:01.516 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:01.514+0000 7f09a2afe700 1 -- 172.21.15.132:0/2698842540 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f099c0f6510 msgr2=0x7f099c1067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:20:01.516 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:01.514+0000 7f09a2afe700 1 --2- 172.21.15.132:0/2698842540 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f099c0f6510 0x7f099c1067e0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 rx=0x7f09900089f0 tx=0x7f099000b040).stop 2024-02-20T00:20:01.516 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:01.514+0000 7f09a2afe700 1 -- 172.21.15.132:0/2698842540 shutdown_connections 2024-02-20T00:20:01.516 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:01.514+0000 7f09a2afe700 1 --2- 172.21.15.132:0/2698842540 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f098c02ff60 0x7f098c032420 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:20:01.516 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:01.514+0000 7f09a2afe700 1 --2- 172.21.15.132:0/2698842540 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f099c0f6510 0x7f099c1067e0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:20:01.516 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:01.514+0000 7f09a2afe700 1 -- 172.21.15.132:0/2698842540 >> 172.21.15.132:0/2698842540 conn(0x7f099c0f1c70 msgr2=0x7f099c0f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:20:01.516 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:01.514+0000 7f09a2afe700 1 -- 172.21.15.132:0/2698842540 shutdown_connections 2024-02-20T00:20:01.516 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:01.515+0000 7f09a2afe700 1 -- 172.21.15.132:0/2698842540 wait complete. 2024-02-20T00:20:01.517 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:20:02.609 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:20:02 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cephadm 2024-02-20T00:20:00.606639+0000 mgr.smithi084.bmqrnf (mgr.14162) 310 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Deploy daemon prometheus.smithi084 ... 2024-02-20T00:20:02.609 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:20:02 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/prometheus:v2.18.1 -e NODE_NAME=smithi084 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/prometheus:v2.18.1 -c %u %g /etc/prometheus 2024-02-20T00:20:02.609 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:20:02 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: stat: stderr Trying to pull docker.io/prom/prometheus:v2.18.1... 2024-02-20T00:20:02.609 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:20:02 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: stat: stderr time="2024-02-20T00:18:08Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-20T00:20:02.609 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:20:02 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: stat: stderr time="2024-02-20T00:18:45Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: copying system image from manifest list: determining manifest MIME type for docker://prom/prometheus:v2.18.1: reading manifest sha256:b4e6cd0275a26750505e539f8528e891053434ebd3972be02645bed5f02f0795 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-20T00:20:02.609 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:20:02 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: stat: stderr time="2024-02-20T00:19:23Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-20T00:20:02.609 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:20:02 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: stat: stderr Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-20T00:20:02.609 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:20:02 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: Traceback (most recent call last): 2024-02-20T00:20:02.609 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:20:02 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: File "/var/lib/ceph/2a2e6c42-cf84-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-02-20T00:20:02.610 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:20:02 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: main() 2024-02-20T00:20:02.610 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:20:02 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: File "/var/lib/ceph/2a2e6c42-cf84-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-02-20T00:20:02.610 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:20:02 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: r = ctx.func(ctx) 2024-02-20T00:20:02.610 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:20:02 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: File "/var/lib/ceph/2a2e6c42-cf84-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-02-20T00:20:02.610 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:20:02 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: return func(ctx) 2024-02-20T00:20:02.610 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:20:02 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: File "/var/lib/ceph/2a2e6c42-cf84-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy 2024-02-20T00:20:02.610 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:20:02 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2024-02-20T00:20:02.610 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:20:02 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: File "/var/lib/ceph/2a2e6c42-cf84-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4185, in extract_uid_gid_monitoring 2024-02-20T00:20:02.610 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:20:02 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: uid, gid = extract_uid_gid(ctx, file_path='/etc/prometheus') 2024-02-20T00:20:02.610 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:20:02 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: File "/var/lib/ceph/2a2e6c42-cf84-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid 2024-02-20T00:20:02.610 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:20:02 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: raise RuntimeError('uid/gid not found') 2024-02-20T00:20:02.610 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:20:02 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: RuntimeError: uid/gid not found 2024-02-20T00:20:02.610 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:20:02 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: Traceback (most recent call last): 2024-02-20T00:20:02.610 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:20:02 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1144, in _remote_connection 2024-02-20T00:20:02.610 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:20:02 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: yield (conn, connr) 2024-02-20T00:20:02.611 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:20:02 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1059, in _run_cephadm 2024-02-20T00:20:02.611 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:20:02 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: code, '\n'.join(err))) 2024-02-20T00:20:02.611 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:20:02 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Deploy daemon prometheus.smithi084 ... 2024-02-20T00:20:02.611 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:20:02 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/prometheus:v2.18.1 -e NODE_NAME=smithi084 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/prometheus:v2.18.1 -c %u %g /etc/prometheus 2024-02-20T00:20:02.611 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:20:02 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: stat: stderr Trying to pull docker.io/prom/prometheus:v2.18.1... 2024-02-20T00:20:02.611 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:20:02 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: stat: stderr time="2024-02-20T00:18:08Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-20T00:20:02.611 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:20:02 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: stat: stderr time="2024-02-20T00:18:45Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: copying system image from manifest list: determining manifest MIME type for docker://prom/prometheus:v2.18.1: reading manifest sha256:b4e6cd0275a26750505e539f8528e891053434ebd3972be02645bed5f02f0795 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-20T00:20:02.611 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:20:02 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: stat: stderr time="2024-02-20T00:19:23Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-20T00:20:02.611 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:20:02 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: stat: stderr Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-20T00:20:02.611 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:20:02 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: Traceback (most recent call last): 2024-02-20T00:20:02.611 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:20:02 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: File "/var/lib/ceph/2a2e6c42-cf84-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-02-20T00:20:02.611 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:20:02 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: main() 2024-02-20T00:20:02.612 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:20:02 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: File "/var/lib/ceph/2a2e6c42-cf84-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-02-20T00:20:02.612 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:20:02 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: r = ctx.func(ctx) 2024-02-20T00:20:02.612 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:20:02 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: File "/var/lib/ceph/2a2e6c42-cf84-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-02-20T00:20:02.612 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:20:02 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: return func(ctx) 2024-02-20T00:20:02.612 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:20:02 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: File "/var/lib/ceph/2a2e6c42-cf84-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy 2024-02-20T00:20:02.612 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:20:02 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2024-02-20T00:20:02.612 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:20:02 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: File "/var/lib/ceph/2a2e6c42-cf84-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4185, in extract_uid_gid_monitoring 2024-02-20T00:20:02.612 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:20:02 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: uid, gid = extract_uid_gid(ctx, file_path='/etc/prometheus') 2024-02-20T00:20:02.612 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:20:02 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: File "/var/lib/ceph/2a2e6c42-cf84-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid 2024-02-20T00:20:02.612 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:20:02 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: raise RuntimeError('uid/gid not found') 2024-02-20T00:20:02.612 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:20:02 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: RuntimeError: uid/gid not found 2024-02-20T00:20:02.612 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:20:02 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:20:00.931840+0000 mgr.smithi084.bmqrnf (mgr.14162) 311 : cluster [DBG] pgmap v296: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:20:02.612 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:20:02 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:20:01.274603+0000 mon.smithi084 (mon.0) 300 : audit [INF] from='mgr.14162 172.21.15.84:0/2152933216' entity='mgr.smithi084.bmqrnf' 2024-02-20T00:20:02.612 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:20:02 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:20:01.512869+0000 mon.smithi084 (mon.0) 301 : audit [DBG] from='client.? 172.21.15.132:0/2698842540' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:20:04.108 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:20:03 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:20:02.721828+0000 mon.smithi084 (mon.0) 302 : audit [INF] from='mgr.14162 172.21.15.84:0/2152933216' entity='mgr.smithi084.bmqrnf' 2024-02-20T00:20:04.108 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:20:03 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:20:02.932071+0000 mgr.smithi084.bmqrnf (mgr.14162) 312 : cluster [DBG] pgmap v297: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:20:04.798 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:20:04.798 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:20:06.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:20:05 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:20:04.875604+0000 mon.smithi084 (mon.0) 303 : audit [INF] from='mgr.14162 172.21.15.84:0/2152933216' entity='mgr.smithi084.bmqrnf' 2024-02-20T00:20:06.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:20:05 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:20:04.932332+0000 mgr.smithi084.bmqrnf (mgr.14162) 313 : cluster [DBG] pgmap v298: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:20:06.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:20:05 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:20:05.468075+0000 mon.smithi084 (mon.0) 304 : audit [INF] from='mgr.14162 172.21.15.84:0/2152933216' entity='mgr.smithi084.bmqrnf' 2024-02-20T00:20:06.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:20:05 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:20:05.473852+0000 mon.smithi084 (mon.0) 305 : audit [INF] from='mgr.14162 172.21.15.84:0/2152933216' entity='mgr.smithi084.bmqrnf' 2024-02-20T00:20:06.742 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:06.714+0000 7f0c7bfff700 1 -- 172.21.15.132:0/1621653406 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0c7c0f62a0 msgr2=0x7f0c7c0f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:20:06.742 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:06.714+0000 7f0c7bfff700 1 --2- 172.21.15.132:0/1621653406 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0c7c0f62a0 0x7f0c7c0f66c0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 rx=0x7f0c70009fb0 tx=0x7f0c70009b30).stop 2024-02-20T00:20:06.742 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:06.714+0000 7f0c7bfff700 1 -- 172.21.15.132:0/1621653406 shutdown_connections 2024-02-20T00:20:06.742 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:06.714+0000 7f0c7bfff700 1 --2- 172.21.15.132:0/1621653406 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0c7c0f62a0 0x7f0c7c0f66c0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:20:06.742 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:06.714+0000 7f0c7bfff700 1 -- 172.21.15.132:0/1621653406 >> 172.21.15.132:0/1621653406 conn(0x7f0c7c0f1aa0 msgr2=0x7f0c7c0f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:20:06.742 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:06.714+0000 7f0c7bfff700 1 -- 172.21.15.132:0/1621653406 shutdown_connections 2024-02-20T00:20:06.742 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:06.714+0000 7f0c7bfff700 1 -- 172.21.15.132:0/1621653406 wait complete. 2024-02-20T00:20:06.743 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:06.714+0000 7f0c7bfff700 1 Processor -- start 2024-02-20T00:20:06.743 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:06.714+0000 7f0c7bfff700 1 -- start start 2024-02-20T00:20:06.743 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:06.715+0000 7f0c7bfff700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0c7c0f62a0 0x7f0c7c1065d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:20:06.743 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:06.715+0000 7f0c7bfff700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c7c106ac0 con 0x7f0c7c0f62a0 2024-02-20T00:20:06.743 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:06.715+0000 7f0c7affd700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0c7c0f62a0 0x7f0c7c1065d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:20:06.743 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:06.715+0000 7f0c7affd700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0c7c0f62a0 0x7f0c7c1065d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:46680/0 (socket says 172.21.15.132:46680) 2024-02-20T00:20:06.743 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:06.715+0000 7f0c7affd700 1 -- 172.21.15.132:0/2281673606 learned_addr learned my addr 172.21.15.132:0/2281673606 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:20:06.743 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:06.715+0000 7f0c7affd700 1 -- 172.21.15.132:0/2281673606 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c70005040 con 0x7f0c7c0f62a0 2024-02-20T00:20:06.743 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:06.715+0000 7f0c7affd700 1 --2- 172.21.15.132:0/2281673606 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0c7c0f62a0 0x7f0c7c1065d0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0x7f0c700089f0 tx=0x7f0c7000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:20:06.743 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:06.715+0000 7f0c637fe700 1 -- 172.21.15.132:0/2281673606 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0c70009320 con 0x7f0c7c0f62a0 2024-02-20T00:20:06.743 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:06.715+0000 7f0c7bfff700 1 -- 172.21.15.132:0/2281673606 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c7c106cc0 con 0x7f0c7c0f62a0 2024-02-20T00:20:06.744 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:06.715+0000 7f0c637fe700 1 -- 172.21.15.132:0/2281673606 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0c70013380 con 0x7f0c7c0f62a0 2024-02-20T00:20:06.744 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:06.715+0000 7f0c637fe700 1 -- 172.21.15.132:0/2281673606 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0c70014e20 con 0x7f0c7c0f62a0 2024-02-20T00:20:06.744 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:06.715+0000 7f0c7bfff700 1 -- 172.21.15.132:0/2281673606 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c7c107150 con 0x7f0c7c0f62a0 2024-02-20T00:20:06.744 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:06.716+0000 7f0c637fe700 1 -- 172.21.15.132:0/2281673606 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0c700263a0 con 0x7f0c7c0f62a0 2024-02-20T00:20:06.744 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:06.716+0000 7f0c637fe700 1 --2- 172.21.15.132:0/2281673606 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f0c6402ff10 0x7f0c640323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:20:06.744 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:06.716+0000 7f0c637fe700 1 -- 172.21.15.132:0/2281673606 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0c70040f30 con 0x7f0c7c0f62a0 2024-02-20T00:20:06.744 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:06.716+0000 7f0c7bfff700 1 -- 172.21.15.132:0/2281673606 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c7c045ad0 con 0x7f0c7c0f62a0 2024-02-20T00:20:06.744 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:06.717+0000 7f0c7a7fc700 1 --2- 172.21.15.132:0/2281673606 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f0c6402ff10 0x7f0c640323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:20:06.744 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:06.717+0000 7f0c7a7fc700 1 --2- 172.21.15.132:0/2281673606 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f0c6402ff10 0x7f0c640323d0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f0c6c0096a0 tx=0x7f0c6c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:20:06.744 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:06.721+0000 7f0c637fe700 1 -- 172.21.15.132:0/2281673606 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0c70012920 con 0x7f0c7c0f62a0 2024-02-20T00:20:06.929 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:06.927+0000 7f0c7bfff700 1 -- 172.21.15.132:0/2281673606 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0c7c107e70 con 0x7f0c7c0f62a0 2024-02-20T00:20:06.929 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:06.927+0000 7f0c637fe700 1 -- 172.21.15.132:0/2281673606 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0c7002b020 con 0x7f0c7c0f62a0 2024-02-20T00:20:06.930 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:20:06.930 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:20:06.931 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:06.929+0000 7f0c7bfff700 1 -- 172.21.15.132:0/2281673606 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f0c6402ff10 msgr2=0x7f0c640323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:20:06.931 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:06.929+0000 7f0c7bfff700 1 --2- 172.21.15.132:0/2281673606 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f0c6402ff10 0x7f0c640323d0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f0c6c0096a0 tx=0x7f0c6c006b40).stop 2024-02-20T00:20:06.931 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:06.929+0000 7f0c7bfff700 1 -- 172.21.15.132:0/2281673606 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0c7c0f62a0 msgr2=0x7f0c7c1065d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:20:06.932 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:06.929+0000 7f0c7bfff700 1 --2- 172.21.15.132:0/2281673606 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0c7c0f62a0 0x7f0c7c1065d0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0x7f0c700089f0 tx=0x7f0c7000b040).stop 2024-02-20T00:20:06.932 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:06.929+0000 7f0c7bfff700 1 -- 172.21.15.132:0/2281673606 shutdown_connections 2024-02-20T00:20:06.932 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:06.929+0000 7f0c7bfff700 1 --2- 172.21.15.132:0/2281673606 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f0c6402ff10 0x7f0c640323d0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:20:06.932 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:06.929+0000 7f0c7bfff700 1 --2- 172.21.15.132:0/2281673606 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0c7c0f62a0 0x7f0c7c1065d0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:20:06.932 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:06.929+0000 7f0c7bfff700 1 -- 172.21.15.132:0/2281673606 >> 172.21.15.132:0/2281673606 conn(0x7f0c7c0f1aa0 msgr2=0x7f0c7c0f2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:20:06.932 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:06.929+0000 7f0c7bfff700 1 -- 172.21.15.132:0/2281673606 shutdown_connections 2024-02-20T00:20:06.932 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:06.929+0000 7f0c7bfff700 1 -- 172.21.15.132:0/2281673606 wait complete. 2024-02-20T00:20:06.932 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:20:07.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:20:06 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:20:05.468930+0000 mgr.smithi084.bmqrnf (mgr.14162) 314 : cluster [DBG] pgmap v299: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:20:07.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:20:06 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cephadm 2024-02-20T00:20:05.475856+0000 mgr.smithi084.bmqrnf (mgr.14162) 315 : cephadm [INF] Deploying daemon alertmanager.smithi132 on smithi132 2024-02-20T00:20:07.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:20:06 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:20:05.874016+0000 mon.smithi084 (mon.0) 306 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-02-20T00:20:08.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:20:07 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:20:06.928916+0000 mon.smithi084 (mon.0) 307 : audit [DBG] from='client.? 172.21.15.132:0/2281673606' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:20:08.708 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:20:08.708 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:20:09.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:20:08 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:20:07.469288+0000 mgr.smithi084.bmqrnf (mgr.14162) 316 : cluster [DBG] pgmap v300: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:20:10.684 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:10.681+0000 7fe076579700 1 -- 172.21.15.132:0/662913658 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe0700f5410 msgr2=0x7fe0700f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:20:10.684 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:10.681+0000 7fe076579700 1 --2- 172.21.15.132:0/662913658 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe0700f5410 0x7fe0700f5830 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 rx=0x7fe060009fb0 tx=0x7fe060009b30).stop 2024-02-20T00:20:10.684 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:10.681+0000 7fe076579700 1 -- 172.21.15.132:0/662913658 shutdown_connections 2024-02-20T00:20:10.684 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:10.681+0000 7fe076579700 1 --2- 172.21.15.132:0/662913658 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe0700f5410 0x7fe0700f5830 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:20:10.684 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:10.681+0000 7fe076579700 1 -- 172.21.15.132:0/662913658 >> 172.21.15.132:0/662913658 conn(0x7fe0700f0b70 msgr2=0x7fe0700f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:20:10.684 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:10.681+0000 7fe076579700 1 -- 172.21.15.132:0/662913658 shutdown_connections 2024-02-20T00:20:10.684 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:10.681+0000 7fe076579700 1 -- 172.21.15.132:0/662913658 wait complete. 2024-02-20T00:20:10.684 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:10.681+0000 7fe076579700 1 Processor -- start 2024-02-20T00:20:10.684 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:10.681+0000 7fe076579700 1 -- start start 2024-02-20T00:20:10.684 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:10.681+0000 7fe076579700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe0700f5410 0x7fe070109500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:20:10.684 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:10.681+0000 7fe076579700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe070060220 con 0x7fe0700f5410 2024-02-20T00:20:10.686 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:10.682+0000 7fe06ffff700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe0700f5410 0x7fe070109500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:20:10.686 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:10.682+0000 7fe06ffff700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe0700f5410 0x7fe070109500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:36874/0 (socket says 172.21.15.132:36874) 2024-02-20T00:20:10.686 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:10.682+0000 7fe06ffff700 1 -- 172.21.15.132:0/682274629 learned_addr learned my addr 172.21.15.132:0/682274629 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:20:10.686 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:10.682+0000 7fe06ffff700 1 -- 172.21.15.132:0/682274629 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe060005040 con 0x7fe0700f5410 2024-02-20T00:20:10.686 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:10.682+0000 7fe06ffff700 1 --2- 172.21.15.132:0/682274629 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe0700f5410 0x7fe070109500 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 rx=0x7fe060005010 tx=0x7fe06000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:20:10.687 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:10.682+0000 7fe06cff9700 1 -- 172.21.15.132:0/682274629 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe060009320 con 0x7fe0700f5410 2024-02-20T00:20:10.687 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:10.682+0000 7fe076579700 1 -- 172.21.15.132:0/682274629 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe070060420 con 0x7fe0700f5410 2024-02-20T00:20:10.687 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:10.682+0000 7fe06cff9700 1 -- 172.21.15.132:0/682274629 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe060013380 con 0x7fe0700f5410 2024-02-20T00:20:10.687 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:10.682+0000 7fe06cff9700 1 -- 172.21.15.132:0/682274629 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe060014e50 con 0x7fe0700f5410 2024-02-20T00:20:10.687 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:10.682+0000 7fe076579700 1 -- 172.21.15.132:0/682274629 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe070109bb0 con 0x7fe0700f5410 2024-02-20T00:20:10.687 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:10.683+0000 7fe06cff9700 1 -- 172.21.15.132:0/682274629 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe0600263a0 con 0x7fe0700f5410 2024-02-20T00:20:10.687 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:10.683+0000 7fe06cff9700 1 --2- 172.21.15.132:0/682274629 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fe05802ff10 0x7fe0580323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:20:10.687 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:10.683+0000 7fe06cff9700 1 -- 172.21.15.132:0/682274629 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe060040f30 con 0x7fe0700f5410 2024-02-20T00:20:10.687 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:10.683+0000 7fe076579700 1 -- 172.21.15.132:0/682274629 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe070059100 con 0x7fe0700f5410 2024-02-20T00:20:10.687 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:10.683+0000 7fe06f7fe700 1 --2- 172.21.15.132:0/682274629 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fe05802ff10 0x7fe0580323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:20:10.687 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:10.684+0000 7fe06f7fe700 1 --2- 172.21.15.132:0/682274629 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fe05802ff10 0x7fe0580323d0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7fe0640096a0 tx=0x7fe064006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:20:10.690 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:10.688+0000 7fe06cff9700 1 -- 172.21.15.132:0/682274629 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe060012920 con 0x7fe0700f5410 2024-02-20T00:20:10.896 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:10.893+0000 7fe076579700 1 -- 172.21.15.132:0/682274629 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe070045ad0 con 0x7fe0700f5410 2024-02-20T00:20:10.896 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:10.894+0000 7fe06cff9700 1 -- 172.21.15.132:0/682274629 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe070045ad0 con 0x7fe0700f5410 2024-02-20T00:20:10.897 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:20:10.897 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:20:10.898 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:10.896+0000 7fe076579700 1 -- 172.21.15.132:0/682274629 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fe05802ff10 msgr2=0x7fe0580323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:20:10.898 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:10.896+0000 7fe076579700 1 --2- 172.21.15.132:0/682274629 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fe05802ff10 0x7fe0580323d0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7fe0640096a0 tx=0x7fe064006b40).stop 2024-02-20T00:20:10.898 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:10.896+0000 7fe076579700 1 -- 172.21.15.132:0/682274629 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe0700f5410 msgr2=0x7fe070109500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:20:10.898 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:10.896+0000 7fe076579700 1 --2- 172.21.15.132:0/682274629 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe0700f5410 0x7fe070109500 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 rx=0x7fe060005010 tx=0x7fe06000b040).stop 2024-02-20T00:20:10.898 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:10.896+0000 7fe076579700 1 -- 172.21.15.132:0/682274629 shutdown_connections 2024-02-20T00:20:10.899 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:10.896+0000 7fe076579700 1 --2- 172.21.15.132:0/682274629 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fe05802ff10 0x7fe0580323d0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:20:10.899 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:10.896+0000 7fe076579700 1 --2- 172.21.15.132:0/682274629 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe0700f5410 0x7fe070109500 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:20:10.899 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:10.896+0000 7fe076579700 1 -- 172.21.15.132:0/682274629 >> 172.21.15.132:0/682274629 conn(0x7fe0700f0b70 msgr2=0x7fe0700f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:20:10.899 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:10.896+0000 7fe076579700 1 -- 172.21.15.132:0/682274629 shutdown_connections 2024-02-20T00:20:10.899 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:10.896+0000 7fe076579700 1 -- 172.21.15.132:0/682274629 wait complete. 2024-02-20T00:20:10.899 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:20:11.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:20:10 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:20:09.469724+0000 mgr.smithi084.bmqrnf (mgr.14162) 317 : cluster [DBG] pgmap v301: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:20:12.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:20:11 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:20:10.895765+0000 mon.smithi084 (mon.0) 308 : audit [DBG] from='client.? 172.21.15.132:0/682274629' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:20:12.609 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:20:12.610 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:20:13.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:20:12 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:20:11.470046+0000 mgr.smithi084.bmqrnf (mgr.14162) 318 : cluster [DBG] pgmap v302: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:20:14.417 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:14.412+0000 7fead5d72700 1 -- 172.21.15.132:0/999618670 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fead0060220 msgr2=0x7fead00f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:20:14.417 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:14.412+0000 7fead5d72700 1 --2- 172.21.15.132:0/999618670 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fead0060220 0x7fead00f9150 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 rx=0x7feac4009fb0 tx=0x7feac4009b30).stop 2024-02-20T00:20:14.417 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:14.412+0000 7fead5d72700 1 -- 172.21.15.132:0/999618670 shutdown_connections 2024-02-20T00:20:14.417 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:14.412+0000 7fead5d72700 1 --2- 172.21.15.132:0/999618670 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fead0060220 0x7fead00f9150 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:20:14.417 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:14.412+0000 7fead5d72700 1 -- 172.21.15.132:0/999618670 >> 172.21.15.132:0/999618670 conn(0x7fead00f0b70 msgr2=0x7fead00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:20:14.417 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:14.412+0000 7fead5d72700 1 -- 172.21.15.132:0/999618670 shutdown_connections 2024-02-20T00:20:14.417 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:14.412+0000 7fead5d72700 1 -- 172.21.15.132:0/999618670 wait complete. 2024-02-20T00:20:14.417 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:14.413+0000 7fead5d72700 1 Processor -- start 2024-02-20T00:20:14.417 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:14.413+0000 7fead5d72700 1 -- start start 2024-02-20T00:20:14.417 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:14.413+0000 7fead5d72700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fead0060220 0x7fead01067a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:20:14.417 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:14.413+0000 7fead5d72700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fead0106c90 con 0x7fead0060220 2024-02-20T00:20:14.417 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:14.413+0000 7feacf7fe700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fead0060220 0x7fead01067a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:20:14.418 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:14.413+0000 7feacf7fe700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fead0060220 0x7fead01067a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:36884/0 (socket says 172.21.15.132:36884) 2024-02-20T00:20:14.418 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:14.413+0000 7feacf7fe700 1 -- 172.21.15.132:0/3519824673 learned_addr learned my addr 172.21.15.132:0/3519824673 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:20:14.418 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:14.414+0000 7feacf7fe700 1 -- 172.21.15.132:0/3519824673 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feac4005040 con 0x7fead0060220 2024-02-20T00:20:14.418 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:14.414+0000 7feacf7fe700 1 --2- 172.21.15.132:0/3519824673 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fead0060220 0x7fead01067a0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0x7feac40089f0 tx=0x7feac400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:20:14.418 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:14.414+0000 7feab7fff700 1 -- 172.21.15.132:0/3519824673 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feac4009320 con 0x7fead0060220 2024-02-20T00:20:14.418 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:14.414+0000 7fead5d72700 1 -- 172.21.15.132:0/3519824673 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fead0106e90 con 0x7fead0060220 2024-02-20T00:20:14.418 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:14.414+0000 7feab7fff700 1 -- 172.21.15.132:0/3519824673 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7feac4013380 con 0x7fead0060220 2024-02-20T00:20:14.418 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:14.414+0000 7feab7fff700 1 -- 172.21.15.132:0/3519824673 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feac4014e20 con 0x7fead0060220 2024-02-20T00:20:14.418 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:14.414+0000 7fead5d72700 1 -- 172.21.15.132:0/3519824673 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fead0107320 con 0x7fead0060220 2024-02-20T00:20:14.419 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:14.415+0000 7feab7fff700 1 -- 172.21.15.132:0/3519824673 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7feac4014420 con 0x7fead0060220 2024-02-20T00:20:14.419 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:14.415+0000 7fead5d72700 1 -- 172.21.15.132:0/3519824673 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fead0045ad0 con 0x7fead0060220 2024-02-20T00:20:14.419 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:14.415+0000 7feab7fff700 1 --2- 172.21.15.132:0/3519824673 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7feab802ff10 0x7feab80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:20:14.419 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:14.415+0000 7feab7fff700 1 -- 172.21.15.132:0/3519824673 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7feac4040f00 con 0x7fead0060220 2024-02-20T00:20:14.419 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:14.415+0000 7feaceffd700 1 --2- 172.21.15.132:0/3519824673 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7feab802ff10 0x7feab80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:20:14.419 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:14.416+0000 7feaceffd700 1 --2- 172.21.15.132:0/3519824673 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7feab802ff10 0x7feab80323d0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7feac00096a0 tx=0x7feac0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:20:14.422 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:14.420+0000 7feab7fff700 1 -- 172.21.15.132:0/3519824673 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7feac40095f0 con 0x7fead0060220 2024-02-20T00:20:14.627 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:14.624+0000 7fead5d72700 1 -- 172.21.15.132:0/3519824673 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fead01081d0 con 0x7fead0060220 2024-02-20T00:20:14.627 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:14.625+0000 7feab7fff700 1 -- 172.21.15.132:0/3519824673 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7feac401db70 con 0x7fead0060220 2024-02-20T00:20:14.627 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:20:14.627 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:20:14.629 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:14.626+0000 7fead5d72700 1 -- 172.21.15.132:0/3519824673 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7feab802ff10 msgr2=0x7feab80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:20:14.629 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:14.626+0000 7fead5d72700 1 --2- 172.21.15.132:0/3519824673 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7feab802ff10 0x7feab80323d0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7feac00096a0 tx=0x7feac0006b40).stop 2024-02-20T00:20:14.629 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:14.626+0000 7fead5d72700 1 -- 172.21.15.132:0/3519824673 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fead0060220 msgr2=0x7fead01067a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:20:14.629 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:14.626+0000 7fead5d72700 1 --2- 172.21.15.132:0/3519824673 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fead0060220 0x7fead01067a0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0x7feac40089f0 tx=0x7feac400b040).stop 2024-02-20T00:20:14.629 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:14.626+0000 7fead5d72700 1 -- 172.21.15.132:0/3519824673 shutdown_connections 2024-02-20T00:20:14.629 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:14.626+0000 7fead5d72700 1 --2- 172.21.15.132:0/3519824673 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7feab802ff10 0x7feab80323d0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:20:14.630 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:14.626+0000 7fead5d72700 1 --2- 172.21.15.132:0/3519824673 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fead0060220 0x7fead01067a0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:20:14.630 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:14.626+0000 7fead5d72700 1 -- 172.21.15.132:0/3519824673 >> 172.21.15.132:0/3519824673 conn(0x7fead00f0b70 msgr2=0x7fead00f1600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:20:14.630 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:14.627+0000 7fead5d72700 1 -- 172.21.15.132:0/3519824673 shutdown_connections 2024-02-20T00:20:14.630 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:14.627+0000 7fead5d72700 1 -- 172.21.15.132:0/3519824673 wait complete. 2024-02-20T00:20:14.630 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:20:15.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:20:14 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:20:13.470448+0000 mgr.smithi084.bmqrnf (mgr.14162) 319 : cluster [DBG] pgmap v303: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:20:15.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:20:14 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:20:14.626359+0000 mon.smithi084 (mon.0) 309 : audit [DBG] from='client.? 172.21.15.132:0/3519824673' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:20:16.379 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:20:16.379 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:20:17.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:20:16 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:20:15.470781+0000 mgr.smithi084.bmqrnf (mgr.14162) 320 : cluster [DBG] pgmap v304: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:20:18.331 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:18.308+0000 7f2e2af71700 1 -- 172.21.15.132:0/1958595789 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2e240f6510 msgr2=0x7f2e240f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:20:18.331 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:18.308+0000 7f2e2af71700 1 --2- 172.21.15.132:0/1958595789 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2e240f6510 0x7f2e240f6930 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 rx=0x7f2e18009fb0 tx=0x7f2e18009b30).stop 2024-02-20T00:20:18.331 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:18.308+0000 7f2e2af71700 1 -- 172.21.15.132:0/1958595789 shutdown_connections 2024-02-20T00:20:18.331 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:18.308+0000 7f2e2af71700 1 --2- 172.21.15.132:0/1958595789 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2e240f6510 0x7f2e240f6930 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:20:18.331 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:18.308+0000 7f2e2af71700 1 -- 172.21.15.132:0/1958595789 >> 172.21.15.132:0/1958595789 conn(0x7f2e240f1c70 msgr2=0x7f2e240f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:20:18.331 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:18.308+0000 7f2e2af71700 1 -- 172.21.15.132:0/1958595789 shutdown_connections 2024-02-20T00:20:18.331 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:18.308+0000 7f2e2af71700 1 -- 172.21.15.132:0/1958595789 wait complete. 2024-02-20T00:20:18.331 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:18.308+0000 7f2e2af71700 1 Processor -- start 2024-02-20T00:20:18.331 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:18.308+0000 7f2e2af71700 1 -- start start 2024-02-20T00:20:18.331 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:18.309+0000 7f2e2af71700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2e240f6510 0x7f2e241067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:20:18.332 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:18.309+0000 7f2e2af71700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e24106cd0 con 0x7f2e240f6510 2024-02-20T00:20:18.332 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:18.309+0000 7f2e28d0d700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2e240f6510 0x7f2e241067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:20:18.332 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:18.309+0000 7f2e28d0d700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2e240f6510 0x7f2e241067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:43436/0 (socket says 172.21.15.132:43436) 2024-02-20T00:20:18.332 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:18.309+0000 7f2e28d0d700 1 -- 172.21.15.132:0/4049127178 learned_addr learned my addr 172.21.15.132:0/4049127178 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:20:18.332 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:18.309+0000 7f2e28d0d700 1 -- 172.21.15.132:0/4049127178 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e18005040 con 0x7f2e240f6510 2024-02-20T00:20:18.332 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:18.309+0000 7f2e28d0d700 1 --2- 172.21.15.132:0/4049127178 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2e240f6510 0x7f2e241067e0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 rx=0x7f2e18005010 tx=0x7f2e1800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:20:18.332 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:18.310+0000 7f2e217fa700 1 -- 172.21.15.132:0/4049127178 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2e18009320 con 0x7f2e240f6510 2024-02-20T00:20:18.332 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:18.310+0000 7f2e2af71700 1 -- 172.21.15.132:0/4049127178 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e24106ed0 con 0x7f2e240f6510 2024-02-20T00:20:18.332 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:18.310+0000 7f2e217fa700 1 -- 172.21.15.132:0/4049127178 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2e18013380 con 0x7f2e240f6510 2024-02-20T00:20:18.332 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:18.310+0000 7f2e217fa700 1 -- 172.21.15.132:0/4049127178 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2e18014e50 con 0x7f2e240f6510 2024-02-20T00:20:18.332 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:18.310+0000 7f2e2af71700 1 -- 172.21.15.132:0/4049127178 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e24107360 con 0x7f2e240f6510 2024-02-20T00:20:18.332 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:18.310+0000 7f2e217fa700 1 -- 172.21.15.132:0/4049127178 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2e18014420 con 0x7f2e240f6510 2024-02-20T00:20:18.332 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:18.311+0000 7f2e2af71700 1 -- 172.21.15.132:0/4049127178 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e24045ad0 con 0x7f2e240f6510 2024-02-20T00:20:18.333 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:18.311+0000 7f2e217fa700 1 --2- 172.21.15.132:0/4049127178 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f2e1402ff10 0x7f2e140323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:20:18.333 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:18.311+0000 7f2e217fa700 1 -- 172.21.15.132:0/4049127178 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2e18040e30 con 0x7f2e240f6510 2024-02-20T00:20:18.333 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:18.311+0000 7f2e23fff700 1 --2- 172.21.15.132:0/4049127178 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f2e1402ff10 0x7f2e140323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:20:18.333 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:18.311+0000 7f2e23fff700 1 --2- 172.21.15.132:0/4049127178 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f2e1402ff10 0x7f2e140323d0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f2e100096a0 tx=0x7f2e10006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:20:18.333 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:18.315+0000 7f2e217fa700 1 -- 172.21.15.132:0/4049127178 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2e180095f0 con 0x7f2e240f6510 2024-02-20T00:20:18.521 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:18.519+0000 7f2e2af71700 1 -- 172.21.15.132:0/4049127178 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2e24108210 con 0x7f2e240f6510 2024-02-20T00:20:18.522 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:18.519+0000 7f2e217fa700 1 -- 172.21.15.132:0/4049127178 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2e1801db70 con 0x7f2e240f6510 2024-02-20T00:20:18.522 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:20:18.522 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:20:18.523 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:18.521+0000 7f2e2af71700 1 -- 172.21.15.132:0/4049127178 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f2e1402ff10 msgr2=0x7f2e140323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:20:18.523 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:18.521+0000 7f2e2af71700 1 --2- 172.21.15.132:0/4049127178 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f2e1402ff10 0x7f2e140323d0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f2e100096a0 tx=0x7f2e10006b40).stop 2024-02-20T00:20:18.523 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:18.521+0000 7f2e2af71700 1 -- 172.21.15.132:0/4049127178 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2e240f6510 msgr2=0x7f2e241067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:20:18.523 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:18.521+0000 7f2e2af71700 1 --2- 172.21.15.132:0/4049127178 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2e240f6510 0x7f2e241067e0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 rx=0x7f2e18005010 tx=0x7f2e1800b040).stop 2024-02-20T00:20:18.524 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:18.521+0000 7f2e2af71700 1 -- 172.21.15.132:0/4049127178 shutdown_connections 2024-02-20T00:20:18.524 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:18.521+0000 7f2e2af71700 1 --2- 172.21.15.132:0/4049127178 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f2e1402ff10 0x7f2e140323d0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:20:18.524 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:18.521+0000 7f2e2af71700 1 --2- 172.21.15.132:0/4049127178 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2e240f6510 0x7f2e241067e0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:20:18.524 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:18.521+0000 7f2e2af71700 1 -- 172.21.15.132:0/4049127178 >> 172.21.15.132:0/4049127178 conn(0x7f2e240f1c70 msgr2=0x7f2e240f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:20:18.524 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:18.521+0000 7f2e2af71700 1 -- 172.21.15.132:0/4049127178 shutdown_connections 2024-02-20T00:20:18.524 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:18.521+0000 7f2e2af71700 1 -- 172.21.15.132:0/4049127178 wait complete. 2024-02-20T00:20:18.524 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:20:19.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:20:18 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:20:17.471166+0000 mgr.smithi084.bmqrnf (mgr.14162) 321 : cluster [DBG] pgmap v305: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:20:19.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:20:18 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:20:18.521127+0000 mon.smithi084 (mon.0) 310 : audit [DBG] from='client.? 172.21.15.132:0/4049127178' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:20:20.124 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:20:20.124 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:20:21.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:20:20 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:20:19.471561+0000 mgr.smithi084.bmqrnf (mgr.14162) 322 : cluster [DBG] pgmap v306: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:20:22.022 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:22.015+0000 7f91a84df700 1 -- 172.21.15.132:0/4197184894 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f91a00f5410 msgr2=0x7f91a00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:20:22.022 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:22.015+0000 7f91a84df700 1 --2- 172.21.15.132:0/4197184894 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f91a00f5410 0x7f91a00f5830 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 rx=0x7f9194009fb0 tx=0x7f9194009b30).stop 2024-02-20T00:20:22.022 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:22.016+0000 7f91a84df700 1 -- 172.21.15.132:0/4197184894 shutdown_connections 2024-02-20T00:20:22.022 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:22.016+0000 7f91a84df700 1 --2- 172.21.15.132:0/4197184894 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f91a00f5410 0x7f91a00f5830 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:20:22.022 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:22.016+0000 7f91a84df700 1 -- 172.21.15.132:0/4197184894 >> 172.21.15.132:0/4197184894 conn(0x7f91a00f0b70 msgr2=0x7f91a00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:20:22.022 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:22.016+0000 7f91a84df700 1 -- 172.21.15.132:0/4197184894 shutdown_connections 2024-02-20T00:20:22.022 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:22.016+0000 7f91a84df700 1 -- 172.21.15.132:0/4197184894 wait complete. 2024-02-20T00:20:22.022 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:22.016+0000 7f91a84df700 1 Processor -- start 2024-02-20T00:20:22.022 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:22.016+0000 7f91a84df700 1 -- start start 2024-02-20T00:20:22.022 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:22.016+0000 7f91a84df700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f91a00f5410 0x7f91a01056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:20:22.022 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:22.016+0000 7f91a84df700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91a0105bd0 con 0x7f91a00f5410 2024-02-20T00:20:22.023 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:22.016+0000 7f91a627b700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f91a00f5410 0x7f91a01056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:20:22.023 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:22.017+0000 7f91a627b700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f91a00f5410 0x7f91a01056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:43454/0 (socket says 172.21.15.132:43454) 2024-02-20T00:20:22.023 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:22.017+0000 7f91a627b700 1 -- 172.21.15.132:0/3884618721 learned_addr learned my addr 172.21.15.132:0/3884618721 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:20:22.023 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:22.017+0000 7f91a627b700 1 -- 172.21.15.132:0/3884618721 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9194005040 con 0x7f91a00f5410 2024-02-20T00:20:22.023 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:22.017+0000 7f91a627b700 1 --2- 172.21.15.132:0/3884618721 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f91a00f5410 0x7f91a01056e0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0x7f9194005010 tx=0x7f919400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:20:22.023 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:22.017+0000 7f9192ffd700 1 -- 172.21.15.132:0/3884618721 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9194009320 con 0x7f91a00f5410 2024-02-20T00:20:22.023 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:22.017+0000 7f91a84df700 1 -- 172.21.15.132:0/3884618721 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91a0105dd0 con 0x7f91a00f5410 2024-02-20T00:20:22.023 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:22.017+0000 7f9192ffd700 1 -- 172.21.15.132:0/3884618721 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9194013380 con 0x7f91a00f5410 2024-02-20T00:20:22.023 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:22.017+0000 7f9192ffd700 1 -- 172.21.15.132:0/3884618721 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9194014e50 con 0x7f91a00f5410 2024-02-20T00:20:22.023 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:22.017+0000 7f91a84df700 1 -- 172.21.15.132:0/3884618721 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91a0106260 con 0x7f91a00f5410 2024-02-20T00:20:22.023 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:22.018+0000 7f9192ffd700 1 -- 172.21.15.132:0/3884618721 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f91940095f0 con 0x7f91a00f5410 2024-02-20T00:20:22.023 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:22.018+0000 7f91a84df700 1 -- 172.21.15.132:0/3884618721 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91a0045ad0 con 0x7f91a00f5410 2024-02-20T00:20:22.023 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:22.018+0000 7f9192ffd700 1 --2- 172.21.15.132:0/3884618721 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f918c02ff10 0x7f918c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:20:22.024 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:22.018+0000 7f9192ffd700 1 -- 172.21.15.132:0/3884618721 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9194040f80 con 0x7f91a00f5410 2024-02-20T00:20:22.024 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:22.018+0000 7f91a5a7a700 1 --2- 172.21.15.132:0/3884618721 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f918c02ff10 0x7f918c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:20:22.024 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:22.019+0000 7f91a5a7a700 1 --2- 172.21.15.132:0/3884618721 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f918c02ff10 0x7f918c0323d0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f919c0096a0 tx=0x7f919c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:20:22.025 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:22.023+0000 7f9192ffd700 1 -- 172.21.15.132:0/3884618721 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9194014420 con 0x7f91a00f5410 2024-02-20T00:20:22.231 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:22.229+0000 7f91a84df700 1 -- 172.21.15.132:0/3884618721 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f91a01070b0 con 0x7f91a00f5410 2024-02-20T00:20:22.232 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:22.231+0000 7f9192ffd700 1 -- 172.21.15.132:0/3884618721 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9194010230 con 0x7f91a00f5410 2024-02-20T00:20:22.232 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:20:22.232 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:20:22.234 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:22.231+0000 7f91a84df700 1 -- 172.21.15.132:0/3884618721 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f918c02ff10 msgr2=0x7f918c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:20:22.234 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:22.231+0000 7f91a84df700 1 --2- 172.21.15.132:0/3884618721 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f918c02ff10 0x7f918c0323d0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f919c0096a0 tx=0x7f919c006b40).stop 2024-02-20T00:20:22.234 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:22.231+0000 7f91a84df700 1 -- 172.21.15.132:0/3884618721 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f91a00f5410 msgr2=0x7f91a01056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:20:22.234 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:22.231+0000 7f91a84df700 1 --2- 172.21.15.132:0/3884618721 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f91a00f5410 0x7f91a01056e0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0x7f9194005010 tx=0x7f919400b040).stop 2024-02-20T00:20:22.234 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:22.231+0000 7f91a84df700 1 -- 172.21.15.132:0/3884618721 shutdown_connections 2024-02-20T00:20:22.234 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:22.231+0000 7f91a84df700 1 --2- 172.21.15.132:0/3884618721 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f918c02ff10 0x7f918c0323d0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:20:22.234 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:22.231+0000 7f91a84df700 1 --2- 172.21.15.132:0/3884618721 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f91a00f5410 0x7f91a01056e0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:20:22.234 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:22.231+0000 7f91a84df700 1 -- 172.21.15.132:0/3884618721 >> 172.21.15.132:0/3884618721 conn(0x7f91a00f0b70 msgr2=0x7f91a00f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:20:22.234 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:22.231+0000 7f91a84df700 1 -- 172.21.15.132:0/3884618721 shutdown_connections 2024-02-20T00:20:22.235 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:22.231+0000 7f91a84df700 1 -- 172.21.15.132:0/3884618721 wait complete. 2024-02-20T00:20:22.235 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:20:23.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:20:22 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:20:21.471946+0000 mgr.smithi084.bmqrnf (mgr.14162) 323 : cluster [DBG] pgmap v307: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:20:23.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:20:22 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:20:22.231305+0000 mon.smithi084 (mon.0) 311 : audit [DBG] from='client.? 172.21.15.132:0/3884618721' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:20:23.802 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:20:23.802 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:20:25.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:20:24 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:20:23.472377+0000 mgr.smithi084.bmqrnf (mgr.14162) 324 : cluster [DBG] pgmap v308: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:20:25.605 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:25.602+0000 7fae119e0700 1 -- 172.21.15.132:0/1823741053 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fae0c0f6510 msgr2=0x7fae0c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:20:25.605 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:25.602+0000 7fae119e0700 1 --2- 172.21.15.132:0/1823741053 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fae0c0f6510 0x7fae0c0f6930 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 rx=0x7fadf4009fb0 tx=0x7fadf4009b30).stop 2024-02-20T00:20:25.605 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:25.602+0000 7fae119e0700 1 -- 172.21.15.132:0/1823741053 shutdown_connections 2024-02-20T00:20:25.606 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:25.602+0000 7fae119e0700 1 --2- 172.21.15.132:0/1823741053 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fae0c0f6510 0x7fae0c0f6930 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:20:25.606 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:25.602+0000 7fae119e0700 1 -- 172.21.15.132:0/1823741053 >> 172.21.15.132:0/1823741053 conn(0x7fae0c0f1c70 msgr2=0x7fae0c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:20:25.606 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:25.602+0000 7fae119e0700 1 -- 172.21.15.132:0/1823741053 shutdown_connections 2024-02-20T00:20:25.606 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:25.602+0000 7fae119e0700 1 -- 172.21.15.132:0/1823741053 wait complete. 2024-02-20T00:20:25.606 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:25.603+0000 7fae119e0700 1 Processor -- start 2024-02-20T00:20:25.606 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:25.603+0000 7fae119e0700 1 -- start start 2024-02-20T00:20:25.606 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:25.603+0000 7fae119e0700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fae0c0f6510 0x7fae0c1067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:20:25.606 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:25.603+0000 7fae119e0700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae0c106cd0 con 0x7fae0c0f6510 2024-02-20T00:20:25.606 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:25.603+0000 7fae0affd700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fae0c0f6510 0x7fae0c1067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:20:25.606 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:25.603+0000 7fae0affd700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fae0c0f6510 0x7fae0c1067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:43470/0 (socket says 172.21.15.132:43470) 2024-02-20T00:20:25.606 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:25.603+0000 7fae0affd700 1 -- 172.21.15.132:0/1787994524 learned_addr learned my addr 172.21.15.132:0/1787994524 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:20:25.606 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:25.604+0000 7fae0affd700 1 -- 172.21.15.132:0/1787994524 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadf4005040 con 0x7fae0c0f6510 2024-02-20T00:20:25.606 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:25.604+0000 7fae0affd700 1 --2- 172.21.15.132:0/1787994524 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fae0c0f6510 0x7fae0c1067e0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 rx=0x7fadf4005010 tx=0x7fadf400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:20:25.606 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:25.604+0000 7fae037fe700 1 -- 172.21.15.132:0/1787994524 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fadf4009320 con 0x7fae0c0f6510 2024-02-20T00:20:25.607 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:25.604+0000 7fae119e0700 1 -- 172.21.15.132:0/1787994524 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae0c106ed0 con 0x7fae0c0f6510 2024-02-20T00:20:25.607 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:25.604+0000 7fae037fe700 1 -- 172.21.15.132:0/1787994524 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fadf4013380 con 0x7fae0c0f6510 2024-02-20T00:20:25.607 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:25.604+0000 7fae037fe700 1 -- 172.21.15.132:0/1787994524 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fadf4014e50 con 0x7fae0c0f6510 2024-02-20T00:20:25.608 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:25.604+0000 7fae119e0700 1 -- 172.21.15.132:0/1787994524 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae0c107360 con 0x7fae0c0f6510 2024-02-20T00:20:25.608 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:25.605+0000 7fae037fe700 1 -- 172.21.15.132:0/1787994524 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fadf4014420 con 0x7fae0c0f6510 2024-02-20T00:20:25.608 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:25.605+0000 7fae119e0700 1 -- 172.21.15.132:0/1787994524 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae0c045ad0 con 0x7fae0c0f6510 2024-02-20T00:20:25.608 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:25.605+0000 7fae037fe700 1 --2- 172.21.15.132:0/1787994524 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fadf802ff10 0x7fadf80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:20:25.608 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:25.605+0000 7fae037fe700 1 -- 172.21.15.132:0/1787994524 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fadf4040f40 con 0x7fae0c0f6510 2024-02-20T00:20:25.608 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:25.605+0000 7fae0a7fc700 1 --2- 172.21.15.132:0/1787994524 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fadf802ff10 0x7fadf80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:20:25.608 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:25.606+0000 7fae0a7fc700 1 --2- 172.21.15.132:0/1787994524 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fadf802ff10 0x7fadf80323d0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7fadfc0096a0 tx=0x7fadfc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:20:25.611 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:25.610+0000 7fae037fe700 1 -- 172.21.15.132:0/1787994524 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fadf401db70 con 0x7fae0c0f6510 2024-02-20T00:20:25.816 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:25.815+0000 7fae119e0700 1 -- 172.21.15.132:0/1787994524 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fae0c108210 con 0x7fae0c0f6510 2024-02-20T00:20:25.816 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:25.815+0000 7fae037fe700 1 -- 172.21.15.132:0/1787994524 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fadf401db70 con 0x7fae0c0f6510 2024-02-20T00:20:25.817 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:20:25.817 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:20:25.818 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:25.817+0000 7fae119e0700 1 -- 172.21.15.132:0/1787994524 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fadf802ff10 msgr2=0x7fadf80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:20:25.818 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:25.817+0000 7fae119e0700 1 --2- 172.21.15.132:0/1787994524 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fadf802ff10 0x7fadf80323d0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7fadfc0096a0 tx=0x7fadfc006b40).stop 2024-02-20T00:20:25.818 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:25.817+0000 7fae119e0700 1 -- 172.21.15.132:0/1787994524 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fae0c0f6510 msgr2=0x7fae0c1067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:20:25.818 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:25.817+0000 7fae119e0700 1 --2- 172.21.15.132:0/1787994524 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fae0c0f6510 0x7fae0c1067e0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 rx=0x7fadf4005010 tx=0x7fadf400b040).stop 2024-02-20T00:20:25.819 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:25.817+0000 7fae119e0700 1 -- 172.21.15.132:0/1787994524 shutdown_connections 2024-02-20T00:20:25.819 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:25.817+0000 7fae119e0700 1 --2- 172.21.15.132:0/1787994524 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fadf802ff10 0x7fadf80323d0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:20:25.819 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:25.817+0000 7fae119e0700 1 --2- 172.21.15.132:0/1787994524 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fae0c0f6510 0x7fae0c1067e0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:20:25.819 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:25.817+0000 7fae119e0700 1 -- 172.21.15.132:0/1787994524 >> 172.21.15.132:0/1787994524 conn(0x7fae0c0f1c70 msgr2=0x7fae0c0f26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:20:25.819 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:25.817+0000 7fae119e0700 1 -- 172.21.15.132:0/1787994524 shutdown_connections 2024-02-20T00:20:25.819 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:25.817+0000 7fae119e0700 1 -- 172.21.15.132:0/1787994524 wait complete. 2024-02-20T00:20:25.819 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:20:26.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:20:25 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:20:25.815864+0000 mon.smithi084 (mon.0) 312 : audit [DBG] from='client.? 172.21.15.132:0/1787994524' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:20:27.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:20:26 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:20:25.472725+0000 mgr.smithi084.bmqrnf (mgr.14162) 325 : cluster [DBG] pgmap v309: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:20:27.544 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:20:27.545 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:20:29.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:20:28 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:20:27.472970+0000 mgr.smithi084.bmqrnf (mgr.14162) 326 : cluster [DBG] pgmap v310: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:20:29.402 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:29.400+0000 7f5bd58ca700 1 -- 172.21.15.132:0/1394843996 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5bd00601e0 msgr2=0x7f5bd00f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:20:29.402 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:29.400+0000 7f5bd58ca700 1 --2- 172.21.15.132:0/1394843996 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5bd00601e0 0x7f5bd00f8f20 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 rx=0x7f5bc4009fb0 tx=0x7f5bc4009b30).stop 2024-02-20T00:20:29.402 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:29.400+0000 7f5bd58ca700 1 -- 172.21.15.132:0/1394843996 shutdown_connections 2024-02-20T00:20:29.402 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:29.400+0000 7f5bd58ca700 1 --2- 172.21.15.132:0/1394843996 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5bd00601e0 0x7f5bd00f8f20 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:20:29.402 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:29.400+0000 7f5bd58ca700 1 -- 172.21.15.132:0/1394843996 >> 172.21.15.132:0/1394843996 conn(0x7f5bd00f09a0 msgr2=0x7f5bd00f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:20:29.403 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:29.400+0000 7f5bd58ca700 1 -- 172.21.15.132:0/1394843996 shutdown_connections 2024-02-20T00:20:29.403 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:29.400+0000 7f5bd58ca700 1 -- 172.21.15.132:0/1394843996 wait complete. 2024-02-20T00:20:29.403 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:29.400+0000 7f5bd58ca700 1 Processor -- start 2024-02-20T00:20:29.403 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:29.401+0000 7f5bd58ca700 1 -- start start 2024-02-20T00:20:29.403 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:29.401+0000 7f5bd58ca700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5bd00601e0 0x7f5bd00f69a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:20:29.403 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:29.401+0000 7f5bd58ca700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bd00f4e20 con 0x7f5bd00601e0 2024-02-20T00:20:29.405 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:29.401+0000 7f5bd48c8700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5bd00601e0 0x7f5bd00f69a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:20:29.405 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:29.401+0000 7f5bd48c8700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5bd00601e0 0x7f5bd00f69a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:33432/0 (socket says 172.21.15.132:33432) 2024-02-20T00:20:29.405 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:29.401+0000 7f5bd48c8700 1 -- 172.21.15.132:0/3169161250 learned_addr learned my addr 172.21.15.132:0/3169161250 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:20:29.405 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:29.401+0000 7f5bd48c8700 1 -- 172.21.15.132:0/3169161250 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bc4005040 con 0x7f5bd00601e0 2024-02-20T00:20:29.405 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:29.402+0000 7f5bd48c8700 1 --2- 172.21.15.132:0/3169161250 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5bd00601e0 0x7f5bd00f69a0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0x7f5bc40089f0 tx=0x7f5bc400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:20:29.405 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:29.402+0000 7f5bcd7fa700 1 -- 172.21.15.132:0/3169161250 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5bc4009320 con 0x7f5bd00601e0 2024-02-20T00:20:29.405 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:29.402+0000 7f5bd58ca700 1 -- 172.21.15.132:0/3169161250 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bd00f4fc0 con 0x7f5bd00601e0 2024-02-20T00:20:29.405 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:29.402+0000 7f5bcd7fa700 1 -- 172.21.15.132:0/3169161250 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5bc4013380 con 0x7f5bd00601e0 2024-02-20T00:20:29.405 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:29.402+0000 7f5bcd7fa700 1 -- 172.21.15.132:0/3169161250 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5bc401ed80 con 0x7f5bd00601e0 2024-02-20T00:20:29.405 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:29.402+0000 7f5bd58ca700 1 -- 172.21.15.132:0/3169161250 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bd00f5450 con 0x7f5bd00601e0 2024-02-20T00:20:29.405 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:29.403+0000 7f5bcd7fa700 1 -- 172.21.15.132:0/3169161250 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5bc401e420 con 0x7f5bd00601e0 2024-02-20T00:20:29.405 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:29.403+0000 7f5bd58ca700 1 -- 172.21.15.132:0/3169161250 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5bd0059100 con 0x7f5bd00601e0 2024-02-20T00:20:29.406 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:29.403+0000 7f5bcd7fa700 1 --2- 172.21.15.132:0/3169161250 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f5bc002ffb0 0x7f5bc0032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:20:29.406 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:29.403+0000 7f5bcd7fa700 1 -- 172.21.15.132:0/3169161250 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5bc4040fc0 con 0x7f5bd00601e0 2024-02-20T00:20:29.406 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:29.403+0000 7f5bcffff700 1 --2- 172.21.15.132:0/3169161250 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f5bc002ffb0 0x7f5bc0032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:20:29.406 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:29.403+0000 7f5bcffff700 1 --2- 172.21.15.132:0/3169161250 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f5bc002ffb0 0x7f5bc0032470 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f5bbc0096a0 tx=0x7f5bbc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:20:29.409 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:29.407+0000 7f5bcd7fa700 1 -- 172.21.15.132:0/3169161250 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5bc40095f0 con 0x7f5bd00601e0 2024-02-20T00:20:29.615 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:29.611+0000 7f5bd58ca700 1 -- 172.21.15.132:0/3169161250 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5bd00fbe10 con 0x7f5bd00601e0 2024-02-20T00:20:29.616 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:29.611+0000 7f5bcd7fa700 1 -- 172.21.15.132:0/3169161250 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5bc401cb10 con 0x7f5bd00601e0 2024-02-20T00:20:29.616 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:20:29.616 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:20:29.617 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:29.613+0000 7f5bd58ca700 1 -- 172.21.15.132:0/3169161250 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f5bc002ffb0 msgr2=0x7f5bc0032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:20:29.617 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:29.613+0000 7f5bd58ca700 1 --2- 172.21.15.132:0/3169161250 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f5bc002ffb0 0x7f5bc0032470 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f5bbc0096a0 tx=0x7f5bbc006b40).stop 2024-02-20T00:20:29.617 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:29.613+0000 7f5bd58ca700 1 -- 172.21.15.132:0/3169161250 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5bd00601e0 msgr2=0x7f5bd00f69a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:20:29.617 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:29.613+0000 7f5bd58ca700 1 --2- 172.21.15.132:0/3169161250 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5bd00601e0 0x7f5bd00f69a0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0x7f5bc40089f0 tx=0x7f5bc400b040).stop 2024-02-20T00:20:29.617 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:29.613+0000 7f5bd58ca700 1 -- 172.21.15.132:0/3169161250 shutdown_connections 2024-02-20T00:20:29.617 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:29.613+0000 7f5bd58ca700 1 --2- 172.21.15.132:0/3169161250 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f5bc002ffb0 0x7f5bc0032470 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:20:29.617 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:29.613+0000 7f5bd58ca700 1 --2- 172.21.15.132:0/3169161250 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5bd00601e0 0x7f5bd00f69a0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:20:29.617 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:29.613+0000 7f5bd58ca700 1 -- 172.21.15.132:0/3169161250 >> 172.21.15.132:0/3169161250 conn(0x7f5bd00f09a0 msgr2=0x7f5bd00f13c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:20:29.617 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:29.613+0000 7f5bd58ca700 1 -- 172.21.15.132:0/3169161250 shutdown_connections 2024-02-20T00:20:29.618 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:29.613+0000 7f5bd58ca700 1 -- 172.21.15.132:0/3169161250 wait complete. 2024-02-20T00:20:29.618 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:20:30.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:20:29 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:20:29.612028+0000 mon.smithi084 (mon.0) 313 : audit [DBG] from='client.? 172.21.15.132:0/3169161250' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:20:31.203 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:20:31.203 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:20:31.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:20:30 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:20:29.473349+0000 mgr.smithi084.bmqrnf (mgr.14162) 327 : cluster [DBG] pgmap v311: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:20:33.176 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:33.166+0000 7f5d81235700 1 -- 172.21.15.132:0/2302113801 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5d7c0f6a10 msgr2=0x7f5d7c0f6e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:20:33.177 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:33.166+0000 7f5d81235700 1 --2- 172.21.15.132:0/2302113801 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5d7c0f6a10 0x7f5d7c0f6e30 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 rx=0x7f5d70009fb0 tx=0x7f5d70009b30).stop 2024-02-20T00:20:33.177 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:33.166+0000 7f5d81235700 1 -- 172.21.15.132:0/2302113801 shutdown_connections 2024-02-20T00:20:33.177 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:33.166+0000 7f5d81235700 1 --2- 172.21.15.132:0/2302113801 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5d7c0f6a10 0x7f5d7c0f6e30 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:20:33.177 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:33.166+0000 7f5d81235700 1 -- 172.21.15.132:0/2302113801 >> 172.21.15.132:0/2302113801 conn(0x7f5d7c0f02e0 msgr2=0x7f5d7c0f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:20:33.177 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:33.166+0000 7f5d81235700 1 -- 172.21.15.132:0/2302113801 shutdown_connections 2024-02-20T00:20:33.177 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:33.166+0000 7f5d81235700 1 -- 172.21.15.132:0/2302113801 wait complete. 2024-02-20T00:20:33.177 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:33.167+0000 7f5d81235700 1 Processor -- start 2024-02-20T00:20:33.177 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:33.167+0000 7f5d81235700 1 -- start start 2024-02-20T00:20:33.177 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:33.167+0000 7f5d81235700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5d7c0f6a10 0x7f5d7c0fd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:20:33.177 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:33.167+0000 7f5d81235700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d7c0fb960 con 0x7f5d7c0f6a10 2024-02-20T00:20:33.177 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:33.167+0000 7f5d7ad9d700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5d7c0f6a10 0x7f5d7c0fd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:20:33.177 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:33.167+0000 7f5d7ad9d700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5d7c0f6a10 0x7f5d7c0fd4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:33462/0 (socket says 172.21.15.132:33462) 2024-02-20T00:20:33.178 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:33.167+0000 7f5d7ad9d700 1 -- 172.21.15.132:0/2244121179 learned_addr learned my addr 172.21.15.132:0/2244121179 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:20:33.178 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:33.168+0000 7f5d7ad9d700 1 -- 172.21.15.132:0/2244121179 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d70005040 con 0x7f5d7c0f6a10 2024-02-20T00:20:33.178 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:33.168+0000 7f5d7ad9d700 1 --2- 172.21.15.132:0/2244121179 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5d7c0f6a10 0x7f5d7c0fd4e0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 rx=0x7f5d70005010 tx=0x7f5d7000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:20:33.178 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:33.168+0000 7f5d637fe700 1 -- 172.21.15.132:0/2244121179 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5d70009320 con 0x7f5d7c0f6a10 2024-02-20T00:20:33.178 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:33.168+0000 7f5d81235700 1 -- 172.21.15.132:0/2244121179 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d7c0fbb00 con 0x7f5d7c0f6a10 2024-02-20T00:20:33.178 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:33.168+0000 7f5d637fe700 1 -- 172.21.15.132:0/2244121179 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5d70013380 con 0x7f5d7c0f6a10 2024-02-20T00:20:33.178 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:33.168+0000 7f5d637fe700 1 -- 172.21.15.132:0/2244121179 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5d70014e50 con 0x7f5d7c0f6a10 2024-02-20T00:20:33.178 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:33.168+0000 7f5d81235700 1 -- 172.21.15.132:0/2244121179 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d7c0fbf90 con 0x7f5d7c0f6a10 2024-02-20T00:20:33.178 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:33.169+0000 7f5d637fe700 1 -- 172.21.15.132:0/2244121179 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5d70014420 con 0x7f5d7c0f6a10 2024-02-20T00:20:33.178 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:33.169+0000 7f5d81235700 1 -- 172.21.15.132:0/2244121179 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d7c059100 con 0x7f5d7c0f6a10 2024-02-20T00:20:33.178 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:33.169+0000 7f5d637fe700 1 --2- 172.21.15.132:0/2244121179 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f5d6402ff10 0x7f5d640323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:20:33.178 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:33.169+0000 7f5d637fe700 1 -- 172.21.15.132:0/2244121179 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5d70040f40 con 0x7f5d7c0f6a10 2024-02-20T00:20:33.178 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:33.169+0000 7f5d7a59c700 1 --2- 172.21.15.132:0/2244121179 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f5d6402ff10 0x7f5d640323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:20:33.178 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:33.170+0000 7f5d7a59c700 1 --2- 172.21.15.132:0/2244121179 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f5d6402ff10 0x7f5d640323d0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7f5d6c0096a0 tx=0x7f5d6c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:20:33.179 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:33.174+0000 7f5d637fe700 1 -- 172.21.15.132:0/2244121179 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5d700095f0 con 0x7f5d7c0f6a10 2024-02-20T00:20:33.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:20:32 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:20:31.473656+0000 mgr.smithi084.bmqrnf (mgr.14162) 328 : cluster [DBG] pgmap v312: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:20:33.380 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:33.379+0000 7f5d81235700 1 -- 172.21.15.132:0/2244121179 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5d7c0fafa0 con 0x7f5d7c0f6a10 2024-02-20T00:20:33.381 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:33.380+0000 7f5d637fe700 1 -- 172.21.15.132:0/2244121179 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5d7001db70 con 0x7f5d7c0f6a10 2024-02-20T00:20:33.382 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:20:33.382 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:20:33.383 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:33.381+0000 7f5d81235700 1 -- 172.21.15.132:0/2244121179 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f5d6402ff10 msgr2=0x7f5d640323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:20:33.384 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:33.381+0000 7f5d81235700 1 --2- 172.21.15.132:0/2244121179 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f5d6402ff10 0x7f5d640323d0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7f5d6c0096a0 tx=0x7f5d6c006b40).stop 2024-02-20T00:20:33.384 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:33.381+0000 7f5d81235700 1 -- 172.21.15.132:0/2244121179 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5d7c0f6a10 msgr2=0x7f5d7c0fd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:20:33.384 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:33.381+0000 7f5d81235700 1 --2- 172.21.15.132:0/2244121179 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5d7c0f6a10 0x7f5d7c0fd4e0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 rx=0x7f5d70005010 tx=0x7f5d7000b040).stop 2024-02-20T00:20:33.384 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:33.381+0000 7f5d81235700 1 -- 172.21.15.132:0/2244121179 shutdown_connections 2024-02-20T00:20:33.384 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:33.381+0000 7f5d81235700 1 --2- 172.21.15.132:0/2244121179 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f5d6402ff10 0x7f5d640323d0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:20:33.385 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:33.381+0000 7f5d81235700 1 --2- 172.21.15.132:0/2244121179 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5d7c0f6a10 0x7f5d7c0fd4e0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:20:33.385 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:33.381+0000 7f5d81235700 1 -- 172.21.15.132:0/2244121179 >> 172.21.15.132:0/2244121179 conn(0x7f5d7c0f02e0 msgr2=0x7f5d7c0f4f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:20:33.385 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:33.381+0000 7f5d81235700 1 -- 172.21.15.132:0/2244121179 shutdown_connections 2024-02-20T00:20:33.385 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:33.381+0000 7f5d81235700 1 -- 172.21.15.132:0/2244121179 wait complete. 2024-02-20T00:20:33.385 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:20:34.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:20:33 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:20:33.380241+0000 mon.smithi084 (mon.0) 314 : audit [DBG] from='client.? 172.21.15.132:0/2244121179' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:20:35.026 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:20:35.026 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:20:35.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:20:34 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:20:33.473928+0000 mgr.smithi084.bmqrnf (mgr.14162) 329 : cluster [DBG] pgmap v313: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:20:36.912 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:36.909+0000 7f2f902cf700 1 -- 172.21.15.132:0/3459653474 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2f880f6510 msgr2=0x7f2f880f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:20:36.912 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:36.909+0000 7f2f902cf700 1 --2- 172.21.15.132:0/3459653474 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2f880f6510 0x7f2f880f6930 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 rx=0x7f2f7c009fb0 tx=0x7f2f7c009b30).stop 2024-02-20T00:20:36.912 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:36.909+0000 7f2f902cf700 1 -- 172.21.15.132:0/3459653474 shutdown_connections 2024-02-20T00:20:36.912 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:36.909+0000 7f2f902cf700 1 --2- 172.21.15.132:0/3459653474 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2f880f6510 0x7f2f880f6930 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:20:36.912 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:36.909+0000 7f2f902cf700 1 -- 172.21.15.132:0/3459653474 >> 172.21.15.132:0/3459653474 conn(0x7f2f880f1c70 msgr2=0x7f2f880f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:20:36.912 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:36.909+0000 7f2f902cf700 1 -- 172.21.15.132:0/3459653474 shutdown_connections 2024-02-20T00:20:36.912 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:36.909+0000 7f2f902cf700 1 -- 172.21.15.132:0/3459653474 wait complete. 2024-02-20T00:20:36.912 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:36.909+0000 7f2f902cf700 1 Processor -- start 2024-02-20T00:20:36.913 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:36.909+0000 7f2f902cf700 1 -- start start 2024-02-20T00:20:36.913 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:36.909+0000 7f2f902cf700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2f880f6510 0x7f2f8806b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:20:36.913 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:36.909+0000 7f2f902cf700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f88069e20 con 0x7f2f880f6510 2024-02-20T00:20:36.913 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:36.910+0000 7f2f8e06b700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2f880f6510 0x7f2f8806b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:20:36.913 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:36.910+0000 7f2f8e06b700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2f880f6510 0x7f2f8806b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:33486/0 (socket says 172.21.15.132:33486) 2024-02-20T00:20:36.913 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:36.910+0000 7f2f8e06b700 1 -- 172.21.15.132:0/3469686656 learned_addr learned my addr 172.21.15.132:0/3469686656 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:20:36.913 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:36.910+0000 7f2f8e06b700 1 -- 172.21.15.132:0/3469686656 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f7c005040 con 0x7f2f880f6510 2024-02-20T00:20:36.913 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:36.910+0000 7f2f8e06b700 1 --2- 172.21.15.132:0/3469686656 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2f880f6510 0x7f2f8806b9a0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 rx=0x7f2f7c005010 tx=0x7f2f7c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:20:36.913 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:36.910+0000 7f2f7affd700 1 -- 172.21.15.132:0/3469686656 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2f7c009320 con 0x7f2f880f6510 2024-02-20T00:20:36.913 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:36.910+0000 7f2f902cf700 1 -- 172.21.15.132:0/3469686656 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f88069fc0 con 0x7f2f880f6510 2024-02-20T00:20:36.913 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:36.910+0000 7f2f902cf700 1 -- 172.21.15.132:0/3469686656 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f8806a450 con 0x7f2f880f6510 2024-02-20T00:20:36.913 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:36.910+0000 7f2f7affd700 1 -- 172.21.15.132:0/3469686656 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2f7c013380 con 0x7f2f880f6510 2024-02-20T00:20:36.913 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:36.910+0000 7f2f7affd700 1 -- 172.21.15.132:0/3469686656 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2f7c014df0 con 0x7f2f880f6510 2024-02-20T00:20:36.914 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:36.911+0000 7f2f7affd700 1 -- 172.21.15.132:0/3469686656 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2f7c01e420 con 0x7f2f880f6510 2024-02-20T00:20:36.914 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:36.911+0000 7f2f902cf700 1 -- 172.21.15.132:0/3469686656 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f70004fa0 con 0x7f2f880f6510 2024-02-20T00:20:36.914 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:36.911+0000 7f2f7affd700 1 --2- 172.21.15.132:0/3469686656 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f2f7402ff10 0x7f2f740323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:20:36.915 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:36.911+0000 7f2f7affd700 1 -- 172.21.15.132:0/3469686656 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2f7c040eb0 con 0x7f2f880f6510 2024-02-20T00:20:36.915 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:36.912+0000 7f2f8d86a700 1 --2- 172.21.15.132:0/3469686656 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f2f7402ff10 0x7f2f740323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:20:36.915 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:36.912+0000 7f2f8d86a700 1 --2- 172.21.15.132:0/3469686656 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f2f7402ff10 0x7f2f740323d0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f2f840096a0 tx=0x7f2f84006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:20:36.917 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:36.916+0000 7f2f7affd700 1 -- 172.21.15.132:0/3469686656 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2f7c026720 con 0x7f2f880f6510 2024-02-20T00:20:37.122 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:37.120+0000 7f2f902cf700 1 -- 172.21.15.132:0/3469686656 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2f70005670 con 0x7f2f880f6510 2024-02-20T00:20:37.122 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:37.121+0000 7f2f7affd700 1 -- 172.21.15.132:0/3469686656 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2f7c01c970 con 0x7f2f880f6510 2024-02-20T00:20:37.123 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:20:37.123 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:20:37.124 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:37.123+0000 7f2f902cf700 1 -- 172.21.15.132:0/3469686656 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f2f7402ff10 msgr2=0x7f2f740323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:20:37.124 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:37.123+0000 7f2f902cf700 1 --2- 172.21.15.132:0/3469686656 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f2f7402ff10 0x7f2f740323d0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f2f840096a0 tx=0x7f2f84006b40).stop 2024-02-20T00:20:37.125 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:37.123+0000 7f2f902cf700 1 -- 172.21.15.132:0/3469686656 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2f880f6510 msgr2=0x7f2f8806b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:20:37.125 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:37.123+0000 7f2f902cf700 1 --2- 172.21.15.132:0/3469686656 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2f880f6510 0x7f2f8806b9a0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 rx=0x7f2f7c005010 tx=0x7f2f7c00b040).stop 2024-02-20T00:20:37.125 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:37.123+0000 7f2f902cf700 1 -- 172.21.15.132:0/3469686656 shutdown_connections 2024-02-20T00:20:37.125 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:37.123+0000 7f2f902cf700 1 --2- 172.21.15.132:0/3469686656 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f2f7402ff10 0x7f2f740323d0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:20:37.125 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:37.123+0000 7f2f902cf700 1 --2- 172.21.15.132:0/3469686656 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2f880f6510 0x7f2f8806b9a0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:20:37.125 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:37.123+0000 7f2f902cf700 1 -- 172.21.15.132:0/3469686656 >> 172.21.15.132:0/3469686656 conn(0x7f2f880f1c70 msgr2=0x7f2f880f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:20:37.125 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:37.123+0000 7f2f902cf700 1 -- 172.21.15.132:0/3469686656 shutdown_connections 2024-02-20T00:20:37.125 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:37.123+0000 7f2f902cf700 1 -- 172.21.15.132:0/3469686656 wait complete. 2024-02-20T00:20:37.125 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:20:37.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:20:36 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:20:35.474297+0000 mgr.smithi084.bmqrnf (mgr.14162) 330 : cluster [DBG] pgmap v314: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:20:38.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:20:37 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:20:37.121797+0000 mon.smithi084 (mon.0) 315 : audit [DBG] from='client.? 172.21.15.132:0/3469686656' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:20:38.707 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:20:38.707 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:20:39.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:20:38 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:20:37.474595+0000 mgr.smithi084.bmqrnf (mgr.14162) 331 : cluster [DBG] pgmap v315: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:20:40.640 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:40.636+0000 7f22d1c53700 1 -- 172.21.15.132:0/2738130356 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f22cc0601e0 msgr2=0x7f22cc0f9190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:20:40.640 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:40.636+0000 7f22d1c53700 1 --2- 172.21.15.132:0/2738130356 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f22cc0601e0 0x7f22cc0f9190 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 rx=0x7f22bc009fb0 tx=0x7f22bc009b30).stop 2024-02-20T00:20:40.640 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:40.636+0000 7f22d1c53700 1 -- 172.21.15.132:0/2738130356 shutdown_connections 2024-02-20T00:20:40.641 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:40.636+0000 7f22d1c53700 1 --2- 172.21.15.132:0/2738130356 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f22cc0601e0 0x7f22cc0f9190 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:20:40.641 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:40.636+0000 7f22d1c53700 1 -- 172.21.15.132:0/2738130356 >> 172.21.15.132:0/2738130356 conn(0x7f22cc0f0b70 msgr2=0x7f22cc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:20:40.641 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:40.636+0000 7f22d1c53700 1 -- 172.21.15.132:0/2738130356 shutdown_connections 2024-02-20T00:20:40.641 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:40.636+0000 7f22d1c53700 1 -- 172.21.15.132:0/2738130356 wait complete. 2024-02-20T00:20:40.641 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:40.636+0000 7f22d1c53700 1 Processor -- start 2024-02-20T00:20:40.641 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:40.636+0000 7f22d1c53700 1 -- start start 2024-02-20T00:20:40.641 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:40.637+0000 7f22d1c53700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f22cc0601e0 0x7f22cc1067d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:20:40.641 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:40.637+0000 7f22d1c53700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22cc106cc0 con 0x7f22cc0601e0 2024-02-20T00:20:40.641 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:40.637+0000 7f22cb7fe700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f22cc0601e0 0x7f22cc1067d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:20:40.641 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:40.637+0000 7f22cb7fe700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f22cc0601e0 0x7f22cc1067d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:46202/0 (socket says 172.21.15.132:46202) 2024-02-20T00:20:40.641 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:40.637+0000 7f22cb7fe700 1 -- 172.21.15.132:0/3319413776 learned_addr learned my addr 172.21.15.132:0/3319413776 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:20:40.641 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:40.637+0000 7f22cb7fe700 1 -- 172.21.15.132:0/3319413776 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22bc005040 con 0x7f22cc0601e0 2024-02-20T00:20:40.641 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:40.637+0000 7f22cb7fe700 1 --2- 172.21.15.132:0/3319413776 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f22cc0601e0 0x7f22cc1067d0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 rx=0x7f22bc007a10 tx=0x7f22bc009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:20:40.641 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:40.637+0000 7f22b3fff700 1 -- 172.21.15.132:0/3319413776 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f22bc018070 con 0x7f22cc0601e0 2024-02-20T00:20:40.642 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:40.637+0000 7f22d1c53700 1 -- 172.21.15.132:0/3319413776 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22cc106ec0 con 0x7f22cc0601e0 2024-02-20T00:20:40.642 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:40.637+0000 7f22b3fff700 1 -- 172.21.15.132:0/3319413776 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f22bc013410 con 0x7f22cc0601e0 2024-02-20T00:20:40.642 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:40.637+0000 7f22b3fff700 1 -- 172.21.15.132:0/3319413776 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f22bc01c3c0 con 0x7f22cc0601e0 2024-02-20T00:20:40.642 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:40.637+0000 7f22d1c53700 1 -- 172.21.15.132:0/3319413776 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22cc107350 con 0x7f22cc0601e0 2024-02-20T00:20:40.643 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:40.638+0000 7f22b3fff700 1 -- 172.21.15.132:0/3319413776 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f22bc01d420 con 0x7f22cc0601e0 2024-02-20T00:20:40.643 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:40.638+0000 7f22d1c53700 1 -- 172.21.15.132:0/3319413776 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22cc045ad0 con 0x7f22cc0601e0 2024-02-20T00:20:40.643 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:40.638+0000 7f22b3fff700 1 --2- 172.21.15.132:0/3319413776 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f22b402ff60 0x7f22b4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:20:40.643 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:40.638+0000 7f22b3fff700 1 -- 172.21.15.132:0/3319413776 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f22bc041480 con 0x7f22cc0601e0 2024-02-20T00:20:40.643 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:40.639+0000 7f22caffd700 1 --2- 172.21.15.132:0/3319413776 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f22b402ff60 0x7f22b4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:20:40.643 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:40.639+0000 7f22caffd700 1 --2- 172.21.15.132:0/3319413776 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f22b402ff60 0x7f22b4032420 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7f22c00096a0 tx=0x7f22c0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:20:40.645 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:40.643+0000 7f22b3fff700 1 -- 172.21.15.132:0/3319413776 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f22bc013ca0 con 0x7f22cc0601e0 2024-02-20T00:20:40.848 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:40.846+0000 7f22d1c53700 1 -- 172.21.15.132:0/3319413776 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f22cc108200 con 0x7f22cc0601e0 2024-02-20T00:20:40.849 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:40.847+0000 7f22b3fff700 1 -- 172.21.15.132:0/3319413776 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f22bc01b370 con 0x7f22cc0601e0 2024-02-20T00:20:40.850 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:20:40.850 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:20:40.851 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:40.848+0000 7f22d1c53700 1 -- 172.21.15.132:0/3319413776 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f22b402ff60 msgr2=0x7f22b4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:20:40.851 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:40.848+0000 7f22d1c53700 1 --2- 172.21.15.132:0/3319413776 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f22b402ff60 0x7f22b4032420 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7f22c00096a0 tx=0x7f22c0006b40).stop 2024-02-20T00:20:40.851 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:40.848+0000 7f22d1c53700 1 -- 172.21.15.132:0/3319413776 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f22cc0601e0 msgr2=0x7f22cc1067d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:20:40.851 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:40.848+0000 7f22d1c53700 1 --2- 172.21.15.132:0/3319413776 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f22cc0601e0 0x7f22cc1067d0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 rx=0x7f22bc007a10 tx=0x7f22bc009b30).stop 2024-02-20T00:20:40.851 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:40.848+0000 7f22d1c53700 1 -- 172.21.15.132:0/3319413776 shutdown_connections 2024-02-20T00:20:40.851 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:40.848+0000 7f22d1c53700 1 --2- 172.21.15.132:0/3319413776 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f22b402ff60 0x7f22b4032420 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:20:40.851 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:40.848+0000 7f22d1c53700 1 --2- 172.21.15.132:0/3319413776 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f22cc0601e0 0x7f22cc1067d0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:20:40.851 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:40.848+0000 7f22d1c53700 1 -- 172.21.15.132:0/3319413776 >> 172.21.15.132:0/3319413776 conn(0x7f22cc0f0b70 msgr2=0x7f22cc0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:20:40.851 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:40.848+0000 7f22d1c53700 1 -- 172.21.15.132:0/3319413776 shutdown_connections 2024-02-20T00:20:40.852 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:40.848+0000 7f22d1c53700 1 -- 172.21.15.132:0/3319413776 wait complete. 2024-02-20T00:20:40.852 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:20:41.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:20:40 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:20:39.474916+0000 mgr.smithi084.bmqrnf (mgr.14162) 332 : cluster [DBG] pgmap v316: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:20:41.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:20:40 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:20:40.848334+0000 mon.smithi084 (mon.0) 316 : audit [DBG] from='client.? 172.21.15.132:0/3319413776' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:20:42.654 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:20:42.654 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:20:43.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:20:42 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:20:41.475203+0000 mgr.smithi084.bmqrnf (mgr.14162) 333 : cluster [DBG] pgmap v317: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:20:44.507 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:44.500+0000 7ff3340ee700 1 -- 172.21.15.132:0/2683803487 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff32c0f5410 msgr2=0x7ff32c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:20:44.507 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:44.500+0000 7ff3340ee700 1 --2- 172.21.15.132:0/2683803487 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff32c0f5410 0x7ff32c0f5830 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 rx=0x7ff31c009fb0 tx=0x7ff31c009b30).stop 2024-02-20T00:20:44.507 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:44.500+0000 7ff3340ee700 1 -- 172.21.15.132:0/2683803487 shutdown_connections 2024-02-20T00:20:44.507 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:44.500+0000 7ff3340ee700 1 --2- 172.21.15.132:0/2683803487 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff32c0f5410 0x7ff32c0f5830 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:20:44.507 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:44.500+0000 7ff3340ee700 1 -- 172.21.15.132:0/2683803487 >> 172.21.15.132:0/2683803487 conn(0x7ff32c0f0b70 msgr2=0x7ff32c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:20:44.507 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:44.501+0000 7ff3340ee700 1 -- 172.21.15.132:0/2683803487 shutdown_connections 2024-02-20T00:20:44.507 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:44.501+0000 7ff3340ee700 1 -- 172.21.15.132:0/2683803487 wait complete. 2024-02-20T00:20:44.507 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:44.501+0000 7ff3340ee700 1 Processor -- start 2024-02-20T00:20:44.507 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:44.501+0000 7ff3340ee700 1 -- start start 2024-02-20T00:20:44.507 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:44.501+0000 7ff3340ee700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff32c0f5410 0x7ff32c1094c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:20:44.507 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:44.501+0000 7ff3340ee700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff32c060240 con 0x7ff32c0f5410 2024-02-20T00:20:44.508 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:44.501+0000 7ff331e8a700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff32c0f5410 0x7ff32c1094c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:20:44.508 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:44.502+0000 7ff331e8a700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff32c0f5410 0x7ff32c1094c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:46212/0 (socket says 172.21.15.132:46212) 2024-02-20T00:20:44.508 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:44.502+0000 7ff331e8a700 1 -- 172.21.15.132:0/2509783129 learned_addr learned my addr 172.21.15.132:0/2509783129 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:20:44.508 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:44.502+0000 7ff331e8a700 1 -- 172.21.15.132:0/2509783129 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff31c005040 con 0x7ff32c0f5410 2024-02-20T00:20:44.508 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:44.502+0000 7ff331e8a700 1 --2- 172.21.15.132:0/2509783129 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff32c0f5410 0x7ff32c1094c0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 rx=0x7ff31c005010 tx=0x7ff31c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:20:44.508 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:44.502+0000 7ff3227fc700 1 -- 172.21.15.132:0/2509783129 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff31c009320 con 0x7ff32c0f5410 2024-02-20T00:20:44.508 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:44.502+0000 7ff3340ee700 1 -- 172.21.15.132:0/2509783129 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff32c060440 con 0x7ff32c0f5410 2024-02-20T00:20:44.508 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:44.502+0000 7ff3227fc700 1 -- 172.21.15.132:0/2509783129 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff31c013380 con 0x7ff32c0f5410 2024-02-20T00:20:44.508 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:44.502+0000 7ff3227fc700 1 -- 172.21.15.132:0/2509783129 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff31c014e50 con 0x7ff32c0f5410 2024-02-20T00:20:44.508 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:44.502+0000 7ff3340ee700 1 -- 172.21.15.132:0/2509783129 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff32c109b70 con 0x7ff32c0f5410 2024-02-20T00:20:44.508 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:44.503+0000 7ff3227fc700 1 -- 172.21.15.132:0/2509783129 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff31c0095f0 con 0x7ff32c0f5410 2024-02-20T00:20:44.508 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:44.503+0000 7ff3340ee700 1 -- 172.21.15.132:0/2509783129 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff32c045ad0 con 0x7ff32c0f5410 2024-02-20T00:20:44.509 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:44.503+0000 7ff3227fc700 1 --2- 172.21.15.132:0/2509783129 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7ff31802ff10 0x7ff3180323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:20:44.509 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:44.503+0000 7ff3227fc700 1 -- 172.21.15.132:0/2509783129 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff31c040f10 con 0x7ff32c0f5410 2024-02-20T00:20:44.509 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:44.503+0000 7ff331689700 1 --2- 172.21.15.132:0/2509783129 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7ff31802ff10 0x7ff3180323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:20:44.509 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:44.504+0000 7ff331689700 1 --2- 172.21.15.132:0/2509783129 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7ff31802ff10 0x7ff3180323d0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7ff3280096a0 tx=0x7ff328006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:20:44.510 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:44.507+0000 7ff3227fc700 1 -- 172.21.15.132:0/2509783129 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff31c014420 con 0x7ff32c0f5410 2024-02-20T00:20:44.717 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:44.714+0000 7ff3340ee700 1 -- 172.21.15.132:0/2509783129 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff32c10a390 con 0x7ff32c0f5410 2024-02-20T00:20:44.717 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:44.715+0000 7ff3227fc700 1 -- 172.21.15.132:0/2509783129 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff31c010230 con 0x7ff32c0f5410 2024-02-20T00:20:44.717 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:20:44.718 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:20:44.719 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:44.717+0000 7ff3340ee700 1 -- 172.21.15.132:0/2509783129 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7ff31802ff10 msgr2=0x7ff3180323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:20:44.719 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:44.717+0000 7ff3340ee700 1 --2- 172.21.15.132:0/2509783129 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7ff31802ff10 0x7ff3180323d0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7ff3280096a0 tx=0x7ff328006b40).stop 2024-02-20T00:20:44.719 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:44.717+0000 7ff3340ee700 1 -- 172.21.15.132:0/2509783129 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff32c0f5410 msgr2=0x7ff32c1094c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:20:44.719 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:44.717+0000 7ff3340ee700 1 --2- 172.21.15.132:0/2509783129 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff32c0f5410 0x7ff32c1094c0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 rx=0x7ff31c005010 tx=0x7ff31c00b040).stop 2024-02-20T00:20:44.719 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:44.717+0000 7ff3340ee700 1 -- 172.21.15.132:0/2509783129 shutdown_connections 2024-02-20T00:20:44.719 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:44.717+0000 7ff3340ee700 1 --2- 172.21.15.132:0/2509783129 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7ff31802ff10 0x7ff3180323d0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:20:44.720 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:44.717+0000 7ff3340ee700 1 --2- 172.21.15.132:0/2509783129 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff32c0f5410 0x7ff32c1094c0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:20:44.720 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:44.717+0000 7ff3340ee700 1 -- 172.21.15.132:0/2509783129 >> 172.21.15.132:0/2509783129 conn(0x7ff32c0f0b70 msgr2=0x7ff32c0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:20:44.720 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:44.717+0000 7ff3340ee700 1 -- 172.21.15.132:0/2509783129 shutdown_connections 2024-02-20T00:20:44.720 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:44.717+0000 7ff3340ee700 1 -- 172.21.15.132:0/2509783129 wait complete. 2024-02-20T00:20:44.720 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:20:45.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:20:44 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:20:43.475471+0000 mgr.smithi084.bmqrnf (mgr.14162) 334 : cluster [DBG] pgmap v318: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:20:45.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:20:44 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:20:44.716684+0000 mon.smithi084 (mon.0) 317 : audit [DBG] from='client.? 172.21.15.132:0/2509783129' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:20:46.328 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:20:46.328 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:20:47.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:20:46 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:20:45.475852+0000 mgr.smithi084.bmqrnf (mgr.14162) 335 : cluster [DBG] pgmap v319: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:20:48.198 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:48.194+0000 7fdd63fff700 1 -- 172.21.15.132:0/176724685 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdd640601e0 msgr2=0x7fdd640f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:20:48.198 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:48.194+0000 7fdd63fff700 1 --2- 172.21.15.132:0/176724685 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdd640601e0 0x7fdd640f8f20 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 rx=0x7fdd54009fb0 tx=0x7fdd54009b30).stop 2024-02-20T00:20:48.198 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:48.194+0000 7fdd63fff700 1 -- 172.21.15.132:0/176724685 shutdown_connections 2024-02-20T00:20:48.199 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:48.194+0000 7fdd63fff700 1 --2- 172.21.15.132:0/176724685 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdd640601e0 0x7fdd640f8f20 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:20:48.199 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:48.194+0000 7fdd63fff700 1 -- 172.21.15.132:0/176724685 >> 172.21.15.132:0/176724685 conn(0x7fdd640f09a0 msgr2=0x7fdd640f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:20:48.199 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:48.194+0000 7fdd63fff700 1 -- 172.21.15.132:0/176724685 shutdown_connections 2024-02-20T00:20:48.199 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:48.194+0000 7fdd63fff700 1 -- 172.21.15.132:0/176724685 wait complete. 2024-02-20T00:20:48.199 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:48.195+0000 7fdd63fff700 1 Processor -- start 2024-02-20T00:20:48.199 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:48.195+0000 7fdd63fff700 1 -- start start 2024-02-20T00:20:48.199 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:48.196+0000 7fdd63fff700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdd640601e0 0x7fdd641087d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:20:48.199 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:48.196+0000 7fdd63fff700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd64108cc0 con 0x7fdd640601e0 2024-02-20T00:20:48.199 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:48.196+0000 7fdd62ffd700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdd640601e0 0x7fdd641087d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:20:48.199 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:48.196+0000 7fdd62ffd700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdd640601e0 0x7fdd641087d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:38282/0 (socket says 172.21.15.132:38282) 2024-02-20T00:20:48.199 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:48.196+0000 7fdd62ffd700 1 -- 172.21.15.132:0/3601799495 learned_addr learned my addr 172.21.15.132:0/3601799495 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:20:48.201 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:48.196+0000 7fdd62ffd700 1 -- 172.21.15.132:0/3601799495 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd54005040 con 0x7fdd640601e0 2024-02-20T00:20:48.201 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:48.196+0000 7fdd62ffd700 1 --2- 172.21.15.132:0/3601799495 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdd640601e0 0x7fdd641087d0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 rx=0x7fdd54005010 tx=0x7fdd5400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:20:48.201 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:48.197+0000 7fdd4b7fe700 1 -- 172.21.15.132:0/3601799495 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdd54009320 con 0x7fdd640601e0 2024-02-20T00:20:48.201 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:48.197+0000 7fdd63fff700 1 -- 172.21.15.132:0/3601799495 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd64108ec0 con 0x7fdd640601e0 2024-02-20T00:20:48.201 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:48.197+0000 7fdd4b7fe700 1 -- 172.21.15.132:0/3601799495 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdd54013380 con 0x7fdd640601e0 2024-02-20T00:20:48.201 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:48.197+0000 7fdd4b7fe700 1 -- 172.21.15.132:0/3601799495 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdd54014e50 con 0x7fdd640601e0 2024-02-20T00:20:48.201 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:48.197+0000 7fdd63fff700 1 -- 172.21.15.132:0/3601799495 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd64109350 con 0x7fdd640601e0 2024-02-20T00:20:48.201 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:48.197+0000 7fdd4b7fe700 1 -- 172.21.15.132:0/3601799495 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdd540263a0 con 0x7fdd640601e0 2024-02-20T00:20:48.201 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:48.198+0000 7fdd4b7fe700 1 --2- 172.21.15.132:0/3601799495 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fdd4c02ff10 0x7fdd4c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:20:48.202 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:48.198+0000 7fdd4b7fe700 1 -- 172.21.15.132:0/3601799495 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdd54040f90 con 0x7fdd640601e0 2024-02-20T00:20:48.202 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:48.198+0000 7fdd63fff700 1 -- 172.21.15.132:0/3601799495 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd64045ad0 con 0x7fdd640601e0 2024-02-20T00:20:48.202 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:48.198+0000 7fdd627fc700 1 --2- 172.21.15.132:0/3601799495 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fdd4c02ff10 0x7fdd4c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:20:48.202 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:48.198+0000 7fdd627fc700 1 --2- 172.21.15.132:0/3601799495 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fdd4c02ff10 0x7fdd4c0323d0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7fdd580096a0 tx=0x7fdd58006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:20:48.205 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:48.203+0000 7fdd4b7fe700 1 -- 172.21.15.132:0/3601799495 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdd54012920 con 0x7fdd640601e0 2024-02-20T00:20:48.413 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:48.410+0000 7fdd63fff700 1 -- 172.21.15.132:0/3601799495 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdd6410a070 con 0x7fdd640601e0 2024-02-20T00:20:48.413 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:48.411+0000 7fdd4b7fe700 1 -- 172.21.15.132:0/3601799495 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fdd5402b020 con 0x7fdd640601e0 2024-02-20T00:20:48.414 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:20:48.414 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:20:48.415 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:48.412+0000 7fdd63fff700 1 -- 172.21.15.132:0/3601799495 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fdd4c02ff10 msgr2=0x7fdd4c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:20:48.415 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:48.412+0000 7fdd63fff700 1 --2- 172.21.15.132:0/3601799495 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fdd4c02ff10 0x7fdd4c0323d0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7fdd580096a0 tx=0x7fdd58006b40).stop 2024-02-20T00:20:48.415 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:48.412+0000 7fdd63fff700 1 -- 172.21.15.132:0/3601799495 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdd640601e0 msgr2=0x7fdd641087d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:20:48.415 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:48.412+0000 7fdd63fff700 1 --2- 172.21.15.132:0/3601799495 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdd640601e0 0x7fdd641087d0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 rx=0x7fdd54005010 tx=0x7fdd5400b040).stop 2024-02-20T00:20:48.415 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:48.412+0000 7fdd63fff700 1 -- 172.21.15.132:0/3601799495 shutdown_connections 2024-02-20T00:20:48.415 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:48.412+0000 7fdd63fff700 1 --2- 172.21.15.132:0/3601799495 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fdd4c02ff10 0x7fdd4c0323d0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:20:48.415 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:48.412+0000 7fdd63fff700 1 --2- 172.21.15.132:0/3601799495 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdd640601e0 0x7fdd641087d0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:20:48.415 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:48.412+0000 7fdd63fff700 1 -- 172.21.15.132:0/3601799495 >> 172.21.15.132:0/3601799495 conn(0x7fdd640f09a0 msgr2=0x7fdd640f1390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:20:48.415 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:48.412+0000 7fdd63fff700 1 -- 172.21.15.132:0/3601799495 shutdown_connections 2024-02-20T00:20:48.415 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:48.412+0000 7fdd63fff700 1 -- 172.21.15.132:0/3601799495 wait complete. 2024-02-20T00:20:48.416 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:20:49.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:20:48 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:20:47.476147+0000 mgr.smithi084.bmqrnf (mgr.14162) 336 : cluster [DBG] pgmap v320: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:20:49.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:20:48 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:20:48.412559+0000 mon.smithi084 (mon.0) 318 : audit [DBG] from='client.? 172.21.15.132:0/3601799495' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:20:50.009 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:20:50.010 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:20:51.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:20:50 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:20:49.476462+0000 mgr.smithi084.bmqrnf (mgr.14162) 337 : cluster [DBG] pgmap v321: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:20:51.823 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:51.820+0000 7f4912d4c700 1 -- 172.21.15.132:0/478539961 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f490c060280 msgr2=0x7f490c0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:20:51.823 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:51.820+0000 7f4912d4c700 1 --2- 172.21.15.132:0/478539961 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f490c060280 0x7f490c0f90f0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 rx=0x7f4900009fb0 tx=0x7f4900009b30).stop 2024-02-20T00:20:51.823 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:51.820+0000 7f4912d4c700 1 -- 172.21.15.132:0/478539961 shutdown_connections 2024-02-20T00:20:51.823 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:51.820+0000 7f4912d4c700 1 --2- 172.21.15.132:0/478539961 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f490c060280 0x7f490c0f90f0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:20:51.823 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:51.820+0000 7f4912d4c700 1 -- 172.21.15.132:0/478539961 >> 172.21.15.132:0/478539961 conn(0x7f490c0f0b70 msgr2=0x7f490c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:20:51.823 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:51.820+0000 7f4912d4c700 1 -- 172.21.15.132:0/478539961 shutdown_connections 2024-02-20T00:20:51.824 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:51.820+0000 7f4912d4c700 1 -- 172.21.15.132:0/478539961 wait complete. 2024-02-20T00:20:51.824 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:51.820+0000 7f4912d4c700 1 Processor -- start 2024-02-20T00:20:51.824 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:51.821+0000 7f4912d4c700 1 -- start start 2024-02-20T00:20:51.824 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:51.821+0000 7f4912d4c700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f490c060280 0x7f490c106780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:20:51.824 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:51.821+0000 7f4912d4c700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f490c106c70 con 0x7f490c060280 2024-02-20T00:20:51.826 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:51.821+0000 7f4910ae8700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f490c060280 0x7f490c106780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:20:51.826 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:51.821+0000 7f4910ae8700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f490c060280 0x7f490c106780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:38314/0 (socket says 172.21.15.132:38314) 2024-02-20T00:20:51.826 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:51.821+0000 7f4910ae8700 1 -- 172.21.15.132:0/2974562530 learned_addr learned my addr 172.21.15.132:0/2974562530 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:20:51.826 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:51.821+0000 7f4910ae8700 1 -- 172.21.15.132:0/2974562530 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4900005040 con 0x7f490c060280 2024-02-20T00:20:51.826 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:51.822+0000 7f4910ae8700 1 --2- 172.21.15.132:0/2974562530 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f490c060280 0x7f490c106780 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0x7f49000089f0 tx=0x7f490000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:20:51.826 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:51.822+0000 7f49097fa700 1 -- 172.21.15.132:0/2974562530 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4900009320 con 0x7f490c060280 2024-02-20T00:20:51.826 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:51.822+0000 7f4912d4c700 1 -- 172.21.15.132:0/2974562530 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f490c106e70 con 0x7f490c060280 2024-02-20T00:20:51.827 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:51.822+0000 7f49097fa700 1 -- 172.21.15.132:0/2974562530 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4900013380 con 0x7f490c060280 2024-02-20T00:20:51.827 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:51.822+0000 7f49097fa700 1 -- 172.21.15.132:0/2974562530 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4900014e20 con 0x7f490c060280 2024-02-20T00:20:51.827 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:51.822+0000 7f4912d4c700 1 -- 172.21.15.132:0/2974562530 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f490c107300 con 0x7f490c060280 2024-02-20T00:20:51.827 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:51.823+0000 7f49097fa700 1 -- 172.21.15.132:0/2974562530 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f49000263a0 con 0x7f490c060280 2024-02-20T00:20:51.827 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:51.823+0000 7f49097fa700 1 --2- 172.21.15.132:0/2974562530 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f48fc02ff60 0x7f48fc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:20:51.827 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:51.823+0000 7f49097fa700 1 -- 172.21.15.132:0/2974562530 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4900040e90 con 0x7f490c060280 2024-02-20T00:20:51.827 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:51.823+0000 7f4912d4c700 1 -- 172.21.15.132:0/2974562530 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f490c045ad0 con 0x7f490c060280 2024-02-20T00:20:51.827 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:51.823+0000 7f490bfff700 1 --2- 172.21.15.132:0/2974562530 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f48fc02ff60 0x7f48fc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:20:51.827 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:51.823+0000 7f490bfff700 1 --2- 172.21.15.132:0/2974562530 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f48fc02ff60 0x7f48fc032420 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f48f80096a0 tx=0x7f48f8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:20:51.830 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:51.828+0000 7f49097fa700 1 -- 172.21.15.132:0/2974562530 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f49000672b0 con 0x7f490c060280 2024-02-20T00:20:52.034 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:52.031+0000 7f4912d4c700 1 -- 172.21.15.132:0/2974562530 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f490c108020 con 0x7f490c060280 2024-02-20T00:20:52.035 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:52.032+0000 7f49097fa700 1 -- 172.21.15.132:0/2974562530 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f490002b020 con 0x7f490c060280 2024-02-20T00:20:52.035 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:20:52.035 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:20:52.036 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:52.033+0000 7f4912d4c700 1 -- 172.21.15.132:0/2974562530 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f48fc02ff60 msgr2=0x7f48fc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:20:52.036 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:52.033+0000 7f4912d4c700 1 --2- 172.21.15.132:0/2974562530 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f48fc02ff60 0x7f48fc032420 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f48f80096a0 tx=0x7f48f8006b40).stop 2024-02-20T00:20:52.036 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:52.033+0000 7f4912d4c700 1 -- 172.21.15.132:0/2974562530 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f490c060280 msgr2=0x7f490c106780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:20:52.036 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:52.033+0000 7f4912d4c700 1 --2- 172.21.15.132:0/2974562530 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f490c060280 0x7f490c106780 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0x7f49000089f0 tx=0x7f490000b040).stop 2024-02-20T00:20:52.036 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:52.034+0000 7f4912d4c700 1 -- 172.21.15.132:0/2974562530 shutdown_connections 2024-02-20T00:20:52.037 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:52.034+0000 7f4912d4c700 1 --2- 172.21.15.132:0/2974562530 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f48fc02ff60 0x7f48fc032420 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:20:52.037 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:52.034+0000 7f4912d4c700 1 --2- 172.21.15.132:0/2974562530 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f490c060280 0x7f490c106780 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:20:52.037 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:52.034+0000 7f4912d4c700 1 -- 172.21.15.132:0/2974562530 >> 172.21.15.132:0/2974562530 conn(0x7f490c0f0b70 msgr2=0x7f490c0f15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:20:52.037 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:52.034+0000 7f4912d4c700 1 -- 172.21.15.132:0/2974562530 shutdown_connections 2024-02-20T00:20:52.037 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:52.034+0000 7f4912d4c700 1 -- 172.21.15.132:0/2974562530 wait complete. 2024-02-20T00:20:52.037 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:20:53.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:20:52 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:20:51.476797+0000 mgr.smithi084.bmqrnf (mgr.14162) 338 : cluster [DBG] pgmap v322: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:20:53.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:20:52 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:20:52.033707+0000 mon.smithi084 (mon.0) 319 : audit [DBG] from='client.? 172.21.15.132:0/2974562530' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:20:53.696 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:20:53.696 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:20:55.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:20:54 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:20:53.477176+0000 mgr.smithi084.bmqrnf (mgr.14162) 339 : cluster [DBG] pgmap v323: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:20:55.651 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:55.647+0000 7f5a77393700 1 -- 172.21.15.132:0/2852103509 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5a700f6510 msgr2=0x7f5a700f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:20:55.651 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:55.647+0000 7f5a77393700 1 --2- 172.21.15.132:0/2852103509 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5a700f6510 0x7f5a700f6930 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 rx=0x7f5a60009fb0 tx=0x7f5a60009b30).stop 2024-02-20T00:20:55.651 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:55.647+0000 7f5a77393700 1 -- 172.21.15.132:0/2852103509 shutdown_connections 2024-02-20T00:20:55.652 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:55.647+0000 7f5a77393700 1 --2- 172.21.15.132:0/2852103509 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5a700f6510 0x7f5a700f6930 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:20:55.652 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:55.647+0000 7f5a77393700 1 -- 172.21.15.132:0/2852103509 >> 172.21.15.132:0/2852103509 conn(0x7f5a700f1c70 msgr2=0x7f5a700f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:20:55.652 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:55.647+0000 7f5a77393700 1 -- 172.21.15.132:0/2852103509 shutdown_connections 2024-02-20T00:20:55.652 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:55.647+0000 7f5a77393700 1 -- 172.21.15.132:0/2852103509 wait complete. 2024-02-20T00:20:55.652 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:55.648+0000 7f5a77393700 1 Processor -- start 2024-02-20T00:20:55.652 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:55.648+0000 7f5a77393700 1 -- start start 2024-02-20T00:20:55.652 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:55.648+0000 7f5a77393700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5a700f6510 0x7f5a701067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:20:55.652 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:55.648+0000 7f5a77393700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a70106cd0 con 0x7f5a700f6510 2024-02-20T00:20:55.652 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:55.648+0000 7f5a7512f700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5a700f6510 0x7f5a701067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:20:55.652 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:55.648+0000 7f5a7512f700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5a700f6510 0x7f5a701067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:38326/0 (socket says 172.21.15.132:38326) 2024-02-20T00:20:55.652 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:55.648+0000 7f5a7512f700 1 -- 172.21.15.132:0/3354997806 learned_addr learned my addr 172.21.15.132:0/3354997806 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:20:55.652 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:55.648+0000 7f5a7512f700 1 -- 172.21.15.132:0/3354997806 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a60005040 con 0x7f5a700f6510 2024-02-20T00:20:55.653 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:55.648+0000 7f5a7512f700 1 --2- 172.21.15.132:0/3354997806 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5a700f6510 0x7f5a701067e0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 rx=0x7f5a60005010 tx=0x7f5a6000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:20:55.653 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:55.649+0000 7f5a65ffb700 1 -- 172.21.15.132:0/3354997806 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5a60009320 con 0x7f5a700f6510 2024-02-20T00:20:55.653 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:55.649+0000 7f5a77393700 1 -- 172.21.15.132:0/3354997806 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a70106ed0 con 0x7f5a700f6510 2024-02-20T00:20:55.653 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:55.649+0000 7f5a65ffb700 1 -- 172.21.15.132:0/3354997806 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5a60013380 con 0x7f5a700f6510 2024-02-20T00:20:55.653 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:55.649+0000 7f5a65ffb700 1 -- 172.21.15.132:0/3354997806 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5a60014e50 con 0x7f5a700f6510 2024-02-20T00:20:55.653 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:55.649+0000 7f5a77393700 1 -- 172.21.15.132:0/3354997806 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a70107360 con 0x7f5a700f6510 2024-02-20T00:20:55.654 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:55.650+0000 7f5a65ffb700 1 -- 172.21.15.132:0/3354997806 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5a600095f0 con 0x7f5a700f6510 2024-02-20T00:20:55.654 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:55.650+0000 7f5a77393700 1 -- 172.21.15.132:0/3354997806 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a70045ad0 con 0x7f5a700f6510 2024-02-20T00:20:55.654 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:55.650+0000 7f5a65ffb700 1 --2- 172.21.15.132:0/3354997806 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f5a5c02ff10 0x7f5a5c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:20:55.654 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:55.650+0000 7f5a65ffb700 1 -- 172.21.15.132:0/3354997806 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5a60040f80 con 0x7f5a700f6510 2024-02-20T00:20:55.654 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:55.650+0000 7f5a7492e700 1 --2- 172.21.15.132:0/3354997806 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f5a5c02ff10 0x7f5a5c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:20:55.654 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:55.650+0000 7f5a7492e700 1 --2- 172.21.15.132:0/3354997806 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f5a5c02ff10 0x7f5a5c0323d0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7f5a6c0096a0 tx=0x7f5a6c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:20:55.656 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:55.654+0000 7f5a65ffb700 1 -- 172.21.15.132:0/3354997806 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5a60014420 con 0x7f5a700f6510 2024-02-20T00:20:55.860 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:55.858+0000 7f5a77393700 1 -- 172.21.15.132:0/3354997806 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5a70108210 con 0x7f5a700f6510 2024-02-20T00:20:55.861 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:55.859+0000 7f5a65ffb700 1 -- 172.21.15.132:0/3354997806 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5a60010230 con 0x7f5a700f6510 2024-02-20T00:20:55.861 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:20:55.861 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:20:55.863 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:55.860+0000 7f5a77393700 1 -- 172.21.15.132:0/3354997806 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f5a5c02ff10 msgr2=0x7f5a5c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:20:55.863 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:55.860+0000 7f5a77393700 1 --2- 172.21.15.132:0/3354997806 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f5a5c02ff10 0x7f5a5c0323d0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7f5a6c0096a0 tx=0x7f5a6c006b40).stop 2024-02-20T00:20:55.863 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:55.860+0000 7f5a77393700 1 -- 172.21.15.132:0/3354997806 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5a700f6510 msgr2=0x7f5a701067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:20:55.863 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:55.860+0000 7f5a77393700 1 --2- 172.21.15.132:0/3354997806 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5a700f6510 0x7f5a701067e0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 rx=0x7f5a60005010 tx=0x7f5a6000b040).stop 2024-02-20T00:20:55.863 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:55.860+0000 7f5a77393700 1 -- 172.21.15.132:0/3354997806 shutdown_connections 2024-02-20T00:20:55.863 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:55.860+0000 7f5a77393700 1 --2- 172.21.15.132:0/3354997806 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f5a5c02ff10 0x7f5a5c0323d0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:20:55.863 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:55.860+0000 7f5a77393700 1 --2- 172.21.15.132:0/3354997806 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5a700f6510 0x7f5a701067e0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:20:55.863 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:55.861+0000 7f5a77393700 1 -- 172.21.15.132:0/3354997806 >> 172.21.15.132:0/3354997806 conn(0x7f5a700f1c70 msgr2=0x7f5a700f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:20:55.863 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:55.861+0000 7f5a77393700 1 -- 172.21.15.132:0/3354997806 shutdown_connections 2024-02-20T00:20:55.864 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:55.861+0000 7f5a77393700 1 -- 172.21.15.132:0/3354997806 wait complete. 2024-02-20T00:20:55.864 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:20:56.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:20:55 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:20:55.860500+0000 mon.smithi084 (mon.0) 320 : audit [DBG] from='client.? 172.21.15.132:0/3354997806' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:20:57.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:20:56 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:20:55.477491+0000 mgr.smithi084.bmqrnf (mgr.14162) 340 : cluster [DBG] pgmap v324: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:20:57.606 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:20:57.607 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:20:58.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:20:57 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:20:56.984096+0000 mon.smithi084 (mon.0) 321 : audit [INF] from='mgr.14162 172.21.15.84:0/2152933216' entity='mgr.smithi084.bmqrnf' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi084.bmqrnf/mirror_snapshot_schedule"}]: dispatch 2024-02-20T00:20:58.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:20:57 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:20:56.988118+0000 mon.smithi084 (mon.0) 322 : audit [INF] from='mgr.14162 172.21.15.84:0/2152933216' entity='mgr.smithi084.bmqrnf' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi084.bmqrnf/trash_purge_schedule"}]: dispatch 2024-02-20T00:20:59.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:20:58 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:20:57.477819+0000 mgr.smithi084.bmqrnf (mgr.14162) 341 : cluster [DBG] pgmap v325: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:20:59.507 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:59.503+0000 7f8680211700 1 -- 172.21.15.132:0/2765690599 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f86780f5870 msgr2=0x7f86780f5c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:20:59.507 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:59.503+0000 7f8680211700 1 --2- 172.21.15.132:0/2765690599 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f86780f5870 0x7f86780f5c90 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 rx=0x7f8674009fb0 tx=0x7f8674009b30).stop 2024-02-20T00:20:59.508 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:59.503+0000 7f8680211700 1 -- 172.21.15.132:0/2765690599 shutdown_connections 2024-02-20T00:20:59.508 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:59.503+0000 7f8680211700 1 --2- 172.21.15.132:0/2765690599 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f86780f5870 0x7f86780f5c90 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:20:59.508 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:59.503+0000 7f8680211700 1 -- 172.21.15.132:0/2765690599 >> 172.21.15.132:0/2765690599 conn(0x7f86780f02e0 msgr2=0x7f86780f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:20:59.508 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:59.503+0000 7f8680211700 1 -- 172.21.15.132:0/2765690599 shutdown_connections 2024-02-20T00:20:59.508 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:59.503+0000 7f8680211700 1 -- 172.21.15.132:0/2765690599 wait complete. 2024-02-20T00:20:59.508 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:59.503+0000 7f8680211700 1 Processor -- start 2024-02-20T00:20:59.508 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:59.503+0000 7f8680211700 1 -- start start 2024-02-20T00:20:59.508 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:59.504+0000 7f8680211700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f86780f5870 0x7f867810ce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:20:59.508 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:59.504+0000 7f8680211700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f867810d310 con 0x7f86780f5870 2024-02-20T00:20:59.508 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:59.504+0000 7f867dfad700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f86780f5870 0x7f867810ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:20:59.508 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:59.504+0000 7f867dfad700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f86780f5870 0x7f867810ce20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:55812/0 (socket says 172.21.15.132:55812) 2024-02-20T00:20:59.508 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:59.504+0000 7f867dfad700 1 -- 172.21.15.132:0/1798540969 learned_addr learned my addr 172.21.15.132:0/1798540969 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:20:59.508 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:59.504+0000 7f867dfad700 1 -- 172.21.15.132:0/1798540969 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8674005040 con 0x7f86780f5870 2024-02-20T00:20:59.509 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:59.504+0000 7f867dfad700 1 --2- 172.21.15.132:0/1798540969 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f86780f5870 0x7f867810ce20 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 rx=0x7f86740089f0 tx=0x7f867400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:20:59.509 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:59.504+0000 7f866e7fc700 1 -- 172.21.15.132:0/1798540969 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8674009320 con 0x7f86780f5870 2024-02-20T00:20:59.509 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:59.505+0000 7f8680211700 1 -- 172.21.15.132:0/1798540969 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f867810d510 con 0x7f86780f5870 2024-02-20T00:20:59.509 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:59.505+0000 7f866e7fc700 1 -- 172.21.15.132:0/1798540969 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8674013380 con 0x7f86780f5870 2024-02-20T00:20:59.509 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:59.505+0000 7f866e7fc700 1 -- 172.21.15.132:0/1798540969 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8674014e20 con 0x7f86780f5870 2024-02-20T00:20:59.509 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:59.505+0000 7f8680211700 1 -- 172.21.15.132:0/1798540969 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f867810e120 con 0x7f86780f5870 2024-02-20T00:20:59.510 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:59.505+0000 7f8680211700 1 -- 172.21.15.132:0/1798540969 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f867810d9a0 con 0x7f86780f5870 2024-02-20T00:20:59.510 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:59.505+0000 7f866e7fc700 1 -- 172.21.15.132:0/1798540969 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f86740095f0 con 0x7f86780f5870 2024-02-20T00:20:59.510 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:59.506+0000 7f866e7fc700 1 --2- 172.21.15.132:0/1798540969 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f866402ff10 0x7f86640323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:20:59.510 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:59.506+0000 7f866e7fc700 1 -- 172.21.15.132:0/1798540969 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8674040f10 con 0x7f86780f5870 2024-02-20T00:20:59.510 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:59.506+0000 7f867d7ac700 1 --2- 172.21.15.132:0/1798540969 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f866402ff10 0x7f86640323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:20:59.510 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:59.506+0000 7f867d7ac700 1 --2- 172.21.15.132:0/1798540969 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f866402ff10 0x7f86640323d0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f86680096a0 tx=0x7f8668006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:20:59.512 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:59.511+0000 7f866e7fc700 1 -- 172.21.15.132:0/1798540969 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f867810d9a0 con 0x7f86780f5870 2024-02-20T00:20:59.716 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:59.713+0000 7f8680211700 1 -- 172.21.15.132:0/1798540969 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f867810d9a0 con 0x7f86780f5870 2024-02-20T00:20:59.716 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:59.714+0000 7f866e7fc700 1 -- 172.21.15.132:0/1798540969 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f867810d9a0 con 0x7f86780f5870 2024-02-20T00:20:59.717 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:20:59.717 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:20:59.719 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:59.716+0000 7f8680211700 1 -- 172.21.15.132:0/1798540969 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f866402ff10 msgr2=0x7f86640323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:20:59.719 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:59.716+0000 7f8680211700 1 --2- 172.21.15.132:0/1798540969 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f866402ff10 0x7f86640323d0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f86680096a0 tx=0x7f8668006b40).stop 2024-02-20T00:20:59.719 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:59.716+0000 7f8680211700 1 -- 172.21.15.132:0/1798540969 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f86780f5870 msgr2=0x7f867810ce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:20:59.719 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:59.716+0000 7f8680211700 1 --2- 172.21.15.132:0/1798540969 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f86780f5870 0x7f867810ce20 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 rx=0x7f86740089f0 tx=0x7f867400b040).stop 2024-02-20T00:20:59.719 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:59.716+0000 7f8680211700 1 -- 172.21.15.132:0/1798540969 shutdown_connections 2024-02-20T00:20:59.719 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:59.716+0000 7f8680211700 1 --2- 172.21.15.132:0/1798540969 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f866402ff10 0x7f86640323d0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:20:59.719 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:59.716+0000 7f8680211700 1 --2- 172.21.15.132:0/1798540969 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f86780f5870 0x7f867810ce20 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:20:59.719 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:59.716+0000 7f8680211700 1 -- 172.21.15.132:0/1798540969 >> 172.21.15.132:0/1798540969 conn(0x7f86780f02e0 msgr2=0x7f86780f26d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:20:59.719 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:59.716+0000 7f8680211700 1 -- 172.21.15.132:0/1798540969 shutdown_connections 2024-02-20T00:20:59.719 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:20:59.716+0000 7f8680211700 1 -- 172.21.15.132:0/1798540969 wait complete. 2024-02-20T00:20:59.719 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:21:00.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:20:59 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:20:59.715799+0000 mon.smithi084 (mon.0) 323 : audit [DBG] from='client.? 172.21.15.132:0/1798540969' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:21:01.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:21:00 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:20:59.478204+0000 mgr.smithi084.bmqrnf (mgr.14162) 342 : cluster [DBG] pgmap v326: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:21:01.379 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:21:01.379 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:21:03.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:21:02 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:21:01.478532+0000 mgr.smithi084.bmqrnf (mgr.14162) 343 : cluster [DBG] pgmap v327: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:21:03.494 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:03.493+0000 7f6fad64e700 1 -- 172.21.15.132:0/958270447 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6fa80f5410 msgr2=0x7f6fa80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:21:03.494 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:03.493+0000 7f6fad64e700 1 --2- 172.21.15.132:0/958270447 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6fa80f5410 0x7f6fa80f5830 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 rx=0x7f6f98009fb0 tx=0x7f6f98009b30).stop 2024-02-20T00:21:03.494 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:03.493+0000 7f6fad64e700 1 -- 172.21.15.132:0/958270447 shutdown_connections 2024-02-20T00:21:03.495 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:03.493+0000 7f6fad64e700 1 --2- 172.21.15.132:0/958270447 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6fa80f5410 0x7f6fa80f5830 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:21:03.495 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:03.493+0000 7f6fad64e700 1 -- 172.21.15.132:0/958270447 >> 172.21.15.132:0/958270447 conn(0x7f6fa80f0b70 msgr2=0x7f6fa80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:21:03.495 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:03.493+0000 7f6fad64e700 1 -- 172.21.15.132:0/958270447 shutdown_connections 2024-02-20T00:21:03.495 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:03.493+0000 7f6fad64e700 1 -- 172.21.15.132:0/958270447 wait complete. 2024-02-20T00:21:03.497 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:03.493+0000 7f6fad64e700 1 Processor -- start 2024-02-20T00:21:03.497 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:03.493+0000 7f6fad64e700 1 -- start start 2024-02-20T00:21:03.497 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:03.493+0000 7f6fad64e700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6fa80f5410 0x7f6fa8106810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:21:03.497 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:03.493+0000 7f6fad64e700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fa8106d00 con 0x7f6fa80f5410 2024-02-20T00:21:03.497 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:03.494+0000 7f6fa6ffd700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6fa80f5410 0x7f6fa8106810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:21:03.497 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:03.494+0000 7f6fa6ffd700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6fa80f5410 0x7f6fa8106810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:55826/0 (socket says 172.21.15.132:55826) 2024-02-20T00:21:03.497 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:03.494+0000 7f6fa6ffd700 1 -- 172.21.15.132:0/739170042 learned_addr learned my addr 172.21.15.132:0/739170042 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:21:03.497 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:03.494+0000 7f6fa6ffd700 1 -- 172.21.15.132:0/739170042 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f98005040 con 0x7f6fa80f5410 2024-02-20T00:21:03.498 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:03.494+0000 7f6fa6ffd700 1 --2- 172.21.15.132:0/739170042 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6fa80f5410 0x7f6fa8106810 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 rx=0x7f6f98005010 tx=0x7f6f9800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:21:03.498 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:03.494+0000 7f6f8f7fe700 1 -- 172.21.15.132:0/739170042 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6f98009320 con 0x7f6fa80f5410 2024-02-20T00:21:03.498 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:03.494+0000 7f6f8f7fe700 1 -- 172.21.15.132:0/739170042 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6f98013380 con 0x7f6fa80f5410 2024-02-20T00:21:03.498 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:03.494+0000 7f6fad64e700 1 -- 172.21.15.132:0/739170042 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fa8106f00 con 0x7f6fa80f5410 2024-02-20T00:21:03.498 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:03.494+0000 7f6f8f7fe700 1 -- 172.21.15.132:0/739170042 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6f98014e50 con 0x7f6fa80f5410 2024-02-20T00:21:03.498 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:03.494+0000 7f6fad64e700 1 -- 172.21.15.132:0/739170042 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fa8107390 con 0x7f6fa80f5410 2024-02-20T00:21:03.498 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:03.495+0000 7f6f8f7fe700 1 -- 172.21.15.132:0/739170042 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6f980263a0 con 0x7f6fa80f5410 2024-02-20T00:21:03.498 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:03.495+0000 7f6f8f7fe700 1 --2- 172.21.15.132:0/739170042 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f6f9002ff10 0x7f6f900323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:21:03.498 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:03.495+0000 7f6f8f7fe700 1 -- 172.21.15.132:0/739170042 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6f98040fa0 con 0x7f6fa80f5410 2024-02-20T00:21:03.498 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:03.495+0000 7f6fad64e700 1 -- 172.21.15.132:0/739170042 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fa8045ad0 con 0x7f6fa80f5410 2024-02-20T00:21:03.498 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:03.496+0000 7f6fa67fc700 1 --2- 172.21.15.132:0/739170042 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f6f9002ff10 0x7f6f900323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:21:03.498 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:03.496+0000 7f6fa67fc700 1 --2- 172.21.15.132:0/739170042 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f6f9002ff10 0x7f6f900323d0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7f6f9c0096a0 tx=0x7f6f9c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:21:03.501 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:03.500+0000 7f6f8f7fe700 1 -- 172.21.15.132:0/739170042 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6f98012920 con 0x7f6fa80f5410 2024-02-20T00:21:03.709 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:03.707+0000 7f6fad64e700 1 -- 172.21.15.132:0/739170042 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6fa81080b0 con 0x7f6fa80f5410 2024-02-20T00:21:03.709 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:03.708+0000 7f6f8f7fe700 1 -- 172.21.15.132:0/739170042 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6f9802b020 con 0x7f6fa80f5410 2024-02-20T00:21:03.710 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:21:03.710 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:21:03.711 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:03.710+0000 7f6fad64e700 1 -- 172.21.15.132:0/739170042 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f6f9002ff10 msgr2=0x7f6f900323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:21:03.711 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:03.710+0000 7f6fad64e700 1 --2- 172.21.15.132:0/739170042 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f6f9002ff10 0x7f6f900323d0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7f6f9c0096a0 tx=0x7f6f9c006b40).stop 2024-02-20T00:21:03.712 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:03.710+0000 7f6fad64e700 1 -- 172.21.15.132:0/739170042 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6fa80f5410 msgr2=0x7f6fa8106810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:21:03.712 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:03.710+0000 7f6fad64e700 1 --2- 172.21.15.132:0/739170042 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6fa80f5410 0x7f6fa8106810 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 rx=0x7f6f98005010 tx=0x7f6f9800b040).stop 2024-02-20T00:21:03.712 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:03.710+0000 7f6fad64e700 1 -- 172.21.15.132:0/739170042 shutdown_connections 2024-02-20T00:21:03.712 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:03.710+0000 7f6fad64e700 1 --2- 172.21.15.132:0/739170042 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f6f9002ff10 0x7f6f900323d0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:21:03.712 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:03.710+0000 7f6fad64e700 1 --2- 172.21.15.132:0/739170042 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6fa80f5410 0x7f6fa8106810 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:21:03.712 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:03.710+0000 7f6fad64e700 1 -- 172.21.15.132:0/739170042 >> 172.21.15.132:0/739170042 conn(0x7f6fa80f0b70 msgr2=0x7f6fa805fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:21:03.712 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:03.710+0000 7f6fad64e700 1 -- 172.21.15.132:0/739170042 shutdown_connections 2024-02-20T00:21:03.712 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:03.710+0000 7f6fad64e700 1 -- 172.21.15.132:0/739170042 wait complete. 2024-02-20T00:21:03.713 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:21:04.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:21:03 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:21:03.708660+0000 mon.smithi084 (mon.0) 324 : audit [DBG] from='client.? 172.21.15.132:0/739170042' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:21:05.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:21:04 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:21:03.478844+0000 mgr.smithi084.bmqrnf (mgr.14162) 344 : cluster [DBG] pgmap v328: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:21:05.433 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:21:05.433 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:21:07.247 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:07.245+0000 7f688949d700 1 -- 172.21.15.132:0/3192634378 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f68840f78e0 msgr2=0x7f68840f7d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:21:07.247 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:07.245+0000 7f688949d700 1 --2- 172.21.15.132:0/3192634378 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f68840f78e0 0x7f68840f7d00 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 rx=0x7f6874009fb0 tx=0x7f6874009b30).stop 2024-02-20T00:21:07.247 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:07.245+0000 7f688949d700 1 -- 172.21.15.132:0/3192634378 shutdown_connections 2024-02-20T00:21:07.247 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:07.245+0000 7f688949d700 1 --2- 172.21.15.132:0/3192634378 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f68840f78e0 0x7f68840f7d00 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:21:07.247 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:07.245+0000 7f688949d700 1 -- 172.21.15.132:0/3192634378 >> 172.21.15.132:0/3192634378 conn(0x7f68840f0120 msgr2=0x7f68840f2540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:21:07.248 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:07.245+0000 7f688949d700 1 -- 172.21.15.132:0/3192634378 shutdown_connections 2024-02-20T00:21:07.248 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:07.245+0000 7f688949d700 1 -- 172.21.15.132:0/3192634378 wait complete. 2024-02-20T00:21:07.248 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:07.246+0000 7f688949d700 1 Processor -- start 2024-02-20T00:21:07.248 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:07.246+0000 7f688949d700 1 -- start start 2024-02-20T00:21:07.249 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:07.246+0000 7f688949d700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f68840f78e0 0x7f6884106600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:21:07.249 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:07.246+0000 7f688949d700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6884106af0 con 0x7f68840f78e0 2024-02-20T00:21:07.250 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:07.246+0000 7f6883fff700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f68840f78e0 0x7f6884106600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:21:07.250 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:07.246+0000 7f6883fff700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f68840f78e0 0x7f6884106600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:55844/0 (socket says 172.21.15.132:55844) 2024-02-20T00:21:07.250 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:07.247+0000 7f6883fff700 1 -- 172.21.15.132:0/1181958364 learned_addr learned my addr 172.21.15.132:0/1181958364 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:21:07.250 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:07.247+0000 7f6883fff700 1 -- 172.21.15.132:0/1181958364 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6874005040 con 0x7f68840f78e0 2024-02-20T00:21:07.250 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:07.247+0000 7f6883fff700 1 --2- 172.21.15.132:0/1181958364 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f68840f78e0 0x7f6884106600 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 rx=0x7f68740089f0 tx=0x7f687400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:21:07.250 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:07.247+0000 7f6880ff9700 1 -- 172.21.15.132:0/1181958364 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6874009320 con 0x7f68840f78e0 2024-02-20T00:21:07.250 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:07.247+0000 7f688949d700 1 -- 172.21.15.132:0/1181958364 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6884106cf0 con 0x7f68840f78e0 2024-02-20T00:21:07.250 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:07.247+0000 7f6880ff9700 1 -- 172.21.15.132:0/1181958364 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6874013380 con 0x7f68840f78e0 2024-02-20T00:21:07.250 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:07.247+0000 7f6880ff9700 1 -- 172.21.15.132:0/1181958364 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6874014e20 con 0x7f68840f78e0 2024-02-20T00:21:07.250 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:07.247+0000 7f688949d700 1 -- 172.21.15.132:0/1181958364 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6884107180 con 0x7f68840f78e0 2024-02-20T00:21:07.250 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:07.248+0000 7f688949d700 1 -- 172.21.15.132:0/1181958364 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6884045ad0 con 0x7f68840f78e0 2024-02-20T00:21:07.250 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:07.248+0000 7f6880ff9700 1 -- 172.21.15.132:0/1181958364 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f68740095f0 con 0x7f68840f78e0 2024-02-20T00:21:07.250 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:07.248+0000 7f6880ff9700 1 --2- 172.21.15.132:0/1181958364 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f686c02ff10 0x7f686c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:21:07.251 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:07.248+0000 7f6880ff9700 1 -- 172.21.15.132:0/1181958364 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6874040ef0 con 0x7f68840f78e0 2024-02-20T00:21:07.251 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:07.248+0000 7f68837fe700 1 --2- 172.21.15.132:0/1181958364 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f686c02ff10 0x7f686c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:21:07.251 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:07.249+0000 7f68837fe700 1 --2- 172.21.15.132:0/1181958364 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f686c02ff10 0x7f686c0323d0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f68780096a0 tx=0x7f6878006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:21:07.254 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:07.253+0000 7f6880ff9700 1 -- 172.21.15.132:0/1181958364 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6874014420 con 0x7f68840f78e0 2024-02-20T00:21:07.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:21:07 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:21:05.479214+0000 mgr.smithi084.bmqrnf (mgr.14162) 345 : cluster [DBG] pgmap v329: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:21:07.457 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:07.456+0000 7f688949d700 1 -- 172.21.15.132:0/1181958364 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6884108030 con 0x7f68840f78e0 2024-02-20T00:21:07.458 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:07.457+0000 7f6880ff9700 1 -- 172.21.15.132:0/1181958364 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6874010230 con 0x7f68840f78e0 2024-02-20T00:21:07.458 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:21:07.458 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:21:07.460 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:07.458+0000 7f688949d700 1 -- 172.21.15.132:0/1181958364 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f686c02ff10 msgr2=0x7f686c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:21:07.460 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:07.458+0000 7f688949d700 1 --2- 172.21.15.132:0/1181958364 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f686c02ff10 0x7f686c0323d0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f68780096a0 tx=0x7f6878006b40).stop 2024-02-20T00:21:07.460 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:07.458+0000 7f688949d700 1 -- 172.21.15.132:0/1181958364 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f68840f78e0 msgr2=0x7f6884106600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:21:07.460 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:07.458+0000 7f688949d700 1 --2- 172.21.15.132:0/1181958364 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f68840f78e0 0x7f6884106600 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 rx=0x7f68740089f0 tx=0x7f687400b040).stop 2024-02-20T00:21:07.460 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:07.458+0000 7f688949d700 1 -- 172.21.15.132:0/1181958364 shutdown_connections 2024-02-20T00:21:07.460 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:07.458+0000 7f688949d700 1 --2- 172.21.15.132:0/1181958364 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f686c02ff10 0x7f686c0323d0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:21:07.460 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:07.458+0000 7f688949d700 1 --2- 172.21.15.132:0/1181958364 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f68840f78e0 0x7f6884106600 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:21:07.460 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:07.458+0000 7f688949d700 1 -- 172.21.15.132:0/1181958364 >> 172.21.15.132:0/1181958364 conn(0x7f68840f0120 msgr2=0x7f68840f0b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:21:07.461 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:07.459+0000 7f688949d700 1 -- 172.21.15.132:0/1181958364 shutdown_connections 2024-02-20T00:21:07.463 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:07.459+0000 7f688949d700 1 -- 172.21.15.132:0/1181958364 wait complete. 2024-02-20T00:21:07.463 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:21:08.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:21:08 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:21:07.457533+0000 mon.smithi084 (mon.0) 325 : audit [DBG] from='client.? 172.21.15.132:0/1181958364' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:21:09.209 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:21:09.210 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:21:09.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:21:09 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:21:07.479621+0000 mgr.smithi084.bmqrnf (mgr.14162) 346 : cluster [DBG] pgmap v330: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:21:11.208 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:11.206+0000 7f7bdd0bc700 1 -- 172.21.15.132:0/2441222989 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7bd80f6510 msgr2=0x7f7bd80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:21:11.209 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:11.206+0000 7f7bdd0bc700 1 --2- 172.21.15.132:0/2441222989 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7bd80f6510 0x7f7bd80f6930 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 rx=0x7f7bc0009fb0 tx=0x7f7bc0009b30).stop 2024-02-20T00:21:11.209 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:11.206+0000 7f7bdd0bc700 1 -- 172.21.15.132:0/2441222989 shutdown_connections 2024-02-20T00:21:11.209 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:11.206+0000 7f7bdd0bc700 1 --2- 172.21.15.132:0/2441222989 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7bd80f6510 0x7f7bd80f6930 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:21:11.209 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:11.206+0000 7f7bdd0bc700 1 -- 172.21.15.132:0/2441222989 >> 172.21.15.132:0/2441222989 conn(0x7f7bd80f1c70 msgr2=0x7f7bd80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:21:11.209 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:11.206+0000 7f7bdd0bc700 1 -- 172.21.15.132:0/2441222989 shutdown_connections 2024-02-20T00:21:11.209 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:11.206+0000 7f7bdd0bc700 1 -- 172.21.15.132:0/2441222989 wait complete. 2024-02-20T00:21:11.209 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:11.206+0000 7f7bdd0bc700 1 Processor -- start 2024-02-20T00:21:11.209 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:11.207+0000 7f7bdd0bc700 1 -- start start 2024-02-20T00:21:11.209 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:11.207+0000 7f7bdd0bc700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7bd80f6510 0x7f7bd806b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:21:11.209 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:11.207+0000 7f7bdd0bc700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bd8069e20 con 0x7f7bd80f6510 2024-02-20T00:21:11.209 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:11.207+0000 7f7bd6d9d700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7bd80f6510 0x7f7bd806b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:21:11.209 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:11.207+0000 7f7bd6d9d700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7bd80f6510 0x7f7bd806b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:56032/0 (socket says 172.21.15.132:56032) 2024-02-20T00:21:11.210 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:11.207+0000 7f7bd6d9d700 1 -- 172.21.15.132:0/1282506948 learned_addr learned my addr 172.21.15.132:0/1282506948 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:21:11.210 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:11.207+0000 7f7bd6d9d700 1 -- 172.21.15.132:0/1282506948 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bc0005040 con 0x7f7bd80f6510 2024-02-20T00:21:11.210 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:11.207+0000 7f7bd6d9d700 1 --2- 172.21.15.132:0/1282506948 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7bd80f6510 0x7f7bd806b9a0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 rx=0x7f7bc0005010 tx=0x7f7bc000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:21:11.211 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:11.207+0000 7f7bcf7fe700 1 -- 172.21.15.132:0/1282506948 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7bc0009320 con 0x7f7bd80f6510 2024-02-20T00:21:11.211 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:11.208+0000 7f7bcf7fe700 1 -- 172.21.15.132:0/1282506948 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7bc0013380 con 0x7f7bd80f6510 2024-02-20T00:21:11.211 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:11.208+0000 7f7bdd0bc700 1 -- 172.21.15.132:0/1282506948 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bd8069fc0 con 0x7f7bd80f6510 2024-02-20T00:21:11.211 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:11.208+0000 7f7bcf7fe700 1 -- 172.21.15.132:0/1282506948 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7bc0014e50 con 0x7f7bd80f6510 2024-02-20T00:21:11.211 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:11.208+0000 7f7bdd0bc700 1 -- 172.21.15.132:0/1282506948 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bd806a450 con 0x7f7bd80f6510 2024-02-20T00:21:11.211 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:11.208+0000 7f7bcf7fe700 1 -- 172.21.15.132:0/1282506948 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7bc00095f0 con 0x7f7bd80f6510 2024-02-20T00:21:11.211 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:11.208+0000 7f7bdd0bc700 1 -- 172.21.15.132:0/1282506948 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7bd8059100 con 0x7f7bd80f6510 2024-02-20T00:21:11.212 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:11.209+0000 7f7bcf7fe700 1 --2- 172.21.15.132:0/1282506948 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f7bc402ff10 0x7f7bc40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:21:11.212 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:11.209+0000 7f7bcf7fe700 1 -- 172.21.15.132:0/1282506948 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7bc0040fa0 con 0x7f7bd80f6510 2024-02-20T00:21:11.212 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:11.209+0000 7f7bd659c700 1 --2- 172.21.15.132:0/1282506948 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f7bc402ff10 0x7f7bc40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:21:11.212 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:11.209+0000 7f7bd659c700 1 --2- 172.21.15.132:0/1282506948 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f7bc402ff10 0x7f7bc40323d0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7f7bc80096a0 tx=0x7f7bc8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:21:11.215 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:11.213+0000 7f7bcf7fe700 1 -- 172.21.15.132:0/1282506948 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7bc0014420 con 0x7f7bd80f6510 2024-02-20T00:21:11.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:21:11 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:21:09.479961+0000 mgr.smithi084.bmqrnf (mgr.14162) 347 : cluster [DBG] pgmap v331: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:21:11.420 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:11.418+0000 7f7bdd0bc700 1 -- 172.21.15.132:0/1282506948 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7bd80fc0a0 con 0x7f7bd80f6510 2024-02-20T00:21:11.420 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:11.419+0000 7f7bcf7fe700 1 -- 172.21.15.132:0/1282506948 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7bc0010230 con 0x7f7bd80f6510 2024-02-20T00:21:11.421 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:21:11.421 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:21:11.422 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:11.421+0000 7f7bdd0bc700 1 -- 172.21.15.132:0/1282506948 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f7bc402ff10 msgr2=0x7f7bc40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:21:11.422 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:11.421+0000 7f7bdd0bc700 1 --2- 172.21.15.132:0/1282506948 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f7bc402ff10 0x7f7bc40323d0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7f7bc80096a0 tx=0x7f7bc8006b40).stop 2024-02-20T00:21:11.422 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:11.421+0000 7f7bdd0bc700 1 -- 172.21.15.132:0/1282506948 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7bd80f6510 msgr2=0x7f7bd806b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:21:11.423 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:11.421+0000 7f7bdd0bc700 1 --2- 172.21.15.132:0/1282506948 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7bd80f6510 0x7f7bd806b9a0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 rx=0x7f7bc0005010 tx=0x7f7bc000b040).stop 2024-02-20T00:21:11.423 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:11.421+0000 7f7bdd0bc700 1 -- 172.21.15.132:0/1282506948 shutdown_connections 2024-02-20T00:21:11.423 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:11.421+0000 7f7bdd0bc700 1 --2- 172.21.15.132:0/1282506948 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f7bc402ff10 0x7f7bc40323d0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:21:11.423 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:11.421+0000 7f7bdd0bc700 1 --2- 172.21.15.132:0/1282506948 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7bd80f6510 0x7f7bd806b9a0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:21:11.423 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:11.421+0000 7f7bdd0bc700 1 -- 172.21.15.132:0/1282506948 >> 172.21.15.132:0/1282506948 conn(0x7f7bd80f1c70 msgr2=0x7f7bd80f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:21:11.423 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:11.421+0000 7f7bdd0bc700 1 -- 172.21.15.132:0/1282506948 shutdown_connections 2024-02-20T00:21:11.423 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:11.421+0000 7f7bdd0bc700 1 -- 172.21.15.132:0/1282506948 wait complete. 2024-02-20T00:21:11.423 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:21:12.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:21:12 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:21:11.419798+0000 mon.smithi084 (mon.0) 326 : audit [DBG] from='client.? 172.21.15.132:0/1282506948' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:21:13.103 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:21:13.103 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:21:13.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:21:13 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:21:11.480258+0000 mgr.smithi084.bmqrnf (mgr.14162) 348 : cluster [DBG] pgmap v332: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:21:15.025 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:15.023+0000 7ff3e1313700 1 -- 172.21.15.132:0/3070739605 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff3dc0f5410 msgr2=0x7ff3dc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:21:15.025 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:15.023+0000 7ff3e1313700 1 --2- 172.21.15.132:0/3070739605 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff3dc0f5410 0x7ff3dc0f5830 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 rx=0x7ff3cc009fb0 tx=0x7ff3cc009b30).stop 2024-02-20T00:21:15.026 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:15.023+0000 7ff3e1313700 1 -- 172.21.15.132:0/3070739605 shutdown_connections 2024-02-20T00:21:15.026 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:15.023+0000 7ff3e1313700 1 --2- 172.21.15.132:0/3070739605 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff3dc0f5410 0x7ff3dc0f5830 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:21:15.026 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:15.023+0000 7ff3e1313700 1 -- 172.21.15.132:0/3070739605 >> 172.21.15.132:0/3070739605 conn(0x7ff3dc0f0b70 msgr2=0x7ff3dc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:21:15.026 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:15.023+0000 7ff3e1313700 1 -- 172.21.15.132:0/3070739605 shutdown_connections 2024-02-20T00:21:15.026 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:15.023+0000 7ff3e1313700 1 -- 172.21.15.132:0/3070739605 wait complete. 2024-02-20T00:21:15.027 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:15.024+0000 7ff3e1313700 1 Processor -- start 2024-02-20T00:21:15.027 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:15.024+0000 7ff3e1313700 1 -- start start 2024-02-20T00:21:15.027 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:15.025+0000 7ff3e1313700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff3dc0f5410 0x7ff3dc106820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:21:15.027 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:15.025+0000 7ff3e1313700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3dc106d10 con 0x7ff3dc0f5410 2024-02-20T00:21:15.027 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:15.025+0000 7ff3daffd700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff3dc0f5410 0x7ff3dc106820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:21:15.028 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:15.025+0000 7ff3daffd700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff3dc0f5410 0x7ff3dc106820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:56050/0 (socket says 172.21.15.132:56050) 2024-02-20T00:21:15.028 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:15.025+0000 7ff3daffd700 1 -- 172.21.15.132:0/599135561 learned_addr learned my addr 172.21.15.132:0/599135561 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:21:15.028 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:15.025+0000 7ff3daffd700 1 -- 172.21.15.132:0/599135561 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3cc005040 con 0x7ff3dc0f5410 2024-02-20T00:21:15.028 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:15.025+0000 7ff3daffd700 1 --2- 172.21.15.132:0/599135561 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff3dc0f5410 0x7ff3dc106820 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 rx=0x7ff3cc005010 tx=0x7ff3cc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:21:15.028 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:15.025+0000 7ff3d37fe700 1 -- 172.21.15.132:0/599135561 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff3cc009320 con 0x7ff3dc0f5410 2024-02-20T00:21:15.028 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:15.025+0000 7ff3e1313700 1 -- 172.21.15.132:0/599135561 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3dc106f10 con 0x7ff3dc0f5410 2024-02-20T00:21:15.028 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:15.025+0000 7ff3d37fe700 1 -- 172.21.15.132:0/599135561 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff3cc013380 con 0x7ff3dc0f5410 2024-02-20T00:21:15.028 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:15.025+0000 7ff3d37fe700 1 -- 172.21.15.132:0/599135561 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff3cc014e50 con 0x7ff3dc0f5410 2024-02-20T00:21:15.028 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:15.025+0000 7ff3e1313700 1 -- 172.21.15.132:0/599135561 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3dc1073a0 con 0x7ff3dc0f5410 2024-02-20T00:21:15.029 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:15.026+0000 7ff3d37fe700 1 -- 172.21.15.132:0/599135561 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff3cc0095f0 con 0x7ff3dc0f5410 2024-02-20T00:21:15.029 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:15.026+0000 7ff3e1313700 1 -- 172.21.15.132:0/599135561 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3dc045ad0 con 0x7ff3dc0f5410 2024-02-20T00:21:15.029 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:15.026+0000 7ff3d37fe700 1 --2- 172.21.15.132:0/599135561 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7ff3c802ff10 0x7ff3c80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:21:15.029 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:15.027+0000 7ff3d37fe700 1 -- 172.21.15.132:0/599135561 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff3cc040e90 con 0x7ff3dc0f5410 2024-02-20T00:21:15.029 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:15.027+0000 7ff3da7fc700 1 --2- 172.21.15.132:0/599135561 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7ff3c802ff10 0x7ff3c80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:21:15.029 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:15.027+0000 7ff3da7fc700 1 --2- 172.21.15.132:0/599135561 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7ff3c802ff10 0x7ff3c80323d0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7ff3c40096a0 tx=0x7ff3c4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:21:15.036 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:15.031+0000 7ff3d37fe700 1 -- 172.21.15.132:0/599135561 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff3cc014420 con 0x7ff3dc0f5410 2024-02-20T00:21:15.235 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:15.234+0000 7ff3e1313700 1 -- 172.21.15.132:0/599135561 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff3dc108250 con 0x7ff3dc0f5410 2024-02-20T00:21:15.236 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:15.235+0000 7ff3d37fe700 1 -- 172.21.15.132:0/599135561 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff3cc010230 con 0x7ff3dc0f5410 2024-02-20T00:21:15.236 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:21:15.236 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:21:15.238 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:15.236+0000 7ff3e1313700 1 -- 172.21.15.132:0/599135561 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7ff3c802ff10 msgr2=0x7ff3c80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:21:15.238 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:15.236+0000 7ff3e1313700 1 --2- 172.21.15.132:0/599135561 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7ff3c802ff10 0x7ff3c80323d0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7ff3c40096a0 tx=0x7ff3c4006b40).stop 2024-02-20T00:21:15.238 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:15.236+0000 7ff3e1313700 1 -- 172.21.15.132:0/599135561 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff3dc0f5410 msgr2=0x7ff3dc106820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:21:15.238 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:15.236+0000 7ff3e1313700 1 --2- 172.21.15.132:0/599135561 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff3dc0f5410 0x7ff3dc106820 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 rx=0x7ff3cc005010 tx=0x7ff3cc00b040).stop 2024-02-20T00:21:15.238 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:15.236+0000 7ff3e1313700 1 -- 172.21.15.132:0/599135561 shutdown_connections 2024-02-20T00:21:15.238 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:15.236+0000 7ff3e1313700 1 --2- 172.21.15.132:0/599135561 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7ff3c802ff10 0x7ff3c80323d0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:21:15.238 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:15.236+0000 7ff3e1313700 1 --2- 172.21.15.132:0/599135561 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff3dc0f5410 0x7ff3dc106820 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:21:15.238 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:15.236+0000 7ff3e1313700 1 -- 172.21.15.132:0/599135561 >> 172.21.15.132:0/599135561 conn(0x7ff3dc0f0b70 msgr2=0x7ff3dc0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:21:15.239 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:15.236+0000 7ff3e1313700 1 -- 172.21.15.132:0/599135561 shutdown_connections 2024-02-20T00:21:15.239 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:15.236+0000 7ff3e1313700 1 -- 172.21.15.132:0/599135561 wait complete. 2024-02-20T00:21:15.239 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:21:15.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:21:15 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:21:13.480641+0000 mgr.smithi084.bmqrnf (mgr.14162) 349 : cluster [DBG] pgmap v333: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:21:16.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:21:16 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:21:15.235205+0000 mon.smithi084 (mon.0) 327 : audit [DBG] from='client.? 172.21.15.132:0/599135561' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:21:16.920 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:21:16.920 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:21:17.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:21:17 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:21:15.480875+0000 mgr.smithi084.bmqrnf (mgr.14162) 350 : cluster [DBG] pgmap v334: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:21:19.017 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:19.015+0000 7f5b210a0700 1 -- 172.21.15.132:0/3875510490 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5b1c0f5410 msgr2=0x7f5b1c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:21:19.018 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:19.015+0000 7f5b210a0700 1 --2- 172.21.15.132:0/3875510490 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5b1c0f5410 0x7f5b1c0f5830 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 rx=0x7f5b10009fb0 tx=0x7f5b10009b30).stop 2024-02-20T00:21:19.018 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:19.015+0000 7f5b210a0700 1 -- 172.21.15.132:0/3875510490 shutdown_connections 2024-02-20T00:21:19.018 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:19.015+0000 7f5b210a0700 1 --2- 172.21.15.132:0/3875510490 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5b1c0f5410 0x7f5b1c0f5830 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:21:19.018 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:19.015+0000 7f5b210a0700 1 -- 172.21.15.132:0/3875510490 >> 172.21.15.132:0/3875510490 conn(0x7f5b1c0f0b70 msgr2=0x7f5b1c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:21:19.018 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:19.015+0000 7f5b210a0700 1 -- 172.21.15.132:0/3875510490 shutdown_connections 2024-02-20T00:21:19.018 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:19.015+0000 7f5b210a0700 1 -- 172.21.15.132:0/3875510490 wait complete. 2024-02-20T00:21:19.023 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:19.016+0000 7f5b210a0700 1 Processor -- start 2024-02-20T00:21:19.023 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:19.016+0000 7f5b210a0700 1 -- start start 2024-02-20T00:21:19.024 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:19.016+0000 7f5b210a0700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5b1c0f5410 0x7f5b1c1067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:21:19.024 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:19.016+0000 7f5b210a0700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b1c106cd0 con 0x7f5b1c0f5410 2024-02-20T00:21:19.024 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:19.016+0000 7f5b1ad9d700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5b1c0f5410 0x7f5b1c1067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:21:19.024 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:19.016+0000 7f5b1ad9d700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5b1c0f5410 0x7f5b1c1067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:60278/0 (socket says 172.21.15.132:60278) 2024-02-20T00:21:19.024 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:19.016+0000 7f5b1ad9d700 1 -- 172.21.15.132:0/809592632 learned_addr learned my addr 172.21.15.132:0/809592632 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:21:19.024 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:19.017+0000 7f5b1ad9d700 1 -- 172.21.15.132:0/809592632 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b10005040 con 0x7f5b1c0f5410 2024-02-20T00:21:19.024 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:19.017+0000 7f5b1ad9d700 1 --2- 172.21.15.132:0/809592632 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5b1c0f5410 0x7f5b1c1067e0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 rx=0x7f5b10005010 tx=0x7f5b1000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:21:19.024 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:19.017+0000 7f5b037fe700 1 -- 172.21.15.132:0/809592632 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5b10009320 con 0x7f5b1c0f5410 2024-02-20T00:21:19.025 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:19.017+0000 7f5b210a0700 1 -- 172.21.15.132:0/809592632 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b1c106ed0 con 0x7f5b1c0f5410 2024-02-20T00:21:19.025 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:19.017+0000 7f5b037fe700 1 -- 172.21.15.132:0/809592632 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5b10013380 con 0x7f5b1c0f5410 2024-02-20T00:21:19.025 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:19.017+0000 7f5b037fe700 1 -- 172.21.15.132:0/809592632 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5b10014e50 con 0x7f5b1c0f5410 2024-02-20T00:21:19.025 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:19.017+0000 7f5b210a0700 1 -- 172.21.15.132:0/809592632 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b1c107360 con 0x7f5b1c0f5410 2024-02-20T00:21:19.025 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:19.018+0000 7f5b037fe700 1 -- 172.21.15.132:0/809592632 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5b100263a0 con 0x7f5b1c0f5410 2024-02-20T00:21:19.025 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:19.018+0000 7f5b210a0700 1 -- 172.21.15.132:0/809592632 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b1c045ad0 con 0x7f5b1c0f5410 2024-02-20T00:21:19.025 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:19.018+0000 7f5b037fe700 1 --2- 172.21.15.132:0/809592632 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f5b0402ff10 0x7f5b040323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:21:19.025 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:19.018+0000 7f5b037fe700 1 -- 172.21.15.132:0/809592632 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5b10040e40 con 0x7f5b1c0f5410 2024-02-20T00:21:19.025 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:19.018+0000 7f5b1a59c700 1 --2- 172.21.15.132:0/809592632 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f5b0402ff10 0x7f5b040323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:21:19.026 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:19.018+0000 7f5b1a59c700 1 --2- 172.21.15.132:0/809592632 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f5b0402ff10 0x7f5b040323d0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7f5b0c0096a0 tx=0x7f5b0c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:21:19.026 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:19.023+0000 7f5b037fe700 1 -- 172.21.15.132:0/809592632 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5b1001ddc0 con 0x7f5b1c0f5410 2024-02-20T00:21:19.229 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:19.226+0000 7f5b210a0700 1 -- 172.21.15.132:0/809592632 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5b1c0fafa0 con 0x7f5b1c0f5410 2024-02-20T00:21:19.229 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:19.227+0000 7f5b037fe700 1 -- 172.21.15.132:0/809592632 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5b1001d9d0 con 0x7f5b1c0f5410 2024-02-20T00:21:19.230 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:21:19.230 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:21:19.231 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:19.229+0000 7f5b210a0700 1 -- 172.21.15.132:0/809592632 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f5b0402ff10 msgr2=0x7f5b040323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:21:19.231 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:19.229+0000 7f5b210a0700 1 --2- 172.21.15.132:0/809592632 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f5b0402ff10 0x7f5b040323d0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7f5b0c0096a0 tx=0x7f5b0c006b40).stop 2024-02-20T00:21:19.231 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:19.229+0000 7f5b210a0700 1 -- 172.21.15.132:0/809592632 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5b1c0f5410 msgr2=0x7f5b1c1067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:21:19.231 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:19.229+0000 7f5b210a0700 1 --2- 172.21.15.132:0/809592632 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5b1c0f5410 0x7f5b1c1067e0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 rx=0x7f5b10005010 tx=0x7f5b1000b040).stop 2024-02-20T00:21:19.232 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:19.229+0000 7f5b210a0700 1 -- 172.21.15.132:0/809592632 shutdown_connections 2024-02-20T00:21:19.232 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:19.229+0000 7f5b210a0700 1 --2- 172.21.15.132:0/809592632 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f5b0402ff10 0x7f5b040323d0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:21:19.232 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:19.229+0000 7f5b210a0700 1 --2- 172.21.15.132:0/809592632 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5b1c0f5410 0x7f5b1c1067e0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:21:19.232 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:19.229+0000 7f5b210a0700 1 -- 172.21.15.132:0/809592632 >> 172.21.15.132:0/809592632 conn(0x7f5b1c0f0b70 msgr2=0x7f5b1c05fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:21:19.232 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:19.229+0000 7f5b210a0700 1 -- 172.21.15.132:0/809592632 shutdown_connections 2024-02-20T00:21:19.232 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:19.229+0000 7f5b210a0700 1 -- 172.21.15.132:0/809592632 wait complete. 2024-02-20T00:21:19.232 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:21:19.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:21:19 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:21:17.481167+0000 mgr.smithi084.bmqrnf (mgr.14162) 351 : cluster [DBG] pgmap v335: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:21:20.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:21:20 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:21:19.228725+0000 mon.smithi084 (mon.0) 328 : audit [DBG] from='client.? 172.21.15.132:0/809592632' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:21:20.843 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:21:20.843 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:21:21.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:21:21 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:21:19.481551+0000 mgr.smithi084.bmqrnf (mgr.14162) 352 : cluster [DBG] pgmap v336: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:21:22.797 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:22.792+0000 7fe12040e700 1 -- 172.21.15.132:0/189718160 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe1180f5410 msgr2=0x7fe1180f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:21:22.797 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:22.792+0000 7fe12040e700 1 --2- 172.21.15.132:0/189718160 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe1180f5410 0x7fe1180f5830 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 rx=0x7fe114009fb0 tx=0x7fe114009b30).stop 2024-02-20T00:21:22.797 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:22.792+0000 7fe12040e700 1 -- 172.21.15.132:0/189718160 shutdown_connections 2024-02-20T00:21:22.797 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:22.792+0000 7fe12040e700 1 --2- 172.21.15.132:0/189718160 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe1180f5410 0x7fe1180f5830 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:21:22.797 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:22.792+0000 7fe12040e700 1 -- 172.21.15.132:0/189718160 >> 172.21.15.132:0/189718160 conn(0x7fe1180f0b70 msgr2=0x7fe1180f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:21:22.797 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:22.792+0000 7fe12040e700 1 -- 172.21.15.132:0/189718160 shutdown_connections 2024-02-20T00:21:22.797 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:22.792+0000 7fe12040e700 1 -- 172.21.15.132:0/189718160 wait complete. 2024-02-20T00:21:22.798 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:22.792+0000 7fe12040e700 1 Processor -- start 2024-02-20T00:21:22.798 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:22.793+0000 7fe12040e700 1 -- start start 2024-02-20T00:21:22.798 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:22.793+0000 7fe12040e700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe1180f5410 0x7fe11810b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:21:22.798 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:22.793+0000 7fe12040e700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1181072f0 con 0x7fe1180f5410 2024-02-20T00:21:22.798 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:22.793+0000 7fe11e1aa700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe1180f5410 0x7fe11810b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:21:22.798 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:22.793+0000 7fe11e1aa700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe1180f5410 0x7fe11810b900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:60302/0 (socket says 172.21.15.132:60302) 2024-02-20T00:21:22.798 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:22.793+0000 7fe11e1aa700 1 -- 172.21.15.132:0/841423174 learned_addr learned my addr 172.21.15.132:0/841423174 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:21:22.798 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:22.793+0000 7fe11e1aa700 1 -- 172.21.15.132:0/841423174 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe114005040 con 0x7fe1180f5410 2024-02-20T00:21:22.798 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:22.793+0000 7fe11e1aa700 1 --2- 172.21.15.132:0/841423174 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe1180f5410 0x7fe11810b900 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 rx=0x7fe1140089f0 tx=0x7fe11400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:21:22.798 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:22.793+0000 7fe10effd700 1 -- 172.21.15.132:0/841423174 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe114009320 con 0x7fe1180f5410 2024-02-20T00:21:22.798 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:22.794+0000 7fe12040e700 1 -- 172.21.15.132:0/841423174 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe118060220 con 0x7fe1180f5410 2024-02-20T00:21:22.798 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:22.794+0000 7fe10effd700 1 -- 172.21.15.132:0/841423174 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe114013380 con 0x7fe1180f5410 2024-02-20T00:21:22.798 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:22.794+0000 7fe10effd700 1 -- 172.21.15.132:0/841423174 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe11401ed80 con 0x7fe1180f5410 2024-02-20T00:21:22.798 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:22.794+0000 7fe12040e700 1 -- 172.21.15.132:0/841423174 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe11810bdf0 con 0x7fe1180f5410 2024-02-20T00:21:22.799 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:22.794+0000 7fe10effd700 1 -- 172.21.15.132:0/841423174 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe11401e420 con 0x7fe1180f5410 2024-02-20T00:21:22.799 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:22.794+0000 7fe12040e700 1 -- 172.21.15.132:0/841423174 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe118045ad0 con 0x7fe1180f5410 2024-02-20T00:21:22.799 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:22.794+0000 7fe10effd700 1 --2- 172.21.15.132:0/841423174 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fe10402ff60 0x7fe104032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:21:22.799 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:22.795+0000 7fe10effd700 1 -- 172.21.15.132:0/841423174 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe114041010 con 0x7fe1180f5410 2024-02-20T00:21:22.799 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:22.795+0000 7fe11d9a9700 1 --2- 172.21.15.132:0/841423174 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fe10402ff60 0x7fe104032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:21:22.799 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:22.795+0000 7fe11d9a9700 1 --2- 172.21.15.132:0/841423174 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fe10402ff60 0x7fe104032420 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7fe1080096a0 tx=0x7fe108006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:21:22.801 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:22.799+0000 7fe10effd700 1 -- 172.21.15.132:0/841423174 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe1140095f0 con 0x7fe1180f5410 2024-02-20T00:21:23.008 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:23.006+0000 7fe12040e700 1 -- 172.21.15.132:0/841423174 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe11810c5b0 con 0x7fe1180f5410 2024-02-20T00:21:23.009 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:23.007+0000 7fe10effd700 1 -- 172.21.15.132:0/841423174 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe11402c2d0 con 0x7fe1180f5410 2024-02-20T00:21:23.009 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:21:23.009 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:21:23.011 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:23.008+0000 7fe12040e700 1 -- 172.21.15.132:0/841423174 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fe10402ff60 msgr2=0x7fe104032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:21:23.011 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:23.008+0000 7fe12040e700 1 --2- 172.21.15.132:0/841423174 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fe10402ff60 0x7fe104032420 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7fe1080096a0 tx=0x7fe108006b40).stop 2024-02-20T00:21:23.011 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:23.008+0000 7fe12040e700 1 -- 172.21.15.132:0/841423174 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe1180f5410 msgr2=0x7fe11810b900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:21:23.011 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:23.008+0000 7fe12040e700 1 --2- 172.21.15.132:0/841423174 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe1180f5410 0x7fe11810b900 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 rx=0x7fe1140089f0 tx=0x7fe11400b040).stop 2024-02-20T00:21:23.011 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:23.008+0000 7fe12040e700 1 -- 172.21.15.132:0/841423174 shutdown_connections 2024-02-20T00:21:23.011 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:23.008+0000 7fe12040e700 1 --2- 172.21.15.132:0/841423174 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fe10402ff60 0x7fe104032420 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:21:23.011 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:23.008+0000 7fe12040e700 1 --2- 172.21.15.132:0/841423174 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe1180f5410 0x7fe11810b900 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:21:23.011 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:23.008+0000 7fe12040e700 1 -- 172.21.15.132:0/841423174 >> 172.21.15.132:0/841423174 conn(0x7fe1180f0b70 msgr2=0x7fe1180f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:21:23.011 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:23.008+0000 7fe12040e700 1 -- 172.21.15.132:0/841423174 shutdown_connections 2024-02-20T00:21:23.011 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:23.008+0000 7fe12040e700 1 -- 172.21.15.132:0/841423174 wait complete. 2024-02-20T00:21:23.012 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:21:23.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:21:23 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:21:21.481919+0000 mgr.smithi084.bmqrnf (mgr.14162) 353 : cluster [DBG] pgmap v337: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:21:23.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:21:23 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:21:23.008254+0000 mon.smithi084 (mon.0) 329 : audit [DBG] from='client.? 172.21.15.132:0/841423174' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:21:24.647 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:21:24.647 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:21:25.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:21:25 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:21:23.482325+0000 mgr.smithi084.bmqrnf (mgr.14162) 354 : cluster [DBG] pgmap v338: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:21:26.519 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:26.516+0000 7f292d7b8700 1 -- 172.21.15.132:0/2776858306 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f29280f62a0 msgr2=0x7f29280f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:21:26.519 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:26.516+0000 7f292d7b8700 1 --2- 172.21.15.132:0/2776858306 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f29280f62a0 0x7f29280f66c0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 rx=0x7f291c009fb0 tx=0x7f291c009b30).stop 2024-02-20T00:21:26.519 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:26.516+0000 7f292d7b8700 1 -- 172.21.15.132:0/2776858306 shutdown_connections 2024-02-20T00:21:26.519 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:26.516+0000 7f292d7b8700 1 --2- 172.21.15.132:0/2776858306 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f29280f62a0 0x7f29280f66c0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:21:26.519 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:26.516+0000 7f292d7b8700 1 -- 172.21.15.132:0/2776858306 >> 172.21.15.132:0/2776858306 conn(0x7f29280f1aa0 msgr2=0x7f29280f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:21:26.519 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:26.516+0000 7f292d7b8700 1 -- 172.21.15.132:0/2776858306 shutdown_connections 2024-02-20T00:21:26.519 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:26.516+0000 7f292d7b8700 1 -- 172.21.15.132:0/2776858306 wait complete. 2024-02-20T00:21:26.519 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:26.516+0000 7f292d7b8700 1 Processor -- start 2024-02-20T00:21:26.520 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:26.516+0000 7f292d7b8700 1 -- start start 2024-02-20T00:21:26.520 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:26.517+0000 7f292d7b8700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f29280f62a0 0x7f292806b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:21:26.520 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:26.517+0000 7f292d7b8700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2928069e20 con 0x7f29280f62a0 2024-02-20T00:21:26.521 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:26.517+0000 7f2927fff700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f29280f62a0 0x7f292806b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:21:26.521 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:26.517+0000 7f2927fff700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f29280f62a0 0x7f292806b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:60326/0 (socket says 172.21.15.132:60326) 2024-02-20T00:21:26.521 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:26.517+0000 7f2927fff700 1 -- 172.21.15.132:0/1563944064 learned_addr learned my addr 172.21.15.132:0/1563944064 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:21:26.522 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:26.517+0000 7f2927fff700 1 -- 172.21.15.132:0/1563944064 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f291c005040 con 0x7f29280f62a0 2024-02-20T00:21:26.522 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:26.517+0000 7f2927fff700 1 --2- 172.21.15.132:0/1563944064 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f29280f62a0 0x7f292806b9a0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 rx=0x7f291c0089f0 tx=0x7f291c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:21:26.522 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:26.517+0000 7f2924ff9700 1 -- 172.21.15.132:0/1563944064 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f291c009320 con 0x7f29280f62a0 2024-02-20T00:21:26.522 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:26.517+0000 7f2924ff9700 1 -- 172.21.15.132:0/1563944064 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f291c013380 con 0x7f29280f62a0 2024-02-20T00:21:26.522 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:26.517+0000 7f292d7b8700 1 -- 172.21.15.132:0/1563944064 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2928069fc0 con 0x7f29280f62a0 2024-02-20T00:21:26.522 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:26.517+0000 7f2924ff9700 1 -- 172.21.15.132:0/1563944064 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f291c014e20 con 0x7f29280f62a0 2024-02-20T00:21:26.522 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:26.518+0000 7f292d7b8700 1 -- 172.21.15.132:0/1563944064 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f292806a450 con 0x7f29280f62a0 2024-02-20T00:21:26.522 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:26.518+0000 7f2924ff9700 1 -- 172.21.15.132:0/1563944064 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f291c0095f0 con 0x7f29280f62a0 2024-02-20T00:21:26.522 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:26.518+0000 7f292d7b8700 1 -- 172.21.15.132:0/1563944064 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2928059100 con 0x7f29280f62a0 2024-02-20T00:21:26.522 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:26.518+0000 7f2924ff9700 1 --2- 172.21.15.132:0/1563944064 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f291002ff60 0x7f2910032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:21:26.522 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:26.518+0000 7f2924ff9700 1 -- 172.21.15.132:0/1563944064 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f291c040e40 con 0x7f29280f62a0 2024-02-20T00:21:26.522 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:26.519+0000 7f29277fe700 1 --2- 172.21.15.132:0/1563944064 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f291002ff60 0x7f2910032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:21:26.522 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:26.519+0000 7f29277fe700 1 --2- 172.21.15.132:0/1563944064 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f291002ff60 0x7f2910032420 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7f29180096a0 tx=0x7f2918006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:21:26.525 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:26.523+0000 7f2924ff9700 1 -- 172.21.15.132:0/1563944064 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f291c014420 con 0x7f29280f62a0 2024-02-20T00:21:26.732 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:26.730+0000 7f292d7b8700 1 -- 172.21.15.132:0/1563944064 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f29280fbe30 con 0x7f29280f62a0 2024-02-20T00:21:26.732 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:26.730+0000 7f2924ff9700 1 -- 172.21.15.132:0/1563944064 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f291c010230 con 0x7f29280f62a0 2024-02-20T00:21:26.733 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:21:26.733 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:21:26.734 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:26.732+0000 7f292d7b8700 1 -- 172.21.15.132:0/1563944064 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f291002ff60 msgr2=0x7f2910032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:21:26.734 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:26.732+0000 7f292d7b8700 1 --2- 172.21.15.132:0/1563944064 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f291002ff60 0x7f2910032420 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7f29180096a0 tx=0x7f2918006b40).stop 2024-02-20T00:21:26.734 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:26.732+0000 7f292d7b8700 1 -- 172.21.15.132:0/1563944064 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f29280f62a0 msgr2=0x7f292806b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:21:26.734 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:26.732+0000 7f292d7b8700 1 --2- 172.21.15.132:0/1563944064 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f29280f62a0 0x7f292806b9a0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 rx=0x7f291c0089f0 tx=0x7f291c00b040).stop 2024-02-20T00:21:26.734 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:26.732+0000 7f292d7b8700 1 -- 172.21.15.132:0/1563944064 shutdown_connections 2024-02-20T00:21:26.735 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:26.732+0000 7f292d7b8700 1 --2- 172.21.15.132:0/1563944064 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f291002ff60 0x7f2910032420 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:21:26.735 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:26.732+0000 7f292d7b8700 1 --2- 172.21.15.132:0/1563944064 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f29280f62a0 0x7f292806b9a0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:21:26.735 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:26.732+0000 7f292d7b8700 1 -- 172.21.15.132:0/1563944064 >> 172.21.15.132:0/1563944064 conn(0x7f29280f1aa0 msgr2=0x7f29280f2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:21:26.735 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:26.732+0000 7f292d7b8700 1 -- 172.21.15.132:0/1563944064 shutdown_connections 2024-02-20T00:21:26.735 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:26.732+0000 7f292d7b8700 1 -- 172.21.15.132:0/1563944064 wait complete. 2024-02-20T00:21:26.735 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:21:27.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:21:27 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:21:25.482726+0000 mgr.smithi084.bmqrnf (mgr.14162) 355 : cluster [DBG] pgmap v339: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:21:27.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:21:27 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:21:26.731895+0000 mon.smithi084 (mon.0) 330 : audit [DBG] from='client.? 172.21.15.132:0/1563944064' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:21:28.403 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:21:28.403 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:21:29.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:21:29 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:21:27.483010+0000 mgr.smithi084.bmqrnf (mgr.14162) 356 : cluster [DBG] pgmap v340: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:21:30.274 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:30.270+0000 7f262522b700 1 -- 172.21.15.132:0/4124682375 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f26200f5410 msgr2=0x7f26200f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:21:30.274 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:30.270+0000 7f262522b700 1 --2- 172.21.15.132:0/4124682375 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f26200f5410 0x7f26200f5830 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 rx=0x7f2608009fb0 tx=0x7f2608009b30).stop 2024-02-20T00:21:30.274 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:30.270+0000 7f262522b700 1 -- 172.21.15.132:0/4124682375 shutdown_connections 2024-02-20T00:21:30.274 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:30.270+0000 7f262522b700 1 --2- 172.21.15.132:0/4124682375 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f26200f5410 0x7f26200f5830 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:21:30.275 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:30.270+0000 7f262522b700 1 -- 172.21.15.132:0/4124682375 >> 172.21.15.132:0/4124682375 conn(0x7f26200f0b70 msgr2=0x7f26200f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:21:30.275 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:30.270+0000 7f262522b700 1 -- 172.21.15.132:0/4124682375 shutdown_connections 2024-02-20T00:21:30.275 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:30.270+0000 7f262522b700 1 -- 172.21.15.132:0/4124682375 wait complete. 2024-02-20T00:21:30.275 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:30.271+0000 7f262522b700 1 Processor -- start 2024-02-20T00:21:30.275 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:30.271+0000 7f262522b700 1 -- start start 2024-02-20T00:21:30.275 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:30.271+0000 7f262522b700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f26200f5410 0x7f26201067d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:21:30.275 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:30.271+0000 7f262522b700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2620106cc0 con 0x7f26200f5410 2024-02-20T00:21:30.275 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:30.271+0000 7f261ed9d700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f26200f5410 0x7f26201067d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:21:30.275 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:30.271+0000 7f261ed9d700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f26200f5410 0x7f26201067d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:49868/0 (socket says 172.21.15.132:49868) 2024-02-20T00:21:30.275 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:30.271+0000 7f261ed9d700 1 -- 172.21.15.132:0/3537407861 learned_addr learned my addr 172.21.15.132:0/3537407861 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:21:30.275 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:30.272+0000 7f261ed9d700 1 -- 172.21.15.132:0/3537407861 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2608005040 con 0x7f26200f5410 2024-02-20T00:21:30.275 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:30.272+0000 7f261ed9d700 1 --2- 172.21.15.132:0/3537407861 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f26200f5410 0x7f26201067d0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 rx=0x7f26080089f0 tx=0x7f260800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:21:30.275 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:30.272+0000 7f26177fe700 1 -- 172.21.15.132:0/3537407861 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2608009320 con 0x7f26200f5410 2024-02-20T00:21:30.276 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:30.272+0000 7f262522b700 1 -- 172.21.15.132:0/3537407861 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2620106ec0 con 0x7f26200f5410 2024-02-20T00:21:30.277 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:30.272+0000 7f26177fe700 1 -- 172.21.15.132:0/3537407861 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2608013380 con 0x7f26200f5410 2024-02-20T00:21:30.277 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:30.272+0000 7f26177fe700 1 -- 172.21.15.132:0/3537407861 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f260801ed80 con 0x7f26200f5410 2024-02-20T00:21:30.277 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:30.272+0000 7f262522b700 1 -- 172.21.15.132:0/3537407861 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2620107350 con 0x7f26200f5410 2024-02-20T00:21:30.277 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:30.273+0000 7f26177fe700 1 -- 172.21.15.132:0/3537407861 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f260801e420 con 0x7f26200f5410 2024-02-20T00:21:30.277 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:30.273+0000 7f262522b700 1 -- 172.21.15.132:0/3537407861 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2620045ad0 con 0x7f26200f5410 2024-02-20T00:21:30.277 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:30.273+0000 7f26177fe700 1 --2- 172.21.15.132:0/3537407861 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f260c02ff60 0x7f260c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:21:30.277 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:30.273+0000 7f26177fe700 1 -- 172.21.15.132:0/3537407861 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2608041120 con 0x7f26200f5410 2024-02-20T00:21:30.277 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:30.273+0000 7f261e59c700 1 --2- 172.21.15.132:0/3537407861 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f260c02ff60 0x7f260c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:21:30.277 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:30.273+0000 7f261e59c700 1 --2- 172.21.15.132:0/3537407861 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f260c02ff60 0x7f260c032420 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f26100096a0 tx=0x7f2610006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:21:30.280 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:30.277+0000 7f26177fe700 1 -- 172.21.15.132:0/3537407861 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f26080095f0 con 0x7f26200f5410 2024-02-20T00:21:30.485 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:30.482+0000 7f262522b700 1 -- 172.21.15.132:0/3537407861 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2620108200 con 0x7f26200f5410 2024-02-20T00:21:30.485 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:30.483+0000 7f26177fe700 1 -- 172.21.15.132:0/3537407861 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f260801cb10 con 0x7f26200f5410 2024-02-20T00:21:30.485 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:21:30.486 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:21:30.487 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:30.484+0000 7f262522b700 1 -- 172.21.15.132:0/3537407861 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f260c02ff60 msgr2=0x7f260c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:21:30.487 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:30.484+0000 7f262522b700 1 --2- 172.21.15.132:0/3537407861 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f260c02ff60 0x7f260c032420 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f26100096a0 tx=0x7f2610006b40).stop 2024-02-20T00:21:30.487 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:30.484+0000 7f262522b700 1 -- 172.21.15.132:0/3537407861 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f26200f5410 msgr2=0x7f26201067d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:21:30.487 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:30.484+0000 7f262522b700 1 --2- 172.21.15.132:0/3537407861 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f26200f5410 0x7f26201067d0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 rx=0x7f26080089f0 tx=0x7f260800b040).stop 2024-02-20T00:21:30.487 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:30.485+0000 7f262522b700 1 -- 172.21.15.132:0/3537407861 shutdown_connections 2024-02-20T00:21:30.488 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:30.485+0000 7f262522b700 1 --2- 172.21.15.132:0/3537407861 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f260c02ff60 0x7f260c032420 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:21:30.488 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:30.485+0000 7f262522b700 1 --2- 172.21.15.132:0/3537407861 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f26200f5410 0x7f26201067d0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:21:30.488 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:30.485+0000 7f262522b700 1 -- 172.21.15.132:0/3537407861 >> 172.21.15.132:0/3537407861 conn(0x7f26200f0b70 msgr2=0x7f262005fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:21:30.488 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:30.485+0000 7f262522b700 1 -- 172.21.15.132:0/3537407861 shutdown_connections 2024-02-20T00:21:30.488 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:30.485+0000 7f262522b700 1 -- 172.21.15.132:0/3537407861 wait complete. 2024-02-20T00:21:30.488 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:21:31.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:21:31 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:21:29.483364+0000 mgr.smithi084.bmqrnf (mgr.14162) 357 : cluster [DBG] pgmap v341: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:21:31.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:21:31 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:21:30.484580+0000 mon.smithi084 (mon.0) 331 : audit [DBG] from='client.? 172.21.15.132:0/3537407861' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:21:32.217 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:21:32.217 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:21:33.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:21:33 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:21:31.483722+0000 mgr.smithi084.bmqrnf (mgr.14162) 358 : cluster [DBG] pgmap v342: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:21:34.424 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:34.421+0000 7fbf95fab700 1 -- 172.21.15.132:0/3399146001 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbf90060220 msgr2=0x7fbf900fb550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:21:34.424 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:34.421+0000 7fbf95fab700 1 --2- 172.21.15.132:0/3399146001 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbf90060220 0x7fbf900fb550 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 rx=0x7fbf80009fb0 tx=0x7fbf80009b30).stop 2024-02-20T00:21:34.424 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:34.421+0000 7fbf95fab700 1 -- 172.21.15.132:0/3399146001 shutdown_connections 2024-02-20T00:21:34.424 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:34.421+0000 7fbf95fab700 1 --2- 172.21.15.132:0/3399146001 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbf90060220 0x7fbf900fb550 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:21:34.424 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:34.421+0000 7fbf95fab700 1 -- 172.21.15.132:0/3399146001 >> 172.21.15.132:0/3399146001 conn(0x7fbf900f0b70 msgr2=0x7fbf900f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:21:34.424 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:34.421+0000 7fbf95fab700 1 -- 172.21.15.132:0/3399146001 shutdown_connections 2024-02-20T00:21:34.424 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:34.421+0000 7fbf95fab700 1 -- 172.21.15.132:0/3399146001 wait complete. 2024-02-20T00:21:34.425 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:34.422+0000 7fbf95fab700 1 Processor -- start 2024-02-20T00:21:34.425 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:34.422+0000 7fbf95fab700 1 -- start start 2024-02-20T00:21:34.427 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:34.422+0000 7fbf95fab700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbf90060220 0x7fbf901089a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:21:34.427 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:34.422+0000 7fbf95fab700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf90108e90 con 0x7fbf90060220 2024-02-20T00:21:34.427 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:34.422+0000 7fbf8f7fe700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbf90060220 0x7fbf901089a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:21:34.427 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:34.422+0000 7fbf8f7fe700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbf90060220 0x7fbf901089a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:49886/0 (socket says 172.21.15.132:49886) 2024-02-20T00:21:34.427 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:34.422+0000 7fbf8f7fe700 1 -- 172.21.15.132:0/2638658603 learned_addr learned my addr 172.21.15.132:0/2638658603 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:21:34.427 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:34.423+0000 7fbf8f7fe700 1 -- 172.21.15.132:0/2638658603 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf80005040 con 0x7fbf90060220 2024-02-20T00:21:34.427 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:34.423+0000 7fbf8f7fe700 1 --2- 172.21.15.132:0/2638658603 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbf90060220 0x7fbf901089a0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 rx=0x7fbf800089f0 tx=0x7fbf8000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:21:34.427 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:34.423+0000 7fbf77fff700 1 -- 172.21.15.132:0/2638658603 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbf80009320 con 0x7fbf90060220 2024-02-20T00:21:34.427 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:34.423+0000 7fbf95fab700 1 -- 172.21.15.132:0/2638658603 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf90109090 con 0x7fbf90060220 2024-02-20T00:21:34.427 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:34.423+0000 7fbf77fff700 1 -- 172.21.15.132:0/2638658603 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbf80013380 con 0x7fbf90060220 2024-02-20T00:21:34.427 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:34.423+0000 7fbf77fff700 1 -- 172.21.15.132:0/2638658603 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbf80014e20 con 0x7fbf90060220 2024-02-20T00:21:34.428 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:34.423+0000 7fbf95fab700 1 -- 172.21.15.132:0/2638658603 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf90109520 con 0x7fbf90060220 2024-02-20T00:21:34.428 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:34.424+0000 7fbf77fff700 1 -- 172.21.15.132:0/2638658603 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbf80014420 con 0x7fbf90060220 2024-02-20T00:21:34.428 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:34.424+0000 7fbf95fab700 1 -- 172.21.15.132:0/2638658603 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf90045ad0 con 0x7fbf90060220 2024-02-20T00:21:34.428 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:34.424+0000 7fbf77fff700 1 --2- 172.21.15.132:0/2638658603 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fbf7802ff10 0x7fbf780323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:21:34.428 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:34.424+0000 7fbf77fff700 1 -- 172.21.15.132:0/2638658603 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbf80041010 con 0x7fbf90060220 2024-02-20T00:21:34.428 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:34.424+0000 7fbf8effd700 1 --2- 172.21.15.132:0/2638658603 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fbf7802ff10 0x7fbf780323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:21:34.428 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:34.425+0000 7fbf8effd700 1 --2- 172.21.15.132:0/2638658603 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fbf7802ff10 0x7fbf780323d0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7fbf840096a0 tx=0x7fbf84006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:21:34.431 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:34.429+0000 7fbf77fff700 1 -- 172.21.15.132:0/2638658603 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbf8006d2b0 con 0x7fbf90060220 2024-02-20T00:21:34.636 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:34.633+0000 7fbf95fab700 1 -- 172.21.15.132:0/2638658603 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbf9010a3d0 con 0x7fbf90060220 2024-02-20T00:21:34.636 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:34.634+0000 7fbf77fff700 1 -- 172.21.15.132:0/2638658603 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fbf800095f0 con 0x7fbf90060220 2024-02-20T00:21:34.637 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:21:34.637 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:21:34.638 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:34.635+0000 7fbf95fab700 1 -- 172.21.15.132:0/2638658603 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fbf7802ff10 msgr2=0x7fbf780323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:21:34.638 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:34.635+0000 7fbf95fab700 1 --2- 172.21.15.132:0/2638658603 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fbf7802ff10 0x7fbf780323d0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7fbf840096a0 tx=0x7fbf84006b40).stop 2024-02-20T00:21:34.638 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:34.635+0000 7fbf95fab700 1 -- 172.21.15.132:0/2638658603 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbf90060220 msgr2=0x7fbf901089a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:21:34.638 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:34.636+0000 7fbf95fab700 1 --2- 172.21.15.132:0/2638658603 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbf90060220 0x7fbf901089a0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 rx=0x7fbf800089f0 tx=0x7fbf8000b040).stop 2024-02-20T00:21:34.638 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:34.636+0000 7fbf95fab700 1 -- 172.21.15.132:0/2638658603 shutdown_connections 2024-02-20T00:21:34.638 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:34.636+0000 7fbf95fab700 1 --2- 172.21.15.132:0/2638658603 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fbf7802ff10 0x7fbf780323d0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:21:34.638 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:34.636+0000 7fbf95fab700 1 --2- 172.21.15.132:0/2638658603 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbf90060220 0x7fbf901089a0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:21:34.639 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:34.636+0000 7fbf95fab700 1 -- 172.21.15.132:0/2638658603 >> 172.21.15.132:0/2638658603 conn(0x7fbf900f0b70 msgr2=0x7fbf900f1600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:21:34.639 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:34.636+0000 7fbf95fab700 1 -- 172.21.15.132:0/2638658603 shutdown_connections 2024-02-20T00:21:34.639 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:34.636+0000 7fbf95fab700 1 -- 172.21.15.132:0/2638658603 wait complete. 2024-02-20T00:21:34.639 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:21:35.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:21:35 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:21:33.484059+0000 mgr.smithi084.bmqrnf (mgr.14162) 359 : cluster [DBG] pgmap v343: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:21:35.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:21:35 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:21:34.635676+0000 mon.smithi084 (mon.0) 332 : audit [DBG] from='client.? 172.21.15.132:0/2638658603' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:21:36.422 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:21:36.423 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:21:37.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:21:37 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:21:35.484411+0000 mgr.smithi084.bmqrnf (mgr.14162) 360 : cluster [DBG] pgmap v344: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:21:38.194 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:38.190+0000 7fb3faece700 1 -- 172.21.15.132:0/3931245510 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb3f40f6510 msgr2=0x7fb3f40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:21:38.195 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:38.190+0000 7fb3faece700 1 --2- 172.21.15.132:0/3931245510 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb3f40f6510 0x7fb3f40f6930 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 rx=0x7fb3e8009fb0 tx=0x7fb3e8009b30).stop 2024-02-20T00:21:38.195 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:38.190+0000 7fb3faece700 1 -- 172.21.15.132:0/3931245510 shutdown_connections 2024-02-20T00:21:38.195 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:38.190+0000 7fb3faece700 1 --2- 172.21.15.132:0/3931245510 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb3f40f6510 0x7fb3f40f6930 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:21:38.195 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:38.190+0000 7fb3faece700 1 -- 172.21.15.132:0/3931245510 >> 172.21.15.132:0/3931245510 conn(0x7fb3f40f1c70 msgr2=0x7fb3f40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:21:38.195 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:38.190+0000 7fb3faece700 1 -- 172.21.15.132:0/3931245510 shutdown_connections 2024-02-20T00:21:38.195 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:38.190+0000 7fb3faece700 1 -- 172.21.15.132:0/3931245510 wait complete. 2024-02-20T00:21:38.195 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:38.191+0000 7fb3faece700 1 Processor -- start 2024-02-20T00:21:38.195 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:38.191+0000 7fb3faece700 1 -- start start 2024-02-20T00:21:38.196 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:38.191+0000 7fb3faece700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb3f40f6510 0x7fb3f41067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:21:38.196 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:38.191+0000 7fb3faece700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3f4106cd0 con 0x7fb3f40f6510 2024-02-20T00:21:38.196 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:38.191+0000 7fb3f8c6a700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb3f40f6510 0x7fb3f41067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:21:38.196 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:38.191+0000 7fb3f8c6a700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb3f40f6510 0x7fb3f41067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:42638/0 (socket says 172.21.15.132:42638) 2024-02-20T00:21:38.196 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:38.191+0000 7fb3f8c6a700 1 -- 172.21.15.132:0/261225427 learned_addr learned my addr 172.21.15.132:0/261225427 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:21:38.196 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:38.192+0000 7fb3f8c6a700 1 -- 172.21.15.132:0/261225427 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3e8005040 con 0x7fb3f40f6510 2024-02-20T00:21:38.196 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:38.192+0000 7fb3f8c6a700 1 --2- 172.21.15.132:0/261225427 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb3f40f6510 0x7fb3f41067e0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0x7fb3e8005010 tx=0x7fb3e800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:21:38.196 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:38.192+0000 7fb3f17fa700 1 -- 172.21.15.132:0/261225427 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb3e8009320 con 0x7fb3f40f6510 2024-02-20T00:21:38.196 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:38.192+0000 7fb3faece700 1 -- 172.21.15.132:0/261225427 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3f4106ed0 con 0x7fb3f40f6510 2024-02-20T00:21:38.196 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:38.192+0000 7fb3f17fa700 1 -- 172.21.15.132:0/261225427 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb3e8013380 con 0x7fb3f40f6510 2024-02-20T00:21:38.196 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:38.192+0000 7fb3faece700 1 -- 172.21.15.132:0/261225427 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3f4107360 con 0x7fb3f40f6510 2024-02-20T00:21:38.196 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:38.192+0000 7fb3f17fa700 1 -- 172.21.15.132:0/261225427 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb3e8014e50 con 0x7fb3f40f6510 2024-02-20T00:21:38.197 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:38.193+0000 7fb3f17fa700 1 -- 172.21.15.132:0/261225427 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb3e80095f0 con 0x7fb3f40f6510 2024-02-20T00:21:38.198 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:38.193+0000 7fb3faece700 1 -- 172.21.15.132:0/261225427 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3f4045ad0 con 0x7fb3f40f6510 2024-02-20T00:21:38.198 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:38.193+0000 7fb3f17fa700 1 --2- 172.21.15.132:0/261225427 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fb3e402ff10 0x7fb3e40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:21:38.198 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:38.193+0000 7fb3f17fa700 1 -- 172.21.15.132:0/261225427 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb3e8040e80 con 0x7fb3f40f6510 2024-02-20T00:21:38.198 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:38.193+0000 7fb3f3fff700 1 --2- 172.21.15.132:0/261225427 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fb3e402ff10 0x7fb3e40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:21:38.198 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:38.194+0000 7fb3f3fff700 1 --2- 172.21.15.132:0/261225427 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fb3e402ff10 0x7fb3e40323d0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7fb3e00096a0 tx=0x7fb3e0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:21:38.200 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:38.198+0000 7fb3f17fa700 1 -- 172.21.15.132:0/261225427 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb3e8014420 con 0x7fb3f40f6510 2024-02-20T00:21:38.404 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:38.402+0000 7fb3faece700 1 -- 172.21.15.132:0/261225427 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb3f41081b0 con 0x7fb3f40f6510 2024-02-20T00:21:38.405 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:38.403+0000 7fb3f17fa700 1 -- 172.21.15.132:0/261225427 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb3e8010230 con 0x7fb3f40f6510 2024-02-20T00:21:38.405 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:21:38.405 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:21:38.407 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:38.404+0000 7fb3faece700 1 -- 172.21.15.132:0/261225427 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fb3e402ff10 msgr2=0x7fb3e40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:21:38.407 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:38.404+0000 7fb3faece700 1 --2- 172.21.15.132:0/261225427 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fb3e402ff10 0x7fb3e40323d0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7fb3e00096a0 tx=0x7fb3e0006b40).stop 2024-02-20T00:21:38.407 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:38.404+0000 7fb3faece700 1 -- 172.21.15.132:0/261225427 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb3f40f6510 msgr2=0x7fb3f41067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:21:38.407 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:38.404+0000 7fb3faece700 1 --2- 172.21.15.132:0/261225427 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb3f40f6510 0x7fb3f41067e0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0x7fb3e8005010 tx=0x7fb3e800b040).stop 2024-02-20T00:21:38.407 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:38.404+0000 7fb3faece700 1 -- 172.21.15.132:0/261225427 shutdown_connections 2024-02-20T00:21:38.408 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:38.404+0000 7fb3faece700 1 --2- 172.21.15.132:0/261225427 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fb3e402ff10 0x7fb3e40323d0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:21:38.408 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:38.404+0000 7fb3faece700 1 --2- 172.21.15.132:0/261225427 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb3f40f6510 0x7fb3f41067e0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:21:38.408 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:38.404+0000 7fb3faece700 1 -- 172.21.15.132:0/261225427 >> 172.21.15.132:0/261225427 conn(0x7fb3f40f1c70 msgr2=0x7fb3f40f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:21:38.408 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:38.404+0000 7fb3faece700 1 -- 172.21.15.132:0/261225427 shutdown_connections 2024-02-20T00:21:38.408 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:38.404+0000 7fb3faece700 1 -- 172.21.15.132:0/261225427 wait complete. 2024-02-20T00:21:38.408 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:21:39.360 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:21:39 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:21:37.484792+0000 mgr.smithi084.bmqrnf (mgr.14162) 361 : cluster [DBG] pgmap v345: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:21:39.360 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:21:39 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:21:38.404096+0000 mon.smithi084 (mon.0) 333 : audit [DBG] from='client.? 172.21.15.132:0/261225427' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:21:40.110 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:21:40.110 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:21:41.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:21:41 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:21:39.485162+0000 mgr.smithi084.bmqrnf (mgr.14162) 362 : cluster [DBG] pgmap v346: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:21:42.007 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:42.005+0000 7fd00e85f700 1 -- 172.21.15.132:0/214336665 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd0080f6480 msgr2=0x7fd0080f68a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:21:42.007 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:42.005+0000 7fd00e85f700 1 --2- 172.21.15.132:0/214336665 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd0080f6480 0x7fd0080f68a0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 rx=0x7fcffc009fb0 tx=0x7fcffc009b30).stop 2024-02-20T00:21:42.007 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:42.005+0000 7fd00e85f700 1 -- 172.21.15.132:0/214336665 shutdown_connections 2024-02-20T00:21:42.007 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:42.005+0000 7fd00e85f700 1 --2- 172.21.15.132:0/214336665 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd0080f6480 0x7fd0080f68a0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:21:42.007 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:42.005+0000 7fd00e85f700 1 -- 172.21.15.132:0/214336665 >> 172.21.15.132:0/214336665 conn(0x7fd0080f1c00 msgr2=0x7fd0080f4020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:21:42.007 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:42.005+0000 7fd00e85f700 1 -- 172.21.15.132:0/214336665 shutdown_connections 2024-02-20T00:21:42.007 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:42.005+0000 7fd00e85f700 1 -- 172.21.15.132:0/214336665 wait complete. 2024-02-20T00:21:42.007 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:42.005+0000 7fd00e85f700 1 Processor -- start 2024-02-20T00:21:42.008 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:42.006+0000 7fd00e85f700 1 -- start start 2024-02-20T00:21:42.008 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:42.006+0000 7fd00e85f700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd0080f6480 0x7fd0081067f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:21:42.008 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:42.006+0000 7fd00e85f700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd008106ce0 con 0x7fd0080f6480 2024-02-20T00:21:42.009 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:42.006+0000 7fd007fff700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd0080f6480 0x7fd0081067f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:21:42.009 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:42.006+0000 7fd007fff700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd0080f6480 0x7fd0081067f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:42660/0 (socket says 172.21.15.132:42660) 2024-02-20T00:21:42.009 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:42.006+0000 7fd007fff700 1 -- 172.21.15.132:0/2131731245 learned_addr learned my addr 172.21.15.132:0/2131731245 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:21:42.009 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:42.006+0000 7fd007fff700 1 -- 172.21.15.132:0/2131731245 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcffc005040 con 0x7fd0080f6480 2024-02-20T00:21:42.009 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:42.007+0000 7fd007fff700 1 --2- 172.21.15.132:0/2131731245 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd0080f6480 0x7fd0081067f0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 rx=0x7fcffc005010 tx=0x7fcffc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:21:42.009 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:42.007+0000 7fd004ff9700 1 -- 172.21.15.132:0/2131731245 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcffc009320 con 0x7fd0080f6480 2024-02-20T00:21:42.009 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:42.007+0000 7fd00e85f700 1 -- 172.21.15.132:0/2131731245 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd008106ee0 con 0x7fd0080f6480 2024-02-20T00:21:42.009 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:42.007+0000 7fd004ff9700 1 -- 172.21.15.132:0/2131731245 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcffc013380 con 0x7fd0080f6480 2024-02-20T00:21:42.009 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:42.007+0000 7fd004ff9700 1 -- 172.21.15.132:0/2131731245 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcffc014e50 con 0x7fd0080f6480 2024-02-20T00:21:42.009 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:42.007+0000 7fd00e85f700 1 -- 172.21.15.132:0/2131731245 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd008107370 con 0x7fd0080f6480 2024-02-20T00:21:42.010 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:42.008+0000 7fd004ff9700 1 -- 172.21.15.132:0/2131731245 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fcffc0095f0 con 0x7fd0080f6480 2024-02-20T00:21:42.010 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:42.008+0000 7fd00e85f700 1 -- 172.21.15.132:0/2131731245 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd008045ad0 con 0x7fd0080f6480 2024-02-20T00:21:42.010 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:42.008+0000 7fd004ff9700 1 --2- 172.21.15.132:0/2131731245 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fcff002ff10 0x7fcff00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:21:42.010 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:42.008+0000 7fd004ff9700 1 -- 172.21.15.132:0/2131731245 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fcffc040ee0 con 0x7fd0080f6480 2024-02-20T00:21:42.010 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:42.008+0000 7fd0077fe700 1 --2- 172.21.15.132:0/2131731245 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fcff002ff10 0x7fcff00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:21:42.010 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:42.008+0000 7fd0077fe700 1 --2- 172.21.15.132:0/2131731245 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fcff002ff10 0x7fcff00323d0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7fcff80096a0 tx=0x7fcff8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:21:42.014 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:42.012+0000 7fd004ff9700 1 -- 172.21.15.132:0/2131731245 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcffc014420 con 0x7fd0080f6480 2024-02-20T00:21:42.217 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:42.216+0000 7fd00e85f700 1 -- 172.21.15.132:0/2131731245 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd0080fc090 con 0x7fd0080f6480 2024-02-20T00:21:42.218 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:42.216+0000 7fd004ff9700 1 -- 172.21.15.132:0/2131731245 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fcffc010230 con 0x7fd0080f6480 2024-02-20T00:21:42.218 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:21:42.218 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:21:42.220 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:42.218+0000 7fd00e85f700 1 -- 172.21.15.132:0/2131731245 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fcff002ff10 msgr2=0x7fcff00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:21:42.220 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:42.218+0000 7fd00e85f700 1 --2- 172.21.15.132:0/2131731245 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fcff002ff10 0x7fcff00323d0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7fcff80096a0 tx=0x7fcff8006b40).stop 2024-02-20T00:21:42.220 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:42.218+0000 7fd00e85f700 1 -- 172.21.15.132:0/2131731245 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd0080f6480 msgr2=0x7fd0081067f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:21:42.220 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:42.218+0000 7fd00e85f700 1 --2- 172.21.15.132:0/2131731245 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd0080f6480 0x7fd0081067f0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 rx=0x7fcffc005010 tx=0x7fcffc00b040).stop 2024-02-20T00:21:42.220 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:42.218+0000 7fd00e85f700 1 -- 172.21.15.132:0/2131731245 shutdown_connections 2024-02-20T00:21:42.220 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:42.218+0000 7fd00e85f700 1 --2- 172.21.15.132:0/2131731245 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fcff002ff10 0x7fcff00323d0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:21:42.220 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:42.218+0000 7fd00e85f700 1 --2- 172.21.15.132:0/2131731245 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd0080f6480 0x7fd0081067f0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:21:42.220 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:42.218+0000 7fd00e85f700 1 -- 172.21.15.132:0/2131731245 >> 172.21.15.132:0/2131731245 conn(0x7fd0080f1c00 msgr2=0x7fd0080f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:21:42.220 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:42.218+0000 7fd00e85f700 1 -- 172.21.15.132:0/2131731245 shutdown_connections 2024-02-20T00:21:42.220 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:42.218+0000 7fd00e85f700 1 -- 172.21.15.132:0/2131731245 wait complete. 2024-02-20T00:21:42.221 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:21:43.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:21:43 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:21:41.485472+0000 mgr.smithi084.bmqrnf (mgr.14162) 363 : cluster [DBG] pgmap v347: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:21:43.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:21:43 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:21:42.216893+0000 mon.smithi084 (mon.0) 334 : audit [DBG] from='client.? 172.21.15.132:0/2131731245' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:21:44.020 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:21:44.021 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:21:45.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:21:45 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:21:43.485740+0000 mgr.smithi084.bmqrnf (mgr.14162) 364 : cluster [DBG] pgmap v348: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:21:45.847 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:45.843+0000 7f21301c3700 1 -- 172.21.15.132:0/2481391484 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f21280f4740 msgr2=0x7f21280f4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:21:45.847 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:45.843+0000 7f21301c3700 1 --2- 172.21.15.132:0/2481391484 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f21280f4740 0x7f21280f4b60 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 rx=0x7f211c009fb0 tx=0x7f211c009b30).stop 2024-02-20T00:21:45.847 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:45.844+0000 7f21301c3700 1 -- 172.21.15.132:0/2481391484 shutdown_connections 2024-02-20T00:21:45.847 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:45.844+0000 7f21301c3700 1 --2- 172.21.15.132:0/2481391484 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f21280f4740 0x7f21280f4b60 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:21:45.847 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:45.844+0000 7f21301c3700 1 -- 172.21.15.132:0/2481391484 >> 172.21.15.132:0/2481391484 conn(0x7f21280f0260 msgr2=0x7f21280f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:21:45.847 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:45.844+0000 7f21301c3700 1 -- 172.21.15.132:0/2481391484 shutdown_connections 2024-02-20T00:21:45.848 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:45.844+0000 7f21301c3700 1 -- 172.21.15.132:0/2481391484 wait complete. 2024-02-20T00:21:45.848 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:45.844+0000 7f21301c3700 1 Processor -- start 2024-02-20T00:21:45.848 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:45.844+0000 7f21301c3700 1 -- start start 2024-02-20T00:21:45.848 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:45.845+0000 7f21301c3700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f21280f4740 0x7f21281067b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:21:45.848 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:45.845+0000 7f21301c3700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2128106ca0 con 0x7f21280f4740 2024-02-20T00:21:45.848 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:45.845+0000 7f212df5f700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f21280f4740 0x7f21281067b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:21:45.848 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:45.845+0000 7f212df5f700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f21280f4740 0x7f21281067b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:42684/0 (socket says 172.21.15.132:42684) 2024-02-20T00:21:45.848 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:45.845+0000 7f212df5f700 1 -- 172.21.15.132:0/1010593752 learned_addr learned my addr 172.21.15.132:0/1010593752 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:21:45.848 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:45.845+0000 7f212df5f700 1 -- 172.21.15.132:0/1010593752 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f211c005040 con 0x7f21280f4740 2024-02-20T00:21:45.848 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:45.845+0000 7f212df5f700 1 --2- 172.21.15.132:0/1010593752 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f21280f4740 0x7f21281067b0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 rx=0x7f211c005010 tx=0x7f211c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:21:45.848 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:45.845+0000 7f211a7fc700 1 -- 172.21.15.132:0/1010593752 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f211c009320 con 0x7f21280f4740 2024-02-20T00:21:45.848 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:45.845+0000 7f211a7fc700 1 -- 172.21.15.132:0/1010593752 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f211c013380 con 0x7f21280f4740 2024-02-20T00:21:45.848 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:45.845+0000 7f21301c3700 1 -- 172.21.15.132:0/1010593752 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2128106ea0 con 0x7f21280f4740 2024-02-20T00:21:45.849 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:45.845+0000 7f211a7fc700 1 -- 172.21.15.132:0/1010593752 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f211c014e50 con 0x7f21280f4740 2024-02-20T00:21:45.849 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:45.845+0000 7f21301c3700 1 -- 172.21.15.132:0/1010593752 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2128107330 con 0x7f21280f4740 2024-02-20T00:21:45.849 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:45.846+0000 7f211a7fc700 1 -- 172.21.15.132:0/1010593752 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f211c0095f0 con 0x7f21280f4740 2024-02-20T00:21:45.850 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:45.846+0000 7f21301c3700 1 -- 172.21.15.132:0/1010593752 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2128045ad0 con 0x7f21280f4740 2024-02-20T00:21:45.850 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:45.846+0000 7f211a7fc700 1 --2- 172.21.15.132:0/1010593752 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f211402ff10 0x7f21140323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:21:45.850 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:45.846+0000 7f211a7fc700 1 -- 172.21.15.132:0/1010593752 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f211c040e90 con 0x7f21280f4740 2024-02-20T00:21:45.850 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:45.847+0000 7f212d75e700 1 --2- 172.21.15.132:0/1010593752 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f211402ff10 0x7f21140323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:21:45.850 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:45.847+0000 7f212d75e700 1 --2- 172.21.15.132:0/1010593752 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f211402ff10 0x7f21140323d0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f21240096a0 tx=0x7f2124006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:21:45.852 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:45.851+0000 7f211a7fc700 1 -- 172.21.15.132:0/1010593752 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f211c014420 con 0x7f21280f4740 2024-02-20T00:21:46.056 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:46.055+0000 7f21301c3700 1 -- 172.21.15.132:0/1010593752 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f21281081e0 con 0x7f21280f4740 2024-02-20T00:21:46.057 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:46.056+0000 7f211a7fc700 1 -- 172.21.15.132:0/1010593752 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f211c010230 con 0x7f21280f4740 2024-02-20T00:21:46.057 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:21:46.057 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:21:46.059 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:46.057+0000 7f21301c3700 1 -- 172.21.15.132:0/1010593752 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f211402ff10 msgr2=0x7f21140323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:21:46.059 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:46.057+0000 7f21301c3700 1 --2- 172.21.15.132:0/1010593752 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f211402ff10 0x7f21140323d0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f21240096a0 tx=0x7f2124006b40).stop 2024-02-20T00:21:46.059 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:46.057+0000 7f21301c3700 1 -- 172.21.15.132:0/1010593752 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f21280f4740 msgr2=0x7f21281067b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:21:46.059 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:46.057+0000 7f21301c3700 1 --2- 172.21.15.132:0/1010593752 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f21280f4740 0x7f21281067b0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 rx=0x7f211c005010 tx=0x7f211c00b040).stop 2024-02-20T00:21:46.059 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:46.057+0000 7f21301c3700 1 -- 172.21.15.132:0/1010593752 shutdown_connections 2024-02-20T00:21:46.059 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:46.057+0000 7f21301c3700 1 --2- 172.21.15.132:0/1010593752 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f211402ff10 0x7f21140323d0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:21:46.059 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:46.057+0000 7f21301c3700 1 --2- 172.21.15.132:0/1010593752 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f21280f4740 0x7f21281067b0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:21:46.059 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:46.057+0000 7f21301c3700 1 -- 172.21.15.132:0/1010593752 >> 172.21.15.132:0/1010593752 conn(0x7f21280f0260 msgr2=0x7f21280f0cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:21:46.059 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:46.057+0000 7f21301c3700 1 -- 172.21.15.132:0/1010593752 shutdown_connections 2024-02-20T00:21:46.059 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:46.057+0000 7f21301c3700 1 -- 172.21.15.132:0/1010593752 wait complete. 2024-02-20T00:21:46.060 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:21:47.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:21:47 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:21:45.486027+0000 mgr.smithi084.bmqrnf (mgr.14162) 365 : cluster [DBG] pgmap v349: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:21:47.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:21:47 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:21:46.056361+0000 mon.smithi084 (mon.0) 335 : audit [DBG] from='client.? 172.21.15.132:0/1010593752' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:21:47.755 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:21:47.756 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:21:49.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:21:49 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:21:47.486428+0000 mgr.smithi084.bmqrnf (mgr.14162) 366 : cluster [DBG] pgmap v350: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:21:49.684 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:49.681+0000 7fec55abe700 1 -- 172.21.15.132:0/1069216775 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fec500f6510 msgr2=0x7fec500f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:21:49.684 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:49.681+0000 7fec55abe700 1 --2- 172.21.15.132:0/1069216775 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fec500f6510 0x7fec500f6930 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 rx=0x7fec44009fb0 tx=0x7fec44009b30).stop 2024-02-20T00:21:49.684 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:49.681+0000 7fec55abe700 1 -- 172.21.15.132:0/1069216775 shutdown_connections 2024-02-20T00:21:49.684 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:49.681+0000 7fec55abe700 1 --2- 172.21.15.132:0/1069216775 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fec500f6510 0x7fec500f6930 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:21:49.684 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:49.681+0000 7fec55abe700 1 -- 172.21.15.132:0/1069216775 >> 172.21.15.132:0/1069216775 conn(0x7fec500f1c70 msgr2=0x7fec500f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:21:49.684 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:49.681+0000 7fec55abe700 1 -- 172.21.15.132:0/1069216775 shutdown_connections 2024-02-20T00:21:49.684 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:49.681+0000 7fec55abe700 1 -- 172.21.15.132:0/1069216775 wait complete. 2024-02-20T00:21:49.684 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:49.681+0000 7fec55abe700 1 Processor -- start 2024-02-20T00:21:49.684 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:49.682+0000 7fec55abe700 1 -- start start 2024-02-20T00:21:49.684 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:49.682+0000 7fec55abe700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fec500f6510 0x7fec5006b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:21:49.684 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:49.682+0000 7fec55abe700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec50069e20 con 0x7fec500f6510 2024-02-20T00:21:49.685 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:49.682+0000 7fec4f7fe700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fec500f6510 0x7fec5006b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:21:49.685 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:49.682+0000 7fec4f7fe700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fec500f6510 0x7fec5006b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:42876/0 (socket says 172.21.15.132:42876) 2024-02-20T00:21:49.685 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:49.682+0000 7fec4f7fe700 1 -- 172.21.15.132:0/556195486 learned_addr learned my addr 172.21.15.132:0/556195486 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:21:49.685 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:49.682+0000 7fec4f7fe700 1 -- 172.21.15.132:0/556195486 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec44005040 con 0x7fec500f6510 2024-02-20T00:21:49.685 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:49.682+0000 7fec4f7fe700 1 --2- 172.21.15.132:0/556195486 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fec500f6510 0x7fec5006b9a0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 rx=0x7fec440089f0 tx=0x7fec4400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:21:49.686 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:49.683+0000 7fec37fff700 1 -- 172.21.15.132:0/556195486 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fec44009320 con 0x7fec500f6510 2024-02-20T00:21:49.686 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:49.683+0000 7fec55abe700 1 -- 172.21.15.132:0/556195486 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec50069fc0 con 0x7fec500f6510 2024-02-20T00:21:49.686 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:49.683+0000 7fec37fff700 1 -- 172.21.15.132:0/556195486 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fec44013380 con 0x7fec500f6510 2024-02-20T00:21:49.686 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:49.683+0000 7fec37fff700 1 -- 172.21.15.132:0/556195486 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fec4401ed80 con 0x7fec500f6510 2024-02-20T00:21:49.686 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:49.683+0000 7fec55abe700 1 -- 172.21.15.132:0/556195486 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec5006a450 con 0x7fec500f6510 2024-02-20T00:21:49.686 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:49.683+0000 7fec37fff700 1 -- 172.21.15.132:0/556195486 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fec4401e420 con 0x7fec500f6510 2024-02-20T00:21:49.687 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:49.683+0000 7fec55abe700 1 -- 172.21.15.132:0/556195486 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec50059100 con 0x7fec500f6510 2024-02-20T00:21:49.687 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:49.684+0000 7fec37fff700 1 --2- 172.21.15.132:0/556195486 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fec3802ff60 0x7fec38032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:21:49.687 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:49.684+0000 7fec37fff700 1 -- 172.21.15.132:0/556195486 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fec44041060 con 0x7fec500f6510 2024-02-20T00:21:49.687 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:49.684+0000 7fec4effd700 1 --2- 172.21.15.132:0/556195486 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fec3802ff60 0x7fec38032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:21:49.687 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:49.684+0000 7fec4effd700 1 --2- 172.21.15.132:0/556195486 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fec3802ff60 0x7fec38032420 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7fec400096a0 tx=0x7fec40006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:21:49.689 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:49.688+0000 7fec37fff700 1 -- 172.21.15.132:0/556195486 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fec440095f0 con 0x7fec500f6510 2024-02-20T00:21:49.894 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:49.892+0000 7fec55abe700 1 -- 172.21.15.132:0/556195486 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fec50045ad0 con 0x7fec500f6510 2024-02-20T00:21:49.895 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:49.893+0000 7fec37fff700 1 -- 172.21.15.132:0/556195486 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fec4401cb10 con 0x7fec500f6510 2024-02-20T00:21:49.895 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:21:49.895 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:21:49.896 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:49.895+0000 7fec55abe700 1 -- 172.21.15.132:0/556195486 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fec3802ff60 msgr2=0x7fec38032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:21:49.897 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:49.895+0000 7fec55abe700 1 --2- 172.21.15.132:0/556195486 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fec3802ff60 0x7fec38032420 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7fec400096a0 tx=0x7fec40006b40).stop 2024-02-20T00:21:49.897 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:49.895+0000 7fec55abe700 1 -- 172.21.15.132:0/556195486 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fec500f6510 msgr2=0x7fec5006b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:21:49.897 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:49.895+0000 7fec55abe700 1 --2- 172.21.15.132:0/556195486 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fec500f6510 0x7fec5006b9a0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 rx=0x7fec440089f0 tx=0x7fec4400b040).stop 2024-02-20T00:21:49.897 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:49.895+0000 7fec55abe700 1 -- 172.21.15.132:0/556195486 shutdown_connections 2024-02-20T00:21:49.897 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:49.895+0000 7fec55abe700 1 --2- 172.21.15.132:0/556195486 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fec3802ff60 0x7fec38032420 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:21:49.897 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:49.895+0000 7fec55abe700 1 --2- 172.21.15.132:0/556195486 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fec500f6510 0x7fec5006b9a0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:21:49.897 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:49.895+0000 7fec55abe700 1 -- 172.21.15.132:0/556195486 >> 172.21.15.132:0/556195486 conn(0x7fec500f1c70 msgr2=0x7fec500f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:21:49.897 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:49.895+0000 7fec55abe700 1 -- 172.21.15.132:0/556195486 shutdown_connections 2024-02-20T00:21:49.897 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:49.895+0000 7fec55abe700 1 -- 172.21.15.132:0/556195486 wait complete. 2024-02-20T00:21:49.897 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:21:50.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:21:50 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:21:49.893900+0000 mon.smithi084 (mon.0) 336 : audit [DBG] from='client.? 172.21.15.132:0/556195486' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:21:51.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:21:51 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:21:49.486770+0000 mgr.smithi084.bmqrnf (mgr.14162) 367 : cluster [DBG] pgmap v351: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:21:51.647 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:21:51.648 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:21:53.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:21:53 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:21:51.487189+0000 mgr.smithi084.bmqrnf (mgr.14162) 368 : cluster [DBG] pgmap v352: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:21:53.504 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:53.485+0000 7f28c1ae3700 1 -- 172.21.15.132:0/3759001858 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f28bc0f6510 msgr2=0x7f28bc0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:21:53.504 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:53.485+0000 7f28c1ae3700 1 --2- 172.21.15.132:0/3759001858 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f28bc0f6510 0x7f28bc0f6930 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 rx=0x7f28b0009fb0 tx=0x7f28b0009b30).stop 2024-02-20T00:21:53.504 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:53.485+0000 7f28c1ae3700 1 -- 172.21.15.132:0/3759001858 shutdown_connections 2024-02-20T00:21:53.504 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:53.485+0000 7f28c1ae3700 1 --2- 172.21.15.132:0/3759001858 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f28bc0f6510 0x7f28bc0f6930 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:21:53.504 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:53.485+0000 7f28c1ae3700 1 -- 172.21.15.132:0/3759001858 >> 172.21.15.132:0/3759001858 conn(0x7f28bc0f1c70 msgr2=0x7f28bc0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:21:53.504 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:53.485+0000 7f28c1ae3700 1 -- 172.21.15.132:0/3759001858 shutdown_connections 2024-02-20T00:21:53.504 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:53.485+0000 7f28c1ae3700 1 -- 172.21.15.132:0/3759001858 wait complete. 2024-02-20T00:21:53.504 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:53.486+0000 7f28c1ae3700 1 Processor -- start 2024-02-20T00:21:53.504 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:53.486+0000 7f28c1ae3700 1 -- start start 2024-02-20T00:21:53.505 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:53.486+0000 7f28c1ae3700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f28bc0f6510 0x7f28bc06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:21:53.505 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:53.486+0000 7f28c1ae3700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28bc069e20 con 0x7f28bc0f6510 2024-02-20T00:21:53.505 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:53.486+0000 7f28bb7fe700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f28bc0f6510 0x7f28bc06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:21:53.505 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:53.487+0000 7f28bb7fe700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f28bc0f6510 0x7f28bc06b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:42896/0 (socket says 172.21.15.132:42896) 2024-02-20T00:21:53.505 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:53.487+0000 7f28bb7fe700 1 -- 172.21.15.132:0/1528447240 learned_addr learned my addr 172.21.15.132:0/1528447240 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:21:53.505 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:53.487+0000 7f28bb7fe700 1 -- 172.21.15.132:0/1528447240 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28b0005040 con 0x7f28bc0f6510 2024-02-20T00:21:53.505 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:53.487+0000 7f28bb7fe700 1 --2- 172.21.15.132:0/1528447240 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f28bc0f6510 0x7f28bc06b9a0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 rx=0x7f28b0005010 tx=0x7f28b000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:21:53.505 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:53.487+0000 7f28a3fff700 1 -- 172.21.15.132:0/1528447240 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f28b0009320 con 0x7f28bc0f6510 2024-02-20T00:21:53.505 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:53.487+0000 7f28c1ae3700 1 -- 172.21.15.132:0/1528447240 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28bc069fc0 con 0x7f28bc0f6510 2024-02-20T00:21:53.505 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:53.487+0000 7f28a3fff700 1 -- 172.21.15.132:0/1528447240 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f28b0013380 con 0x7f28bc0f6510 2024-02-20T00:21:53.505 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:53.487+0000 7f28a3fff700 1 -- 172.21.15.132:0/1528447240 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f28b0014e50 con 0x7f28bc0f6510 2024-02-20T00:21:53.505 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:53.487+0000 7f28c1ae3700 1 -- 172.21.15.132:0/1528447240 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28bc06a450 con 0x7f28bc0f6510 2024-02-20T00:21:53.505 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:53.488+0000 7f28a3fff700 1 -- 172.21.15.132:0/1528447240 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f28b00263a0 con 0x7f28bc0f6510 2024-02-20T00:21:53.505 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:53.488+0000 7f28a3fff700 1 --2- 172.21.15.132:0/1528447240 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f28a402ff10 0x7f28a40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:21:53.506 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:53.488+0000 7f28a3fff700 1 -- 172.21.15.132:0/1528447240 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f28b0040fc0 con 0x7f28bc0f6510 2024-02-20T00:21:53.506 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:53.488+0000 7f28c1ae3700 1 -- 172.21.15.132:0/1528447240 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28bc059100 con 0x7f28bc0f6510 2024-02-20T00:21:53.506 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:53.488+0000 7f28baffd700 1 --2- 172.21.15.132:0/1528447240 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f28a402ff10 0x7f28a40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:21:53.506 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:53.489+0000 7f28baffd700 1 --2- 172.21.15.132:0/1528447240 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f28a402ff10 0x7f28a40323d0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f28ac0096a0 tx=0x7f28ac006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:21:53.506 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:53.493+0000 7f28a3fff700 1 -- 172.21.15.132:0/1528447240 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f28b0012920 con 0x7f28bc0f6510 2024-02-20T00:21:53.698 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:53.697+0000 7f28c1ae3700 1 -- 172.21.15.132:0/1528447240 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f28bc06ab50 con 0x7f28bc0f6510 2024-02-20T00:21:53.699 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:53.697+0000 7f28a3fff700 1 -- 172.21.15.132:0/1528447240 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f28b0012330 con 0x7f28bc0f6510 2024-02-20T00:21:53.699 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:21:53.699 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:21:53.700 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:53.699+0000 7f28c1ae3700 1 -- 172.21.15.132:0/1528447240 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f28a402ff10 msgr2=0x7f28a40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:21:53.701 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:53.699+0000 7f28c1ae3700 1 --2- 172.21.15.132:0/1528447240 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f28a402ff10 0x7f28a40323d0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f28ac0096a0 tx=0x7f28ac006b40).stop 2024-02-20T00:21:53.701 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:53.699+0000 7f28c1ae3700 1 -- 172.21.15.132:0/1528447240 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f28bc0f6510 msgr2=0x7f28bc06b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:21:53.701 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:53.699+0000 7f28c1ae3700 1 --2- 172.21.15.132:0/1528447240 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f28bc0f6510 0x7f28bc06b9a0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 rx=0x7f28b0005010 tx=0x7f28b000b040).stop 2024-02-20T00:21:53.701 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:53.699+0000 7f28c1ae3700 1 -- 172.21.15.132:0/1528447240 shutdown_connections 2024-02-20T00:21:53.701 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:53.699+0000 7f28c1ae3700 1 --2- 172.21.15.132:0/1528447240 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f28a402ff10 0x7f28a40323d0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:21:53.701 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:53.699+0000 7f28c1ae3700 1 --2- 172.21.15.132:0/1528447240 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f28bc0f6510 0x7f28bc06b9a0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:21:53.701 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:53.699+0000 7f28c1ae3700 1 -- 172.21.15.132:0/1528447240 >> 172.21.15.132:0/1528447240 conn(0x7f28bc0f1c70 msgr2=0x7f28bc0f26f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:21:53.701 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:53.699+0000 7f28c1ae3700 1 -- 172.21.15.132:0/1528447240 shutdown_connections 2024-02-20T00:21:53.701 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:53.699+0000 7f28c1ae3700 1 -- 172.21.15.132:0/1528447240 wait complete. 2024-02-20T00:21:53.702 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:21:54.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:21:54 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:21:53.698220+0000 mon.smithi084 (mon.0) 337 : audit [DBG] from='client.? 172.21.15.132:0/1528447240' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:21:55.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:21:55 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:21:53.487477+0000 mgr.smithi084.bmqrnf (mgr.14162) 369 : cluster [DBG] pgmap v353: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:21:55.443 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:21:55.443 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:21:57.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:21:57 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:21:55.487798+0000 mgr.smithi084.bmqrnf (mgr.14162) 370 : cluster [DBG] pgmap v354: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:21:57.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:21:57 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:21:56.985149+0000 mon.smithi084 (mon.0) 338 : audit [INF] from='mgr.14162 172.21.15.84:0/2152933216' entity='mgr.smithi084.bmqrnf' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi084.bmqrnf/mirror_snapshot_schedule"}]: dispatch 2024-02-20T00:21:57.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:21:57 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:21:56.988981+0000 mon.smithi084 (mon.0) 339 : audit [INF] from='mgr.14162 172.21.15.84:0/2152933216' entity='mgr.smithi084.bmqrnf' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi084.bmqrnf/trash_purge_schedule"}]: dispatch 2024-02-20T00:21:57.540 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:57.535+0000 7fc80af33700 1 -- 172.21.15.132:0/3074122759 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc8040f5410 msgr2=0x7fc8040f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:21:57.540 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:57.535+0000 7fc80af33700 1 --2- 172.21.15.132:0/3074122759 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc8040f5410 0x7fc8040f5830 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 rx=0x7fc7f8009fb0 tx=0x7fc7f8009b30).stop 2024-02-20T00:21:57.541 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:57.535+0000 7fc80af33700 1 -- 172.21.15.132:0/3074122759 shutdown_connections 2024-02-20T00:21:57.541 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:57.535+0000 7fc80af33700 1 --2- 172.21.15.132:0/3074122759 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc8040f5410 0x7fc8040f5830 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:21:57.541 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:57.535+0000 7fc80af33700 1 -- 172.21.15.132:0/3074122759 >> 172.21.15.132:0/3074122759 conn(0x7fc8040f0b70 msgr2=0x7fc8040f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:21:57.541 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:57.535+0000 7fc80af33700 1 -- 172.21.15.132:0/3074122759 shutdown_connections 2024-02-20T00:21:57.541 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:57.535+0000 7fc80af33700 1 -- 172.21.15.132:0/3074122759 wait complete. 2024-02-20T00:21:57.541 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:57.536+0000 7fc80af33700 1 Processor -- start 2024-02-20T00:21:57.541 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:57.536+0000 7fc80af33700 1 -- start start 2024-02-20T00:21:57.541 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:57.536+0000 7fc80af33700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc8040f5410 0x7fc8041067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:21:57.541 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:57.536+0000 7fc80af33700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc804106cd0 con 0x7fc8040f5410 2024-02-20T00:21:57.541 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:57.536+0000 7fc808ccf700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc8040f5410 0x7fc8041067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:21:57.541 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:57.536+0000 7fc808ccf700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc8040f5410 0x7fc8041067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:53536/0 (socket says 172.21.15.132:53536) 2024-02-20T00:21:57.542 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:57.536+0000 7fc808ccf700 1 -- 172.21.15.132:0/3155986935 learned_addr learned my addr 172.21.15.132:0/3155986935 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:21:57.542 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:57.537+0000 7fc808ccf700 1 -- 172.21.15.132:0/3155986935 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7f8005040 con 0x7fc8040f5410 2024-02-20T00:21:57.542 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:57.537+0000 7fc808ccf700 1 --2- 172.21.15.132:0/3155986935 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc8040f5410 0x7fc8041067e0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 rx=0x7fc7f80089f0 tx=0x7fc7f800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:21:57.542 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:57.537+0000 7fc8017fa700 1 -- 172.21.15.132:0/3155986935 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc7f8009320 con 0x7fc8040f5410 2024-02-20T00:21:57.542 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:57.537+0000 7fc80af33700 1 -- 172.21.15.132:0/3155986935 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc804106ed0 con 0x7fc8040f5410 2024-02-20T00:21:57.542 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:57.537+0000 7fc8017fa700 1 -- 172.21.15.132:0/3155986935 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc7f8013380 con 0x7fc8040f5410 2024-02-20T00:21:57.542 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:57.537+0000 7fc8017fa700 1 -- 172.21.15.132:0/3155986935 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc7f801ed80 con 0x7fc8040f5410 2024-02-20T00:21:57.543 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:57.537+0000 7fc80af33700 1 -- 172.21.15.132:0/3155986935 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc804107360 con 0x7fc8040f5410 2024-02-20T00:21:57.543 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:57.538+0000 7fc8017fa700 1 -- 172.21.15.132:0/3155986935 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc7f801e420 con 0x7fc8040f5410 2024-02-20T00:21:57.543 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:57.538+0000 7fc80af33700 1 -- 172.21.15.132:0/3155986935 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc804045ad0 con 0x7fc8040f5410 2024-02-20T00:21:57.543 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:57.538+0000 7fc8017fa700 1 --2- 172.21.15.132:0/3155986935 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fc7f402ffb0 0x7fc7f4032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:21:57.543 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:57.538+0000 7fc8017fa700 1 -- 172.21.15.132:0/3155986935 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc7f8040fc0 con 0x7fc8040f5410 2024-02-20T00:21:57.543 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:57.538+0000 7fc803fff700 1 --2- 172.21.15.132:0/3155986935 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fc7f402ffb0 0x7fc7f4032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:21:57.543 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:57.539+0000 7fc803fff700 1 --2- 172.21.15.132:0/3155986935 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fc7f402ffb0 0x7fc7f4032470 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7fc7f00096a0 tx=0x7fc7f0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:21:57.545 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:57.543+0000 7fc8017fa700 1 -- 172.21.15.132:0/3155986935 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc7f80095f0 con 0x7fc8040f5410 2024-02-20T00:21:57.753 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:57.750+0000 7fc80af33700 1 -- 172.21.15.132:0/3155986935 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc804108210 con 0x7fc8040f5410 2024-02-20T00:21:57.754 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:57.751+0000 7fc8017fa700 1 -- 172.21.15.132:0/3155986935 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc7f801cb10 con 0x7fc8040f5410 2024-02-20T00:21:57.754 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:21:57.755 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:21:57.756 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:57.753+0000 7fc80af33700 1 -- 172.21.15.132:0/3155986935 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fc7f402ffb0 msgr2=0x7fc7f4032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:21:57.756 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:57.753+0000 7fc80af33700 1 --2- 172.21.15.132:0/3155986935 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fc7f402ffb0 0x7fc7f4032470 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7fc7f00096a0 tx=0x7fc7f0006b40).stop 2024-02-20T00:21:57.757 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:57.753+0000 7fc80af33700 1 -- 172.21.15.132:0/3155986935 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc8040f5410 msgr2=0x7fc8041067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:21:57.757 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:57.753+0000 7fc80af33700 1 --2- 172.21.15.132:0/3155986935 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc8040f5410 0x7fc8041067e0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 rx=0x7fc7f80089f0 tx=0x7fc7f800b040).stop 2024-02-20T00:21:57.757 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:57.753+0000 7fc80af33700 1 -- 172.21.15.132:0/3155986935 shutdown_connections 2024-02-20T00:21:57.757 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:57.753+0000 7fc80af33700 1 --2- 172.21.15.132:0/3155986935 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7fc7f402ffb0 0x7fc7f4032470 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:21:57.757 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:57.753+0000 7fc80af33700 1 --2- 172.21.15.132:0/3155986935 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc8040f5410 0x7fc8041067e0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:21:57.757 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:57.753+0000 7fc80af33700 1 -- 172.21.15.132:0/3155986935 >> 172.21.15.132:0/3155986935 conn(0x7fc8040f0b70 msgr2=0x7fc8040f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:21:57.758 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:57.753+0000 7fc80af33700 1 -- 172.21.15.132:0/3155986935 shutdown_connections 2024-02-20T00:21:57.758 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:21:57.753+0000 7fc80af33700 1 -- 172.21.15.132:0/3155986935 wait complete. 2024-02-20T00:21:57.758 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:21:58.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:21:58 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:21:57.752784+0000 mon.smithi084 (mon.0) 340 : audit [DBG] from='client.? 172.21.15.132:0/3155986935' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:21:59.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:21:59 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:21:57.488124+0000 mgr.smithi084.bmqrnf (mgr.14162) 371 : cluster [DBG] pgmap v355: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:21:59.465 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:21:59.465 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:22:01.293 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:22:01.285+0000 7f4e3dc31700 1 -- 172.21.15.132:0/3201138653 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4e380f6510 msgr2=0x7f4e380f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:22:01.294 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:22:01.285+0000 7f4e3dc31700 1 --2- 172.21.15.132:0/3201138653 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4e380f6510 0x7f4e380f6930 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 rx=0x7f4e2c009fb0 tx=0x7f4e2c009b30).stop 2024-02-20T00:22:01.294 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:22:01.285+0000 7f4e3dc31700 1 -- 172.21.15.132:0/3201138653 shutdown_connections 2024-02-20T00:22:01.294 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:22:01.285+0000 7f4e3dc31700 1 --2- 172.21.15.132:0/3201138653 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4e380f6510 0x7f4e380f6930 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:22:01.294 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:22:01.285+0000 7f4e3dc31700 1 -- 172.21.15.132:0/3201138653 >> 172.21.15.132:0/3201138653 conn(0x7f4e380f1c70 msgr2=0x7f4e380f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:22:01.294 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:22:01.285+0000 7f4e3dc31700 1 -- 172.21.15.132:0/3201138653 shutdown_connections 2024-02-20T00:22:01.294 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:22:01.285+0000 7f4e3dc31700 1 -- 172.21.15.132:0/3201138653 wait complete. 2024-02-20T00:22:01.294 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:22:01.286+0000 7f4e3dc31700 1 Processor -- start 2024-02-20T00:22:01.294 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:22:01.286+0000 7f4e3dc31700 1 -- start start 2024-02-20T00:22:01.294 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:22:01.287+0000 7f4e3dc31700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4e380f6510 0x7f4e381067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:22:01.294 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:22:01.287+0000 7f4e3dc31700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e38106cd0 con 0x7f4e380f6510 2024-02-20T00:22:01.294 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:22:01.287+0000 7f4e377fe700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4e380f6510 0x7f4e381067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:22:01.295 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:22:01.287+0000 7f4e377fe700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4e380f6510 0x7f4e381067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:53560/0 (socket says 172.21.15.132:53560) 2024-02-20T00:22:01.295 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:22:01.287+0000 7f4e377fe700 1 -- 172.21.15.132:0/2356808292 learned_addr learned my addr 172.21.15.132:0/2356808292 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:22:01.295 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:22:01.287+0000 7f4e377fe700 1 -- 172.21.15.132:0/2356808292 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e2c005040 con 0x7f4e380f6510 2024-02-20T00:22:01.295 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:22:01.287+0000 7f4e377fe700 1 --2- 172.21.15.132:0/2356808292 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4e380f6510 0x7f4e381067e0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 rx=0x7f4e2c005010 tx=0x7f4e2c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:22:01.295 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:22:01.287+0000 7f4e1ffff700 1 -- 172.21.15.132:0/2356808292 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4e2c009320 con 0x7f4e380f6510 2024-02-20T00:22:01.295 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:22:01.287+0000 7f4e3dc31700 1 -- 172.21.15.132:0/2356808292 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e38106ed0 con 0x7f4e380f6510 2024-02-20T00:22:01.295 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:22:01.287+0000 7f4e1ffff700 1 -- 172.21.15.132:0/2356808292 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4e2c013380 con 0x7f4e380f6510 2024-02-20T00:22:01.295 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:22:01.287+0000 7f4e1ffff700 1 -- 172.21.15.132:0/2356808292 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4e2c014e50 con 0x7f4e380f6510 2024-02-20T00:22:01.295 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:22:01.287+0000 7f4e3dc31700 1 -- 172.21.15.132:0/2356808292 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e38107360 con 0x7f4e380f6510 2024-02-20T00:22:01.295 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:22:01.288+0000 7f4e1ffff700 1 -- 172.21.15.132:0/2356808292 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4e2c0263a0 con 0x7f4e380f6510 2024-02-20T00:22:01.295 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:22:01.288+0000 7f4e3dc31700 1 -- 172.21.15.132:0/2356808292 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e38045ad0 con 0x7f4e380f6510 2024-02-20T00:22:01.296 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:22:01.288+0000 7f4e1ffff700 1 --2- 172.21.15.132:0/2356808292 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f4e2002ff10 0x7f4e200323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:22:01.296 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:22:01.288+0000 7f4e1ffff700 1 -- 172.21.15.132:0/2356808292 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4e2c040ec0 con 0x7f4e380f6510 2024-02-20T00:22:01.296 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:22:01.289+0000 7f4e36ffd700 1 --2- 172.21.15.132:0/2356808292 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f4e2002ff10 0x7f4e200323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:22:01.296 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:22:01.289+0000 7f4e36ffd700 1 --2- 172.21.15.132:0/2356808292 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f4e2002ff10 0x7f4e200323d0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f4e280096a0 tx=0x7f4e28006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:22:01.296 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:22:01.293+0000 7f4e1ffff700 1 -- 172.21.15.132:0/2356808292 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4e2c0126f0 con 0x7f4e380f6510 2024-02-20T00:22:01.357 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:22:01 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:21:59.488479+0000 mgr.smithi084.bmqrnf (mgr.14162) 372 : cluster [DBG] pgmap v356: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:22:01.499 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:22:01.496+0000 7f4e3dc31700 1 -- 172.21.15.132:0/2356808292 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4e38108210 con 0x7f4e380f6510 2024-02-20T00:22:01.499 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:22:01.497+0000 7f4e1ffff700 1 -- 172.21.15.132:0/2356808292 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4e2c012330 con 0x7f4e380f6510 2024-02-20T00:22:01.500 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:22:01.500 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:22:01.501 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:22:01.499+0000 7f4e3dc31700 1 -- 172.21.15.132:0/2356808292 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f4e2002ff10 msgr2=0x7f4e200323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:22:01.501 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:22:01.499+0000 7f4e3dc31700 1 --2- 172.21.15.132:0/2356808292 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f4e2002ff10 0x7f4e200323d0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f4e280096a0 tx=0x7f4e28006b40).stop 2024-02-20T00:22:01.501 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:22:01.499+0000 7f4e3dc31700 1 -- 172.21.15.132:0/2356808292 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4e380f6510 msgr2=0x7f4e381067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:22:01.501 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:22:01.499+0000 7f4e3dc31700 1 --2- 172.21.15.132:0/2356808292 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4e380f6510 0x7f4e381067e0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 rx=0x7f4e2c005010 tx=0x7f4e2c00b040).stop 2024-02-20T00:22:01.502 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:22:01.499+0000 7f4e3dc31700 1 -- 172.21.15.132:0/2356808292 shutdown_connections 2024-02-20T00:22:01.502 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:22:01.499+0000 7f4e3dc31700 1 --2- 172.21.15.132:0/2356808292 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f4e2002ff10 0x7f4e200323d0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:22:01.502 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:22:01.499+0000 7f4e3dc31700 1 --2- 172.21.15.132:0/2356808292 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4e380f6510 0x7f4e381067e0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:22:01.502 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:22:01.499+0000 7f4e3dc31700 1 -- 172.21.15.132:0/2356808292 >> 172.21.15.132:0/2356808292 conn(0x7f4e380f1c70 msgr2=0x7f4e380f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:22:01.502 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:22:01.499+0000 7f4e3dc31700 1 -- 172.21.15.132:0/2356808292 shutdown_connections 2024-02-20T00:22:01.502 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:22:01.499+0000 7f4e3dc31700 1 -- 172.21.15.132:0/2356808292 wait complete. 2024-02-20T00:22:01.502 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:22:02.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:22:02 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:22:01.498604+0000 mon.smithi084 (mon.0) 341 : audit [DBG] from='client.? 172.21.15.132:0/2356808292' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:22:03.238 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T00:22:03.238 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T00:22:03.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:22:03 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:22:01.488775+0000 mgr.smithi084.bmqrnf (mgr.14162) 373 : cluster [DBG] pgmap v357: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:22:05.267 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:22:05.263+0000 7f0f6bfd8700 1 -- 172.21.15.132:0/1889909120 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0f640f4740 msgr2=0x7f0f640f4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:22:05.267 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:22:05.263+0000 7f0f6bfd8700 1 --2- 172.21.15.132:0/1889909120 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0f640f4740 0x7f0f640f4b60 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 rx=0x7f0f54009fb0 tx=0x7f0f54009b30).stop 2024-02-20T00:22:05.267 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:22:05.263+0000 7f0f6bfd8700 1 -- 172.21.15.132:0/1889909120 shutdown_connections 2024-02-20T00:22:05.267 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:22:05.263+0000 7f0f6bfd8700 1 --2- 172.21.15.132:0/1889909120 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0f640f4740 0x7f0f640f4b60 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:22:05.268 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:22:05.263+0000 7f0f6bfd8700 1 -- 172.21.15.132:0/1889909120 >> 172.21.15.132:0/1889909120 conn(0x7f0f640f0260 msgr2=0x7f0f640f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:22:05.268 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:22:05.263+0000 7f0f6bfd8700 1 -- 172.21.15.132:0/1889909120 shutdown_connections 2024-02-20T00:22:05.268 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:22:05.263+0000 7f0f6bfd8700 1 -- 172.21.15.132:0/1889909120 wait complete. 2024-02-20T00:22:05.268 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:22:05.264+0000 7f0f6bfd8700 1 Processor -- start 2024-02-20T00:22:05.268 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:22:05.264+0000 7f0f6bfd8700 1 -- start start 2024-02-20T00:22:05.268 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:22:05.264+0000 7f0f6bfd8700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0f640f4740 0x7f0f640f72f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:22:05.268 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:22:05.264+0000 7f0f6bfd8700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f640f5770 con 0x7f0f640f4740 2024-02-20T00:22:05.268 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:22:05.265+0000 7f0f69d74700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0f640f4740 0x7f0f640f72f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:22:05.268 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:22:05.265+0000 7f0f69d74700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0f640f4740 0x7f0f640f72f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.132:53576/0 (socket says 172.21.15.132:53576) 2024-02-20T00:22:05.268 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:22:05.265+0000 7f0f69d74700 1 -- 172.21.15.132:0/55235592 learned_addr learned my addr 172.21.15.132:0/55235592 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-02-20T00:22:05.268 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:22:05.265+0000 7f0f69d74700 1 -- 172.21.15.132:0/55235592 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f54005040 con 0x7f0f640f4740 2024-02-20T00:22:05.268 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:22:05.265+0000 7f0f69d74700 1 --2- 172.21.15.132:0/55235592 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0f640f4740 0x7f0f640f72f0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 rx=0x7f0f540089f0 tx=0x7f0f5400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:22:05.270 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:22:05.265+0000 7f0f5a7fc700 1 -- 172.21.15.132:0/55235592 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0f54009320 con 0x7f0f640f4740 2024-02-20T00:22:05.270 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:22:05.265+0000 7f0f6bfd8700 1 -- 172.21.15.132:0/55235592 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f640f5910 con 0x7f0f640f4740 2024-02-20T00:22:05.270 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:22:05.265+0000 7f0f5a7fc700 1 -- 172.21.15.132:0/55235592 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0f54013380 con 0x7f0f640f4740 2024-02-20T00:22:05.270 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:22:05.265+0000 7f0f5a7fc700 1 -- 172.21.15.132:0/55235592 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0f5401ed80 con 0x7f0f640f4740 2024-02-20T00:22:05.270 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:22:05.265+0000 7f0f6bfd8700 1 -- 172.21.15.132:0/55235592 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f640f5da0 con 0x7f0f640f4740 2024-02-20T00:22:05.270 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:22:05.266+0000 7f0f5a7fc700 1 -- 172.21.15.132:0/55235592 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0f54014420 con 0x7f0f640f4740 2024-02-20T00:22:05.270 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:22:05.266+0000 7f0f5a7fc700 1 --2- 172.21.15.132:0/55235592 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f0f5002ffb0 0x7f0f50032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-20T00:22:05.270 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:22:05.266+0000 7f0f5a7fc700 1 -- 172.21.15.132:0/55235592 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0f54040f20 con 0x7f0f640f4740 2024-02-20T00:22:05.270 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:22:05.266+0000 7f0f6bfd8700 1 -- 172.21.15.132:0/55235592 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f64059100 con 0x7f0f640f4740 2024-02-20T00:22:05.270 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:22:05.266+0000 7f0f69573700 1 --2- 172.21.15.132:0/55235592 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f0f5002ffb0 0x7f0f50032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-20T00:22:05.271 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:22:05.267+0000 7f0f69573700 1 --2- 172.21.15.132:0/55235592 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f0f5002ffb0 0x7f0f50032470 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7f0f600096a0 tx=0x7f0f60006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T00:22:05.273 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:22:05.271+0000 7f0f5a7fc700 1 -- 172.21.15.132:0/55235592 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0f5401c910 con 0x7f0f640f4740 2024-02-20T00:22:05.358 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:22:05 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: cluster 2024-02-20T00:22:03.489122+0000 mgr.smithi084.bmqrnf (mgr.14162) 374 : cluster [DBG] pgmap v358: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T00:22:05.479 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:22:05.477+0000 7f0f6bfd8700 1 -- 172.21.15.132:0/55235592 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0f640f64a0 con 0x7f0f640f4740 2024-02-20T00:22:05.480 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:22:05.478+0000 7f0f5a7fc700 1 -- 172.21.15.132:0/55235592 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0f5401c320 con 0x7f0f640f4740 2024-02-20T00:22:05.480 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-02-20T00:22:05.480 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"2a2e6c42-cf84-11ee-95bb-87774f69a715","modified":"2024-02-20T00:08:53.282221Z","created":"2024-02-20T00:08:53.282221Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T00:22:05.482 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:22:05.479+0000 7f0f6bfd8700 1 -- 172.21.15.132:0/55235592 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f0f5002ffb0 msgr2=0x7f0f50032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:22:05.482 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:22:05.479+0000 7f0f6bfd8700 1 --2- 172.21.15.132:0/55235592 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f0f5002ffb0 0x7f0f50032470 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7f0f600096a0 tx=0x7f0f60006b40).stop 2024-02-20T00:22:05.482 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:22:05.479+0000 7f0f6bfd8700 1 -- 172.21.15.132:0/55235592 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0f640f4740 msgr2=0x7f0f640f72f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T00:22:05.482 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:22:05.479+0000 7f0f6bfd8700 1 --2- 172.21.15.132:0/55235592 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0f640f4740 0x7f0f640f72f0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 rx=0x7f0f540089f0 tx=0x7f0f5400b040).stop 2024-02-20T00:22:05.482 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:22:05.479+0000 7f0f6bfd8700 1 -- 172.21.15.132:0/55235592 shutdown_connections 2024-02-20T00:22:05.482 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:22:05.479+0000 7f0f6bfd8700 1 --2- 172.21.15.132:0/55235592 >> [v2:172.21.15.84:6800/1303158740,v1:172.21.15.84:6801/1303158740] conn(0x7f0f5002ffb0 0x7f0f50032470 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:22:05.482 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:22:05.479+0000 7f0f6bfd8700 1 --2- 172.21.15.132:0/55235592 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0f640f4740 0x7f0f640f72f0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-20T00:22:05.482 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:22:05.479+0000 7f0f6bfd8700 1 -- 172.21.15.132:0/55235592 >> 172.21.15.132:0/55235592 conn(0x7f0f640f0260 msgr2=0x7f0f640f0cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T00:22:05.483 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:22:05.479+0000 7f0f6bfd8700 1 -- 172.21.15.132:0/55235592 shutdown_connections 2024-02-20T00:22:05.483 INFO:teuthology.orchestra.run.smithi132.stderr:2024-02-20T00:22:05.480+0000 7f0f6bfd8700 1 -- 172.21.15.132:0/55235592 wait complete. 2024-02-20T00:22:05.483 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-02-20T00:22:06.226 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_19854089e18d4f65dda2b6cd74e737367c2514bd/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds 2024-02-20T00:22:06.227 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-02-20T00:22:06.227 DEBUG:teuthology.orchestra.run.smithi084:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-02-20T00:22:06.252 DEBUG:teuthology.orchestra.run.smithi132:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-02-20T00:22:06.271 INFO:tasks.cephadm:Stopping all daemons... 2024-02-20T00:22:06.271 INFO:tasks.cephadm.mon.smithi084:Stopping mon.smithi084... 2024-02-20T00:22:06.271 DEBUG:teuthology.orchestra.run.smithi084:> sudo systemctl stop ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715@mon.smithi084 2024-02-20T00:22:06.333 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:22:06 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: audit 2024-02-20T00:22:05.479281+0000 mon.smithi084 (mon.0) 342 : audit [DBG] from='client.? 172.21.15.132:0/55235592' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T00:22:06.608 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:22:06 smithi084 systemd[1]: Stopping Ceph mon.smithi084 for 2a2e6c42-cf84-11ee-95bb-87774f69a715... 2024-02-20T00:22:06.608 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:22:06 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:22:06.476+0000 7fb594135700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi084 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-02-20T00:22:06.608 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:22:06 smithi084 ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084[38114]: debug 2024-02-20T00:22:06.476+0000 7fb594135700 -1 mon.smithi084@0(leader) e1 *** Got Signal Terminated *** 2024-02-20T00:22:07.108 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 20 00:22:06 smithi084 podman[43913]: ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715-mon.smithi084 2024-02-20T00:22:07.179 DEBUG:teuthology.orchestra.run.smithi084:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715@mon.smithi084.service' 2024-02-20T00:22:07.267 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-20T00:22:07.268 INFO:tasks.cephadm.mon.smithi084:Stopped mon.smithi084 2024-02-20T00:22:07.268 INFO:tasks.cephadm.mon.smithi132:Stopping mon.smithi132... 2024-02-20T00:22:07.268 DEBUG:teuthology.orchestra.run.smithi132:> sudo systemctl stop ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715@mon.smithi132 2024-02-20T00:22:07.307 INFO:teuthology.orchestra.run.smithi132.stderr:Failed to stop ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715@mon.smithi132.service: Unit ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715@mon.smithi132.service not loaded. 2024-02-20T00:22:07.308 DEBUG:teuthology.orchestra.run:got remote process result: 5 2024-02-20T00:22:07.309 ERROR:tasks.cephadm:Failed to stop "ceph.mon.smithi132" Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_19854089e18d4f65dda2b6cd74e737367c2514bd/qa/tasks/cephadm.py", line 512, in ceph_bootstrap yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_19854089e18d4f65dda2b6cd74e737367c2514bd/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_19854089e18d4f65dda2b6cd74e737367c2514bd/qa/tasks/cephadm.py", line 531, in ceph_bootstrap ctx.daemons.get_daemon(type_, id_, cluster).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/daemon/cephadmunit.py", line 154, in stop self.remote.sh(self.stop_cmd) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 97, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi132 with status 5: 'sudo systemctl stop ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715@mon.smithi132' 2024-02-20T00:22:07.310 INFO:tasks.cephadm:Archiving crash dumps... 2024-02-20T00:22:07.312 DEBUG:teuthology.misc:Transferring archived files from smithi084:/var/lib/ceph/2a2e6c42-cf84-11ee-95bb-87774f69a715/crash to /home/teuthworker/archive/gabrioux-2024-02-19_22:54:55-orch:cephadm-wip-guits-testing-6-2024-02-19-2119-pacific-distro-default-smithi/7568131/remote/ubuntu@smithi084.front.sepia.ceph.com/crash 2024-02-20T00:22:07.313 DEBUG:teuthology.orchestra.run.smithi084:> sudo tar c -f - -C /var/lib/ceph/2a2e6c42-cf84-11ee-95bb-87774f69a715/crash -- . 2024-02-20T00:22:07.354 DEBUG:teuthology.misc:Transferring archived files from smithi132:/var/lib/ceph/2a2e6c42-cf84-11ee-95bb-87774f69a715/crash to /home/teuthworker/archive/gabrioux-2024-02-19_22:54:55-orch:cephadm-wip-guits-testing-6-2024-02-19-2119-pacific-distro-default-smithi/7568131/remote/ubuntu@smithi132.front.sepia.ceph.com/crash 2024-02-20T00:22:07.355 DEBUG:teuthology.orchestra.run.smithi132:> sudo tar c -f - -C /var/lib/ceph/2a2e6c42-cf84-11ee-95bb-87774f69a715/crash -- . 2024-02-20T00:22:07.392 INFO:teuthology.orchestra.run.smithi132.stderr:tar: /var/lib/ceph/2a2e6c42-cf84-11ee-95bb-87774f69a715/crash: Cannot open: No such file or directory 2024-02-20T00:22:07.392 INFO:teuthology.orchestra.run.smithi132.stderr:tar: Error is not recoverable: exiting now 2024-02-20T00:22:07.393 INFO:tasks.cephadm:Checking cluster log for badness... 2024-02-20T00:22:07.393 DEBUG:teuthology.orchestra.run.smithi084:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/2a2e6c42-cf84-11ee-95bb-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v '\(PG_DEGRADED\)' | egrep -v 'Degraded data redundancy' | egrep -v '\(PG_' | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v 'deprecated feature inline_data' | egrep -v 'compat changed unexpectedly' | egrep -v 'overall HEALTH_' | egrep -v OSD_DOWN | egrep -v OSD_ | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2024-02-20T00:22:07.432 INFO:tasks.cephadm:Compressing logs... 2024-02-20T00:22:07.432 DEBUG:teuthology.orchestra.run.smithi084:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-02-20T00:22:07.474 DEBUG:teuthology.orchestra.run.smithi132:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-02-20T00:22:07.500 INFO:teuthology.orchestra.run.smithi084.stderr:find: ‘/var/log/rbd-target-api’gzip : No such file or directory-5 2024-02-20T00:22:07.500 INFO:teuthology.orchestra.run.smithi084.stderr:--verbose -- /var/log/ceph/cephadm.log 2024-02-20T00:22:07.500 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/2a2e6c42-cf84-11ee-95bb-87774f69a715/ceph-volume.log 2024-02-20T00:22:07.500 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/2a2e6c42-cf84-11ee-95bb-87774f69a715/ceph-mgr.smithi084.bmqrnf.log 2024-02-20T00:22:07.502 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/cephadm.log: gzip /var/log/ceph/2a2e6c42-cf84-11ee-95bb-87774f69a715/ceph-volume.log: -5 --verbose -- /var/log/ceph/2a2e6c42-cf84-11ee-95bb-87774f69a715/ceph.audit.log 2024-02-20T00:22:07.502 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/2a2e6c42-cf84-11ee-95bb-87774f69a715/ceph-mgr.smithi084.bmqrnf.log: gzip -5 --verbose -- /var/log/ceph/2a2e6c42-cf84-11ee-95bb-87774f69a715/ceph.cephadm.log 2024-02-20T00:22:07.502 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/2a2e6c42-cf84-11ee-95bb-87774f69a715/ceph-mon.smithi084.log 2024-02-20T00:22:07.502 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/2a2e6c42-cf84-11ee-95bb-87774f69a715/ceph.audit.log: 88.4% -- replaced with /var/log/ceph/2a2e6c42-cf84-11ee-95bb-87774f69a715/ceph-volume.log.gz 2024-02-20T00:22:07.502 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/2a2e6c42-cf84-11ee-95bb-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/2a2e6c42-cf84-11ee-95bb-87774f69a715/ceph.log 2024-02-20T00:22:07.502 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/2a2e6c42-cf84-11ee-95bb-87774f69a715/ceph-mon.smithi084.log: 89.6% -- replaced with /var/log/ceph/2a2e6c42-cf84-11ee-95bb-87774f69a715/ceph.cephadm.log.gz 2024-02-20T00:22:07.502 INFO:teuthology.orchestra.run.smithi084.stderr: 88.8% -- replaced with /var/log/ceph/2a2e6c42-cf84-11ee-95bb-87774f69a715/ceph.audit.log.gz 2024-02-20T00:22:07.503 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/2a2e6c42-cf84-11ee-95bb-87774f69a715/ceph.log: 88.8% -- replaced with /var/log/ceph/2a2e6c42-cf84-11ee-95bb-87774f69a715/ceph.log.gz 2024-02-20T00:22:07.504 INFO:teuthology.orchestra.run.smithi132.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-02-20T00:22:07.504 INFO:teuthology.orchestra.run.smithi132.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-02-20T00:22:07.505 INFO:teuthology.orchestra.run.smithi132.stderr:gzip -5 --verbose -- /var/log/ceph/2a2e6c42-cf84-11ee-95bb-87774f69a715/ceph-volume.log 2024-02-20T00:22:07.505 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/cephadm.log: /var/log/ceph/2a2e6c42-cf84-11ee-95bb-87774f69a715/ceph-volume.log: 88.4% -- replaced with /var/log/ceph/2a2e6c42-cf84-11ee-95bb-87774f69a715/ceph-volume.log.gz 2024-02-20T00:22:07.506 INFO:teuthology.orchestra.run.smithi084.stderr: 91.8% -- replaced with /var/log/ceph/cephadm.log.gz 2024-02-20T00:22:07.506 INFO:teuthology.orchestra.run.smithi132.stderr: 95.5% -- replaced with /var/log/ceph/cephadm.log.gz 2024-02-20T00:22:07.508 INFO:teuthology.orchestra.run.smithi132.stderr: 2024-02-20T00:22:07.508 INFO:teuthology.orchestra.run.smithi132.stderr:real 0m0.015s 2024-02-20T00:22:07.508 INFO:teuthology.orchestra.run.smithi132.stderr:user 0m0.011s 2024-02-20T00:22:07.508 INFO:teuthology.orchestra.run.smithi132.stderr:sys 0m0.014s 2024-02-20T00:22:07.596 INFO:teuthology.orchestra.run.smithi084.stderr:gzip: /var/log/ceph/2a2e6c42-cf84-11ee-95bb-87774f69a715/ceph-mgr.smithi084.bmqrnf.log: file size changed while zipping 2024-02-20T00:22:07.596 INFO:teuthology.orchestra.run.smithi084.stderr: 92.6% -- replaced with /var/log/ceph/2a2e6c42-cf84-11ee-95bb-87774f69a715/ceph-mgr.smithi084.bmqrnf.log.gz 2024-02-20T00:22:07.669 INFO:teuthology.orchestra.run.smithi084.stderr: 90.6% -- replaced with /var/log/ceph/2a2e6c42-cf84-11ee-95bb-87774f69a715/ceph-mon.smithi084.log.gz 2024-02-20T00:22:07.671 INFO:teuthology.orchestra.run.smithi084.stderr: 2024-02-20T00:22:07.671 INFO:teuthology.orchestra.run.smithi084.stderr:real 0m0.182s 2024-02-20T00:22:07.671 INFO:teuthology.orchestra.run.smithi084.stderr:user 0m0.269s 2024-02-20T00:22:07.671 INFO:teuthology.orchestra.run.smithi084.stderr:sys 0m0.025s 2024-02-20T00:22:07.672 INFO:tasks.cephadm:Archiving logs... 2024-02-20T00:22:07.672 DEBUG:teuthology.misc:Transferring archived files from smithi084:/var/log/ceph to /home/teuthworker/archive/gabrioux-2024-02-19_22:54:55-orch:cephadm-wip-guits-testing-6-2024-02-19-2119-pacific-distro-default-smithi/7568131/remote/ubuntu@smithi084.front.sepia.ceph.com/log 2024-02-20T00:22:07.673 DEBUG:teuthology.orchestra.run.smithi084:> sudo tar c -f - -C /var/log/ceph -- . 2024-02-20T00:22:07.793 DEBUG:teuthology.misc:Transferring archived files from smithi132:/var/log/ceph to /home/teuthworker/archive/gabrioux-2024-02-19_22:54:55-orch:cephadm-wip-guits-testing-6-2024-02-19-2119-pacific-distro-default-smithi/7568131/remote/ubuntu@smithi132.front.sepia.ceph.com/log 2024-02-20T00:22:07.793 DEBUG:teuthology.orchestra.run.smithi132:> sudo tar c -f - -C /var/log/ceph -- . 2024-02-20T00:22:07.827 INFO:tasks.cephadm:Removing cluster... 2024-02-20T00:22:07.828 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 --force 2024-02-20T00:22:10.975 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 2a2e6c42-cf84-11ee-95bb-87774f69a715 --force 2024-02-20T00:25:04.227 INFO:tasks.cephadm:Removing cephadm ... 2024-02-20T00:25:04.227 DEBUG:teuthology.orchestra.run.smithi084:> rm -rf /home/ubuntu/cephtest/cephadm 2024-02-20T00:25:04.252 DEBUG:teuthology.orchestra.run.smithi132:> rm -rf /home/ubuntu/cephtest/cephadm 2024-02-20T00:25:04.270 INFO:tasks.cephadm:Teardown complete 2024-02-20T00:25:04.271 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_19854089e18d4f65dda2b6cd74e737367c2514bd/qa/tasks/cephadm.py", line 512, in ceph_bootstrap yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_19854089e18d4f65dda2b6cd74e737367c2514bd/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_19854089e18d4f65dda2b6cd74e737367c2514bd/qa/tasks/cephadm.py", line 1463, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_19854089e18d4f65dda2b6cd74e737367c2514bd/qa/tasks/cephadm.py", line 1376, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_19854089e18d4f65dda2b6cd74e737367c2514bd/qa/tasks/cephadm.py", line 531, in ceph_bootstrap ctx.daemons.get_daemon(type_, id_, cluster).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/daemon/cephadmunit.py", line 154, in stop self.remote.sh(self.stop_cmd) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 97, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi132 with status 5: 'sudo systemctl stop ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715@mon.smithi132' 2024-02-20T00:25:04.505 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=00265e73e8aa4bccbb4e2406eac3f378 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_19854089e18d4f65dda2b6cd74e737367c2514bd/qa/tasks/cephadm.py", line 512, in ceph_bootstrap yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_19854089e18d4f65dda2b6cd74e737367c2514bd/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_19854089e18d4f65dda2b6cd74e737367c2514bd/qa/tasks/cephadm.py", line 1463, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_19854089e18d4f65dda2b6cd74e737367c2514bd/qa/tasks/cephadm.py", line 1376, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_19854089e18d4f65dda2b6cd74e737367c2514bd/qa/tasks/cephadm.py", line 531, in ceph_bootstrap ctx.daemons.get_daemon(type_, id_, cluster).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/daemon/cephadmunit.py", line 154, in stop self.remote.sh(self.stop_cmd) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 97, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi132 with status 5: 'sudo systemctl stop ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715@mon.smithi132' 2024-02-20T00:25:04.510 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-02-20T00:25:04.520 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-02-20T00:25:04.534 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-02-20T00:25:04.534 DEBUG:teuthology.orchestra.run.smithi084:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-02-20T00:25:04.559 DEBUG:teuthology.orchestra.run.smithi132:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-02-20T00:25:04.637 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-02-20T00:25:04.646 INFO:teuthology.task.clock:Checking final clock skew... 2024-02-20T00:25:04.646 DEBUG:teuthology.orchestra.run.smithi084:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-20T00:25:04.649 DEBUG:teuthology.orchestra.run.smithi132:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-20T00:25:04.665 INFO:teuthology.orchestra.run.smithi084.stderr:bash: ntpq: command not found 2024-02-20T00:25:04.667 INFO:teuthology.orchestra.run.smithi132.stderr:bash: ntpq: command not found 2024-02-20T00:25:04.670 INFO:teuthology.orchestra.run.smithi084.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-20T00:25:04.670 INFO:teuthology.orchestra.run.smithi084.stdout:=============================================================================== 2024-02-20T00:25:04.670 INFO:teuthology.orchestra.run.smithi084.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 32 +863us[ +864us] +/- 99ms 2024-02-20T00:25:04.670 INFO:teuthology.orchestra.run.smithi084.stdout:^* hv02.front.sepia.ceph.com 3 6 377 28 -901us[ -900us] +/- 50ms 2024-02-20T00:25:04.671 INFO:teuthology.orchestra.run.smithi084.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 29 +715us[ +715us] +/- 86ms 2024-02-20T00:25:04.671 INFO:teuthology.orchestra.run.smithi084.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-02-20T00:25:04.673 INFO:teuthology.orchestra.run.smithi132.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-20T00:25:04.673 INFO:teuthology.orchestra.run.smithi132.stdout:=============================================================================== 2024-02-20T00:25:04.674 INFO:teuthology.orchestra.run.smithi132.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 31 +854us[ +854us] +/- 99ms 2024-02-20T00:25:04.674 INFO:teuthology.orchestra.run.smithi132.stdout:^* hv02.front.sepia.ceph.com 3 6 377 32 -898us[ -909us] +/- 50ms 2024-02-20T00:25:04.674 INFO:teuthology.orchestra.run.smithi132.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 31 +716us[ +716us] +/- 86ms 2024-02-20T00:25:04.674 INFO:teuthology.orchestra.run.smithi132.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-02-20T00:25:04.674 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-02-20T00:25:04.683 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-02-20T00:25:04.684 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-02-20T00:25:04.693 DEBUG:teuthology.orchestra.run.smithi084:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-02-20T00:25:04.778 DEBUG:teuthology.orchestra.run.smithi132:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-02-20T00:25:04.845 DEBUG:teuthology.orchestra.run.smithi084:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-20T00:25:04.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-20T00:25:04.875 DEBUG:teuthology.orchestra.run.smithi132:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-20T00:25:04.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-20T00:25:04.906 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-02-20T00:25:04.916 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-02-20T00:25:04.925 INFO:teuthology.task.internal:Duration was 1549.626277 seconds 2024-02-20T00:25:04.925 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-02-20T00:25:04.935 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-02-20T00:25:04.935 DEBUG:teuthology.orchestra.run.smithi084:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-02-20T00:25:04.938 DEBUG:teuthology.orchestra.run.smithi132:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-02-20T00:25:04.982 INFO:teuthology.orchestra.run.smithi084.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-20T00:25:04.993 INFO:teuthology.orchestra.run.smithi132.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-20T00:25:05.371 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-02-20T00:25:05.371 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi084.front.sepia.ceph.com 2024-02-20T00:25:05.371 DEBUG:teuthology.orchestra.run.smithi084:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-02-20T00:25:05.395 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi132.front.sepia.ceph.com 2024-02-20T00:25:05.395 DEBUG:teuthology.orchestra.run.smithi132:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-02-20T00:25:05.418 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-02-20T00:25:05.419 DEBUG:teuthology.orchestra.run.smithi084:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-02-20T00:25:05.438 DEBUG:teuthology.orchestra.run.smithi132:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-02-20T00:25:05.510 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-02-20T00:25:05.511 DEBUG:teuthology.orchestra.run.smithi084:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-02-20T00:25:05.514 DEBUG:teuthology.orchestra.run.smithi132:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-02-20T00:25:05.627 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-02-20T00:25:05.644 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-02-20T00:25:05.645 DEBUG:teuthology.orchestra.run.smithi084:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-02-20T00:25:05.673 DEBUG:teuthology.orchestra.run.smithi132:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-02-20T00:25:05.701 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-02-20T00:25:05.718 DEBUG:teuthology.orchestra.run.smithi084:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-02-20T00:25:05.721 DEBUG:teuthology.orchestra.run.smithi132:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-02-20T00:25:05.744 INFO:teuthology.orchestra.run.smithi084.stdout:kernel.core_pattern = core 2024-02-20T00:25:05.767 INFO:teuthology.orchestra.run.smithi132.stdout:kernel.core_pattern = core 2024-02-20T00:25:05.794 DEBUG:teuthology.orchestra.run.smithi084:> test -e /home/ubuntu/cephtest/archive/coredump 2024-02-20T00:25:05.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-20T00:25:05.829 DEBUG:teuthology.orchestra.run.smithi132:> test -e /home/ubuntu/cephtest/archive/coredump 2024-02-20T00:25:05.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-20T00:25:05.851 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-02-20T00:25:05.861 INFO:teuthology.task.internal:Transferring archived files... 2024-02-20T00:25:05.862 DEBUG:teuthology.misc:Transferring archived files from smithi084:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/gabrioux-2024-02-19_22:54:55-orch:cephadm-wip-guits-testing-6-2024-02-19-2119-pacific-distro-default-smithi/7568131/remote/smithi084 2024-02-20T00:25:05.863 DEBUG:teuthology.orchestra.run.smithi084:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-02-20T00:25:05.908 DEBUG:teuthology.misc:Transferring archived files from smithi132:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/gabrioux-2024-02-19_22:54:55-orch:cephadm-wip-guits-testing-6-2024-02-19-2119-pacific-distro-default-smithi/7568131/remote/smithi132 2024-02-20T00:25:05.909 DEBUG:teuthology.orchestra.run.smithi132:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-02-20T00:25:05.949 INFO:teuthology.task.internal:Removing archive directory... 2024-02-20T00:25:05.949 DEBUG:teuthology.orchestra.run.smithi084:> rm -rf -- /home/ubuntu/cephtest/archive 2024-02-20T00:25:05.951 DEBUG:teuthology.orchestra.run.smithi132:> rm -rf -- /home/ubuntu/cephtest/archive 2024-02-20T00:25:06.001 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-02-20T00:25:06.013 INFO:teuthology.task.internal:Not uploading archives. 2024-02-20T00:25:06.014 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-02-20T00:25:06.023 INFO:teuthology.task.internal:Tidying up after the test... 2024-02-20T00:25:06.024 DEBUG:teuthology.orchestra.run.smithi084:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-02-20T00:25:06.026 DEBUG:teuthology.orchestra.run.smithi132:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-02-20T00:25:06.040 INFO:teuthology.orchestra.run.smithi084.stdout: 525357 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 20 00:25 /home/ubuntu/cephtest 2024-02-20T00:25:06.058 INFO:teuthology.orchestra.run.smithi132.stdout: 525357 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 20 00:25 /home/ubuntu/cephtest 2024-02-20T00:25:06.060 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-02-20T00:25:06.070 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-02-20T00:25:06.167 INFO:teuthology.nuke:Checking targets against current locks 2024-02-20T00:25:06.195 DEBUG:teuthology.nuke:shortname: smithi084 2024-02-20T00:25:06.196 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-20T00:25:06.223 DEBUG:teuthology.nuke:shortname: smithi132 2024-02-20T00:25:06.223 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-20T00:25:06.228 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi084.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/gabrioux-2024-02-19_22:54:55-orch:cephadm-wip-guits-testing-6-2024-02-19-2119-pacific-distro-default-smithi/7568131', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-19 23:50:31.397255', 'locked_by': 'scheduled_gabrioux@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPNDTO6YWTW+/cXAw1cHDFMHP2Z6Mck+lVm3Zr/19y56KcjLXZcZTPFPbsbMmRH2/7ACg4UxKAjEljm75Wq2Tv4='} 2024-02-20T00:25:06.255 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi132.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/gabrioux-2024-02-19_22:54:55-orch:cephadm-wip-guits-testing-6-2024-02-19-2119-pacific-distro-default-smithi/7568131', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-19 23:50:31.398722', 'locked_by': 'scheduled_gabrioux@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKO7VWT3OFStyyg3pKrCB0HbjeNRE/yRBAu8QNQYTwU8KFhSKWl9zsWsMZ8QipkP2tfE8vtTV0BeqyBYoaWcnfY='} 2024-02-20T00:25:06.282 INFO:teuthology.orchestra.console.smithi084:Power off 2024-02-20T00:25:06.282 DEBUG:teuthology.orchestra.console.smithi084:pexpect command: ipmitool -H smithi084.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-20T00:25:06.307 DEBUG:teuthology.orchestra.console.smithi084:power off output: Chassis Power Control: Down/Off 2024-02-20T00:25:06.307 DEBUG:teuthology.orchestra.console.smithi084:pexpect command: ipmitool -H smithi084.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-20T00:25:06.325 INFO:teuthology.orchestra.console.smithi132:Power off 2024-02-20T00:25:06.325 DEBUG:teuthology.orchestra.console.smithi132:pexpect command: ipmitool -H smithi132.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-20T00:25:06.334 DEBUG:teuthology.orchestra.console.smithi084:check power output: Chassis Power is on 2024-02-20T00:25:06.349 DEBUG:teuthology.orchestra.console.smithi132:power off output: Chassis Power Control: Down/Off 2024-02-20T00:25:06.349 DEBUG:teuthology.orchestra.console.smithi132:pexpect command: ipmitool -H smithi132.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-20T00:25:06.371 DEBUG:teuthology.orchestra.console.smithi132:check power output: Chassis Power is on 2024-02-20T00:25:10.335 DEBUG:teuthology.orchestra.console.smithi084:pexpect command: ipmitool -H smithi084.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-20T00:25:10.372 DEBUG:teuthology.orchestra.console.smithi132:pexpect command: ipmitool -H smithi132.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-20T00:25:10.446 DEBUG:teuthology.orchestra.console.smithi084:check power output: Chassis Power is on 2024-02-20T00:25:10.482 DEBUG:teuthology.orchestra.console.smithi132:check power output: Chassis Power is on 2024-02-20T00:25:14.448 DEBUG:teuthology.orchestra.console.smithi084:pexpect command: ipmitool -H smithi084.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-20T00:25:14.483 DEBUG:teuthology.orchestra.console.smithi132:pexpect command: ipmitool -H smithi132.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-20T00:25:14.558 DEBUG:teuthology.orchestra.console.smithi084:check power output: Chassis Power is off 2024-02-20T00:25:14.594 DEBUG:teuthology.orchestra.console.smithi132:check power output: Chassis Power is on 2024-02-20T00:25:14.660 INFO:teuthology.orchestra.console.smithi084:Power off completed 2024-02-20T00:25:18.596 DEBUG:teuthology.orchestra.console.smithi132:pexpect command: ipmitool -H smithi132.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-20T00:25:18.706 DEBUG:teuthology.orchestra.console.smithi132:check power output: Chassis Power is off 2024-02-20T00:25:18.806 INFO:teuthology.orchestra.console.smithi132:Power off completed 2024-02-20T00:25:18.912 INFO:teuthology.run:Summary data: description: orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} duration: 1549.6262769699097 failure_reason: 'Command failed on smithi132 with status 5: ''sudo systemctl stop ceph-2a2e6c42-cf84-11ee-95bb-87774f69a715@mon.smithi132''' flavor: default owner: scheduled_gabrioux@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=00265e73e8aa4bccbb4e2406eac3f378 status: fail success: false 2024-02-20T00:25:18.912 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-02-20T00:25:18.999 INFO:teuthology.run:FAIL