2024-02-19T21:42:09.516 INFO:root:teuthology version: 0.0.1.dev260+gb1dac55 2024-02-19T21:42:09.516 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --owner scheduled_yuriw@teuthology --description fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} --name yuriw-2024-02-19_19:27:49-fs-pacific-release-distro-default-smithi --archive /home/teuthworker/archive/yuriw-2024-02-19_19:27:49-fs-pacific-release-distro-default-smithi/7566790 -- /home/teuthworker/archive/yuriw-2024-02-19_19:27:49-fs-pacific-release-distro-default-smithi/7566790/orig.config.yaml 2024-02-19T21:42:09.567 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-02-19T21:42:09.657 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2024-02-19_19:27:49-fs-pacific-release-distro-default-smithi/7566790 branch: pacific-release description: fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} email: null first_in_suite: false job_id: '7566790' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi meta: - desc: 'setup ceph/pacific v16.2.4 ' name: yuriw-2024-02-19_19:27:49-fs-pacific-release-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 8.stream overrides: admin_socket: branch: pacific-release ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: mon pg warn min per osd: 0 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - \(PG_DEGRADED\) - Degraded data redundancy - \(PG_ - MDS_INSUFFICIENT_STANDBY - deprecated feature inline_data - compat changed unexpectedly - overall HEALTH_ - OSD_DOWN - OSD_ - but it is still running - is not responding sha1: 81bd20d634209c7cb82c18be12b4b5a05643ebf1 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 81bd20d634209c7cb82c18be12b4b5a05643ebf1 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: pacific-release sha1: 81bd20d634209c7cb82c18be12b4b5a05643ebf1 owner: scheduled_yuriw@teuthology priority: 71 repo: https://github.com/ceph/ceph-ci.git roles: - - host.a - client.0 - osd.0 - osd.1 - osd.2 - - host.b - client.1 - osd.3 - osd.4 - osd.5 seed: 9538 sha1: 81bd20d634209c7cb82c18be12b4b5a05643ebf1 sleep_before_teardown: 0 subset: 111/120000 suite: fs suite_branch: pacific-release suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_81bd20d634209c7cb82c18be12b4b5a05643ebf1/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 81bd20d634209c7cb82c18be12b4b5a05643ebf1 targets: smithi053.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOZmqTcx8AszQ0DRj1FdK+b1wOzQvjOnoRF1Cyd0hy4AkDcvdcGOVx59SKOTgOfBKymcBUFepHbZAEQBEnwCto8= smithi086.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLZ0e+5X3hmPqCyniFmfDHi38amlVmWCZs9h7B6MrbloeJ3ONBlr9vwa9fKYCNgwke229OMcwn/jF/1/u5+91Bc= tasks: - pexec: all: - sudo cp /etc/containers/registries.conf /etc/containers/registries.conf.backup - sudo dnf -y module reset container-tools - sudo dnf -y module install container-tools --allowerasing --nobest - sudo cp /etc/containers/registries.conf.backup /etc/containers/registries.conf - pexec: clients: - sudo modprobe -r ceph - sudo modprobe ceph disable_send_metrics=on - install: exclude_packages: - ceph-volume tag: v16.2.4 - print: '**** done install task...' - cephadm: avoid_pacific_features: true cephadm_branch: v16.2.4 cephadm_git_url: https://github.com/ceph/ceph image: docker.io/ceph/ceph:v16.2.4 roleless: true - print: '**** done starting v16.2.4' - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: host.a: - ceph fs volume create cephfs --placement=4 - ceph fs dump - cephadm.shell: host.a: - ceph fs set cephfs max_mds 1 - cephadm.shell: host.a: - ceph fs set cephfs allow_standby_replay false - cephadm.shell: host.a: - ceph fs set cephfs inline_data false - cephadm.shell: host.a: - ceph fs dump - ceph --format=json fs dump | jq -e ".filesystems | length == 1" - while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done - fs.pre_upgrade_save: null - kclient: null - print: '**** done client' - parallel: - upgrade-tasks - workload-tasks - cephadm.shell: host.a: - ceph fs dump - fs.post_upgrade_checks: null teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: b1dac5519c57c269ea98a22fb7729016a1d0e4d2 timestamp: 2024-02-19_19:27:49 tube: smithi upgrade-tasks: sequential: - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 host.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph fs dump; ceph orch upgrade status ; sleep 30 ; done - ceph orch ps - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 workload-tasks: sequential: - workunit: clients: all: - suites/fsstress.sh 2024-02-19T21:42:09.658 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_81bd20d634209c7cb82c18be12b4b5a05643ebf1/qa; will attempt to use it 2024-02-19T21:42:09.658 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_81bd20d634209c7cb82c18be12b4b5a05643ebf1/qa/tasks 2024-02-19T21:42:09.658 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-02-19T21:42:09.660 INFO:teuthology.task.internal:Checking packages... 2024-02-19T21:42:09.684 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '81bd20d634209c7cb82c18be12b4b5a05643ebf1' 2024-02-19T21:42:09.684 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-02-19T21:42:09.684 INFO:teuthology.packaging:ref: None 2024-02-19T21:42:09.684 INFO:teuthology.packaging:tag: None 2024-02-19T21:42:09.684 INFO:teuthology.packaging:branch: pacific-release 2024-02-19T21:42:09.684 INFO:teuthology.packaging:sha1: 81bd20d634209c7cb82c18be12b4b5a05643ebf1 2024-02-19T21:42:09.684 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=pacific-release 2024-02-19T21:42:09.830 INFO:teuthology.task.internal:Found packages for ceph version 16.2.14-548.g81bd20d6 2024-02-19T21:42:09.831 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-02-19T21:42:09.838 INFO:teuthology.task.internal:no buildpackages task found 2024-02-19T21:42:09.838 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-02-19T21:42:09.859 INFO:teuthology.task.internal:Saving configuration 2024-02-19T21:42:09.879 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-02-19T21:42:09.884 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-19T21:42:09.912 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi053.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-19_19:27:49-fs-pacific-release-distro-default-smithi/7566790', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-19 21:36:58.616045', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOZmqTcx8AszQ0DRj1FdK+b1wOzQvjOnoRF1Cyd0hy4AkDcvdcGOVx59SKOTgOfBKymcBUFepHbZAEQBEnwCto8='} 2024-02-19T21:42:09.937 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi086.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-19_19:27:49-fs-pacific-release-distro-default-smithi/7566790', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-19 21:36:58.617595', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLZ0e+5X3hmPqCyniFmfDHi38amlVmWCZs9h7B6MrbloeJ3ONBlr9vwa9fKYCNgwke229OMcwn/jF/1/u5+91Bc='} 2024-02-19T21:42:09.937 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-02-19T21:42:09.943 INFO:teuthology.task.internal:roles: ubuntu@smithi053.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2'] 2024-02-19T21:42:09.943 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5'] 2024-02-19T21:42:09.943 INFO:teuthology.run_tasks:Running task console_log... 2024-02-19T21:42:10.020 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f0f47b53f70>, signals=[15]) 2024-02-19T21:42:10.020 INFO:teuthology.run_tasks:Running task internal.connect... 2024-02-19T21:42:10.026 INFO:teuthology.task.internal:Opening connections... 2024-02-19T21:42:10.026 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi053.front.sepia.ceph.com 2024-02-19T21:42:10.028 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi053.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T21:42:10.104 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi086.front.sepia.ceph.com 2024-02-19T21:42:10.105 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T21:42:10.221 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-02-19T21:42:10.228 DEBUG:teuthology.orchestra.run.smithi053:> uname -m 2024-02-19T21:42:10.251 INFO:teuthology.orchestra.run.smithi053.stdout:x86_64 2024-02-19T21:42:10.251 DEBUG:teuthology.orchestra.run.smithi053:> cat /etc/os-release 2024-02-19T21:42:10.313 INFO:teuthology.orchestra.run.smithi053.stdout:NAME="CentOS Stream" 2024-02-19T21:42:10.313 INFO:teuthology.orchestra.run.smithi053.stdout:VERSION="8" 2024-02-19T21:42:10.314 INFO:teuthology.orchestra.run.smithi053.stdout:ID="centos" 2024-02-19T21:42:10.314 INFO:teuthology.orchestra.run.smithi053.stdout:ID_LIKE="rhel fedora" 2024-02-19T21:42:10.314 INFO:teuthology.orchestra.run.smithi053.stdout:VERSION_ID="8" 2024-02-19T21:42:10.314 INFO:teuthology.orchestra.run.smithi053.stdout:PLATFORM_ID="platform:el8" 2024-02-19T21:42:10.314 INFO:teuthology.orchestra.run.smithi053.stdout:PRETTY_NAME="CentOS Stream 8" 2024-02-19T21:42:10.314 INFO:teuthology.orchestra.run.smithi053.stdout:ANSI_COLOR="0;31" 2024-02-19T21:42:10.314 INFO:teuthology.orchestra.run.smithi053.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-02-19T21:42:10.314 INFO:teuthology.orchestra.run.smithi053.stdout:HOME_URL="https://centos.org/" 2024-02-19T21:42:10.314 INFO:teuthology.orchestra.run.smithi053.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-02-19T21:42:10.314 INFO:teuthology.orchestra.run.smithi053.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-02-19T21:42:10.314 INFO:teuthology.orchestra.run.smithi053.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-02-19T21:42:10.315 INFO:teuthology.lock.ops:Updating smithi053.front.sepia.ceph.com on lock server 2024-02-19T21:42:10.344 DEBUG:teuthology.orchestra.run.smithi086:> uname -m 2024-02-19T21:42:10.367 INFO:teuthology.orchestra.run.smithi086.stdout:x86_64 2024-02-19T21:42:10.367 DEBUG:teuthology.orchestra.run.smithi086:> cat /etc/os-release 2024-02-19T21:42:10.429 INFO:teuthology.orchestra.run.smithi086.stdout:NAME="CentOS Stream" 2024-02-19T21:42:10.429 INFO:teuthology.orchestra.run.smithi086.stdout:VERSION="8" 2024-02-19T21:42:10.429 INFO:teuthology.orchestra.run.smithi086.stdout:ID="centos" 2024-02-19T21:42:10.429 INFO:teuthology.orchestra.run.smithi086.stdout:ID_LIKE="rhel fedora" 2024-02-19T21:42:10.429 INFO:teuthology.orchestra.run.smithi086.stdout:VERSION_ID="8" 2024-02-19T21:42:10.430 INFO:teuthology.orchestra.run.smithi086.stdout:PLATFORM_ID="platform:el8" 2024-02-19T21:42:10.430 INFO:teuthology.orchestra.run.smithi086.stdout:PRETTY_NAME="CentOS Stream 8" 2024-02-19T21:42:10.430 INFO:teuthology.orchestra.run.smithi086.stdout:ANSI_COLOR="0;31" 2024-02-19T21:42:10.430 INFO:teuthology.orchestra.run.smithi086.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-02-19T21:42:10.430 INFO:teuthology.orchestra.run.smithi086.stdout:HOME_URL="https://centos.org/" 2024-02-19T21:42:10.430 INFO:teuthology.orchestra.run.smithi086.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-02-19T21:42:10.430 INFO:teuthology.orchestra.run.smithi086.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-02-19T21:42:10.430 INFO:teuthology.orchestra.run.smithi086.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-02-19T21:42:10.430 INFO:teuthology.lock.ops:Updating smithi086.front.sepia.ceph.com on lock server 2024-02-19T21:42:10.452 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-02-19T21:42:10.461 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-02-19T21:42:10.466 INFO:teuthology.task.internal:Checking for old test directory... 2024-02-19T21:42:10.467 DEBUG:teuthology.orchestra.run.smithi053:> test '!' -e /home/ubuntu/cephtest 2024-02-19T21:42:10.469 DEBUG:teuthology.orchestra.run.smithi086:> test '!' -e /home/ubuntu/cephtest 2024-02-19T21:42:10.489 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-02-19T21:42:10.495 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-02-19T21:42:10.495 DEBUG:teuthology.orchestra.run.smithi053:> test -z $(ls -A /var/lib/ceph) 2024-02-19T21:42:10.532 DEBUG:teuthology.orchestra.run.smithi086:> test -z $(ls -A /var/lib/ceph) 2024-02-19T21:42:10.556 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-02-19T21:42:10.613 INFO:teuthology.run_tasks:Running task kernel... 2024-02-19T21:42:10.627 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-02-19T21:42:10.628 INFO:teuthology.task.kernel:config {'host.a': {'kdb': True, 'sha1': 'distro'}, 'host.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-02-19T21:42:10.628 DEBUG:teuthology.orchestra.run.smithi053:> test -f /run/.containerenv -o -f /.dockerenv 2024-02-19T21:42:10.629 DEBUG:teuthology.orchestra.run.smithi086:> test -f /run/.containerenv -o -f /.dockerenv 2024-02-19T21:42:10.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-19T21:42:10.648 DEBUG:teuthology.orchestra.run.smithi086:> uname -r 2024-02-19T21:42:10.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-19T21:42:10.649 DEBUG:teuthology.orchestra.run.smithi053:> uname -r 2024-02-19T21:42:10.707 INFO:teuthology.orchestra.run.smithi086.stdout:4.18.0-489.el8.x86_64 2024-02-19T21:42:10.707 INFO:teuthology.task.kernel:Running kernel on smithi086: 4.18.0-489.el8.x86_64 2024-02-19T21:42:10.708 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum install -y kernel 2024-02-19T21:42:10.711 INFO:teuthology.orchestra.run.smithi053.stdout:4.18.0-489.el8.x86_64 2024-02-19T21:42:10.711 INFO:teuthology.task.kernel:Running kernel on smithi053: 4.18.0-489.el8.x86_64 2024-02-19T21:42:10.711 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum install -y kernel 2024-02-19T21:42:22.846 INFO:teuthology.orchestra.run.smithi053.stdout:CentOS Stream 8 - AppStream 2.8 MB/s | 27 MB 00:09 2024-02-19T21:42:25.023 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS Stream 8 - AppStream 2.3 MB/s | 27 MB 00:11 2024-02-19T21:42:28.354 INFO:teuthology.orchestra.run.smithi053.stdout:CentOS Stream 8 - BaseOS 26 MB/s | 10 MB 00:00 2024-02-19T21:42:30.188 INFO:teuthology.orchestra.run.smithi053.stdout:CentOS Stream 8 - Extras 140 kB/s | 18 kB 00:00 2024-02-19T21:42:30.440 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS Stream 8 - BaseOS 18 MB/s | 10 MB 00:00 2024-02-19T21:42:30.447 INFO:teuthology.orchestra.run.smithi053.stdout:CentOS Stream 8 - Extras common packages 49 kB/s | 6.9 kB 00:00 2024-02-19T21:42:30.674 INFO:teuthology.orchestra.run.smithi053.stdout:Copr repo for python3-asyncssh owned by ceph 27 kB/s | 3.5 kB 00:00 2024-02-19T21:42:32.200 INFO:teuthology.orchestra.run.smithi053.stdout:Extra Packages for Enterprise Linux 11 MB/s | 16 MB 00:01 2024-02-19T21:42:32.332 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS Stream 8 - Extras 66 kB/s | 18 kB 00:00 2024-02-19T21:42:32.716 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS Stream 8 - Extras common packages 25 kB/s | 6.9 kB 00:00 2024-02-19T21:42:32.934 INFO:teuthology.orchestra.run.smithi086.stdout:Copr repo for python3-asyncssh owned by ceph 27 kB/s | 3.5 kB 00:00 2024-02-19T21:42:33.812 INFO:teuthology.orchestra.run.smithi086.stdout:Extra Packages for Enterprise Linux 21 MB/s | 16 MB 00:00 2024-02-19T21:42:36.233 INFO:teuthology.orchestra.run.smithi053.stdout:lab-extras 523 kB/s | 24 kB 00:00 2024-02-19T21:42:37.518 INFO:teuthology.orchestra.run.smithi086.stdout:lab-extras 437 kB/s | 24 kB 00:00 2024-02-19T21:42:39.346 INFO:teuthology.orchestra.run.smithi053.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-02-19T21:42:39.347 INFO:teuthology.orchestra.run.smithi053.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-02-19T21:42:39.453 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-02-19T21:42:39.454 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-02-19T21:42:39.454 INFO:teuthology.orchestra.run.smithi053.stdout: Package Architecture Version Repository Size 2024-02-19T21:42:39.454 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-02-19T21:42:39.454 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2024-02-19T21:42:39.454 INFO:teuthology.orchestra.run.smithi053.stdout: kernel x86_64 4.18.0-539.el8 baseos 10 M 2024-02-19T21:42:39.454 INFO:teuthology.orchestra.run.smithi053.stdout:Installing dependencies: 2024-02-19T21:42:39.454 INFO:teuthology.orchestra.run.smithi053.stdout: kernel-core x86_64 4.18.0-539.el8 baseos 43 M 2024-02-19T21:42:39.454 INFO:teuthology.orchestra.run.smithi053.stdout: kernel-modules x86_64 4.18.0-539.el8 baseos 36 M 2024-02-19T21:42:39.454 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-02-19T21:42:39.454 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2024-02-19T21:42:39.454 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-02-19T21:42:39.454 INFO:teuthology.orchestra.run.smithi053.stdout:Install 3 Packages 2024-02-19T21:42:39.455 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-02-19T21:42:39.455 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 90 M 2024-02-19T21:42:39.455 INFO:teuthology.orchestra.run.smithi053.stdout:Installed size: 97 M 2024-02-19T21:42:39.455 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2024-02-19T21:42:40.457 INFO:teuthology.orchestra.run.smithi086.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-02-19T21:42:40.457 INFO:teuthology.orchestra.run.smithi086.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-02-19T21:42:40.566 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-02-19T21:42:40.567 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-02-19T21:42:40.567 INFO:teuthology.orchestra.run.smithi086.stdout: Package Architecture Version Repository Size 2024-02-19T21:42:40.567 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-02-19T21:42:40.567 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-02-19T21:42:40.567 INFO:teuthology.orchestra.run.smithi086.stdout: kernel x86_64 4.18.0-539.el8 baseos 10 M 2024-02-19T21:42:40.567 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2024-02-19T21:42:40.567 INFO:teuthology.orchestra.run.smithi086.stdout: kernel-core x86_64 4.18.0-539.el8 baseos 43 M 2024-02-19T21:42:40.567 INFO:teuthology.orchestra.run.smithi086.stdout: kernel-modules x86_64 4.18.0-539.el8 baseos 36 M 2024-02-19T21:42:40.567 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T21:42:40.568 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-02-19T21:42:40.568 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-02-19T21:42:40.568 INFO:teuthology.orchestra.run.smithi086.stdout:Install 3 Packages 2024-02-19T21:42:40.568 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T21:42:40.568 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 90 M 2024-02-19T21:42:40.568 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 97 M 2024-02-19T21:42:40.568 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-02-19T21:42:41.026 INFO:teuthology.orchestra.run.smithi053.stdout:(1/3): kernel-4.18.0-539.el8.x86_64.rpm 7.6 MB/s | 10 MB 00:01 2024-02-19T21:42:42.078 INFO:teuthology.orchestra.run.smithi086.stdout:(1/3): kernel-4.18.0-539.el8.x86_64.rpm 7.2 MB/s | 10 MB 00:01 2024-02-19T21:42:42.859 INFO:teuthology.orchestra.run.smithi053.stdout:(2/3): kernel-core-4.18.0-539.el8.x86_64.rpm 14 MB/s | 43 MB 00:03 2024-02-19T21:42:43.176 INFO:teuthology.orchestra.run.smithi053.stdout:(3/3): kernel-modules-4.18.0-539.el8.x86_64.rpm 10 MB/s | 36 MB 00:03 2024-02-19T21:42:43.177 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2024-02-19T21:42:43.177 INFO:teuthology.orchestra.run.smithi053.stdout:Total 24 MB/s | 90 MB 00:03 2024-02-19T21:42:43.387 INFO:teuthology.orchestra.run.smithi086.stdout:(2/3): kernel-modules-4.18.0-539.el8.x86_64.rpm 13 MB/s | 36 MB 00:02 2024-02-19T21:42:43.779 INFO:teuthology.orchestra.run.smithi086.stdout:(3/3): kernel-core-4.18.0-539.el8.x86_64.rpm 14 MB/s | 43 MB 00:03 2024-02-19T21:42:43.779 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-02-19T21:42:43.779 INFO:teuthology.orchestra.run.smithi086.stdout:Total 28 MB/s | 90 MB 00:03 2024-02-19T21:42:43.887 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2024-02-19T21:42:44.340 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2024-02-19T21:42:44.340 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2024-02-19T21:42:44.467 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-02-19T21:42:44.892 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-02-19T21:42:44.892 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-02-19T21:42:45.654 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2024-02-19T21:42:45.673 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2024-02-19T21:42:46.155 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-02-19T21:42:46.173 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-02-19T21:42:48.614 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2024-02-19T21:42:48.960 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-02-19T21:42:49.114 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : kernel-core-4.18.0-539.el8.x86_64 1/3 2024-02-19T21:42:49.516 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : kernel-core-4.18.0-539.el8.x86_64 1/3 2024-02-19T21:42:52.829 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: kernel-core-4.18.0-539.el8.x86_64 1/3 2024-02-19T21:42:53.074 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: kernel-core-4.18.0-539.el8.x86_64 1/3 2024-02-19T21:42:53.214 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : kernel-modules-4.18.0-539.el8.x86_64 2/3 2024-02-19T21:42:53.457 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : kernel-modules-4.18.0-539.el8.x86_64 2/3 2024-02-19T21:42:56.889 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: kernel-modules-4.18.0-539.el8.x86_64 2/3 2024-02-19T21:42:56.896 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: kernel-modules-4.18.0-539.el8.x86_64 2/3 2024-02-19T21:42:57.181 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : kernel-4.18.0-539.el8.x86_64 3/3 2024-02-19T21:42:57.192 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : kernel-4.18.0-539.el8.x86_64 3/3 2024-02-19T21:43:19.126 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: kernel-core-4.18.0-539.el8.x86_64 3/3 2024-02-19T21:43:19.150 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: kernel-modules-4.18.0-539.el8.x86_64 3/3 2024-02-19T21:43:19.510 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: kernel-4.18.0-539.el8.x86_64 3/3 2024-02-19T21:43:19.510 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : kernel-4.18.0-539.el8.x86_64 1/3 2024-02-19T21:43:19.510 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : kernel-core-4.18.0-539.el8.x86_64 2/3 2024-02-19T21:43:20.029 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : kernel-modules-4.18.0-539.el8.x86_64 3/3 2024-02-19T21:43:20.030 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T21:43:20.030 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-02-19T21:43:20.030 INFO:teuthology.orchestra.run.smithi086.stdout: kernel-4.18.0-539.el8.x86_64 kernel-core-4.18.0-539.el8.x86_64 2024-02-19T21:43:20.030 INFO:teuthology.orchestra.run.smithi086.stdout: kernel-modules-4.18.0-539.el8.x86_64 2024-02-19T21:43:20.030 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T21:43:20.030 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-02-19T21:43:20.170 DEBUG:teuthology.orchestra.run.smithi086:> rpm -q kernel | sort -rV | head -n 1 2024-02-19T21:43:20.435 INFO:teuthology.orchestra.run.smithi086.stdout:kernel-4.18.0-539.el8.x86_64 2024-02-19T21:43:20.436 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-539.el8.x86_64 2024-02-19T21:43:20.436 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-539.el8.x86_64 2024-02-19T21:43:20.436 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-02-19T21:43:20.436 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-02-19T21:43:20.436 INFO:teuthology.task.kernel:Installing distro kernel on host.b... 2024-02-19T21:43:20.436 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi086.front.sepia.ceph.com, path=None, version=distro) 2024-02-19T21:43:20.437 DEBUG:teuthology.orchestra.run.smithi086:> rpm -q kernel | sort -rV | head -n 1 2024-02-19T21:43:20.647 INFO:teuthology.orchestra.run.smithi086.stdout:kernel-4.18.0-539.el8.x86_64 2024-02-19T21:43:20.647 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-539.el8.x86_64 2024-02-19T21:43:20.647 DEBUG:teuthology.orchestra.run.smithi086:> sudo rpm -qi grub2-tools 2024-02-19T21:43:20.676 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: kernel-core-4.18.0-539.el8.x86_64 3/3 2024-02-19T21:43:20.680 INFO:teuthology.orchestra.run.smithi086.stdout:Name : grub2-tools 2024-02-19T21:43:20.680 INFO:teuthology.orchestra.run.smithi086.stdout:Epoch : 1 2024-02-19T21:43:20.680 INFO:teuthology.orchestra.run.smithi086.stdout:Version : 2.02 2024-02-19T21:43:20.680 INFO:teuthology.orchestra.run.smithi086.stdout:Release : 129.el8 2024-02-19T21:43:20.682 INFO:teuthology.orchestra.run.smithi086.stdout:Architecture: x86_64 2024-02-19T21:43:20.682 INFO:teuthology.orchestra.run.smithi086.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-02-19T21:43:20.682 INFO:teuthology.orchestra.run.smithi086.stdout:Group : System Environment/Base 2024-02-19T21:43:20.682 INFO:teuthology.orchestra.run.smithi086.stdout:Size : 9265586 2024-02-19T21:43:20.682 INFO:teuthology.orchestra.run.smithi086.stdout:License : GPLv3+ 2024-02-19T21:43:20.682 INFO:teuthology.orchestra.run.smithi086.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-02-19T21:43:20.682 INFO:teuthology.orchestra.run.smithi086.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-02-19T21:43:20.682 INFO:teuthology.orchestra.run.smithi086.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-02-19T21:43:20.682 INFO:teuthology.orchestra.run.smithi086.stdout:Build Host : kbuilder.bsys.centos.org 2024-02-19T21:43:20.682 INFO:teuthology.orchestra.run.smithi086.stdout:Relocations : (not relocatable) 2024-02-19T21:43:20.682 INFO:teuthology.orchestra.run.smithi086.stdout:Packager : CentOS BuildSystem 2024-02-19T21:43:20.682 INFO:teuthology.orchestra.run.smithi086.stdout:Vendor : CentOS 2024-02-19T21:43:20.682 INFO:teuthology.orchestra.run.smithi086.stdout:URL : http://www.gnu.org/software/grub/ 2024-02-19T21:43:20.682 INFO:teuthology.orchestra.run.smithi086.stdout:Summary : Support tools for GRUB. 2024-02-19T21:43:20.683 INFO:teuthology.orchestra.run.smithi086.stdout:Description : 2024-02-19T21:43:20.683 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T21:43:20.683 INFO:teuthology.orchestra.run.smithi086.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-02-19T21:43:20.683 INFO:teuthology.orchestra.run.smithi086.stdout:customizable bootloader with modular architecture. It supports a rich 2024-02-19T21:43:20.683 INFO:teuthology.orchestra.run.smithi086.stdout:variety of kernel formats, file systems, computer architectures and 2024-02-19T21:43:20.683 INFO:teuthology.orchestra.run.smithi086.stdout:hardware devices. 2024-02-19T21:43:20.683 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T21:43:20.683 INFO:teuthology.orchestra.run.smithi086.stdout:This subpackage provides tools for support of all platforms. 2024-02-19T21:43:20.684 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-02-19T21:43:20.684 INFO:teuthology.task.kernel:Updating grub on smithi086 to boot 4.18.0-539.el8.x86_64 2024-02-19T21:43:20.684 DEBUG:teuthology.orchestra.run.smithi086:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-02-19T21:43:20.699 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: kernel-modules-4.18.0-539.el8.x86_64 3/3 2024-02-19T21:43:21.061 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: kernel-4.18.0-539.el8.x86_64 3/3 2024-02-19T21:43:21.061 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : kernel-4.18.0-539.el8.x86_64 1/3 2024-02-19T21:43:21.062 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : kernel-core-4.18.0-539.el8.x86_64 2/3 2024-02-19T21:43:21.541 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : kernel-modules-4.18.0-539.el8.x86_64 3/3 2024-02-19T21:43:21.541 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-02-19T21:43:21.541 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2024-02-19T21:43:21.541 INFO:teuthology.orchestra.run.smithi053.stdout: kernel-4.18.0-539.el8.x86_64 kernel-core-4.18.0-539.el8.x86_64 2024-02-19T21:43:21.541 INFO:teuthology.orchestra.run.smithi053.stdout: kernel-modules-4.18.0-539.el8.x86_64 2024-02-19T21:43:21.541 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-02-19T21:43:21.542 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-02-19T21:43:21.691 DEBUG:teuthology.orchestra.run.smithi053:> rpm -q kernel | sort -rV | head -n 1 2024-02-19T21:43:21.962 INFO:teuthology.orchestra.run.smithi053.stdout:kernel-4.18.0-539.el8.x86_64 2024-02-19T21:43:21.963 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-539.el8.x86_64 2024-02-19T21:43:21.963 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-539.el8.x86_64 2024-02-19T21:43:21.963 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-02-19T21:43:21.963 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-02-19T21:43:21.963 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2024-02-19T21:43:21.963 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi053.front.sepia.ceph.com, path=None, version=distro) 2024-02-19T21:43:21.963 DEBUG:teuthology.orchestra.run.smithi053:> rpm -q kernel | sort -rV | head -n 1 2024-02-19T21:43:22.183 INFO:teuthology.orchestra.run.smithi053.stdout:kernel-4.18.0-539.el8.x86_64 2024-02-19T21:43:22.183 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-539.el8.x86_64 2024-02-19T21:43:22.184 DEBUG:teuthology.orchestra.run.smithi053:> sudo rpm -qi grub2-tools 2024-02-19T21:43:22.214 INFO:teuthology.orchestra.run.smithi086.stderr:Generating grub configuration file ... 2024-02-19T21:43:22.218 INFO:teuthology.orchestra.run.smithi053.stdout:Name : grub2-tools 2024-02-19T21:43:22.218 INFO:teuthology.orchestra.run.smithi053.stdout:Epoch : 1 2024-02-19T21:43:22.218 INFO:teuthology.orchestra.run.smithi053.stdout:Version : 2.02 2024-02-19T21:43:22.218 INFO:teuthology.orchestra.run.smithi053.stdout:Release : 129.el8 2024-02-19T21:43:22.219 INFO:teuthology.orchestra.run.smithi053.stdout:Architecture: x86_64 2024-02-19T21:43:22.219 INFO:teuthology.orchestra.run.smithi053.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-02-19T21:43:22.219 INFO:teuthology.orchestra.run.smithi053.stdout:Group : System Environment/Base 2024-02-19T21:43:22.219 INFO:teuthology.orchestra.run.smithi053.stdout:Size : 9265586 2024-02-19T21:43:22.219 INFO:teuthology.orchestra.run.smithi053.stdout:License : GPLv3+ 2024-02-19T21:43:22.219 INFO:teuthology.orchestra.run.smithi053.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-02-19T21:43:22.219 INFO:teuthology.orchestra.run.smithi053.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-02-19T21:43:22.219 INFO:teuthology.orchestra.run.smithi053.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-02-19T21:43:22.219 INFO:teuthology.orchestra.run.smithi053.stdout:Build Host : kbuilder.bsys.centos.org 2024-02-19T21:43:22.219 INFO:teuthology.orchestra.run.smithi053.stdout:Relocations : (not relocatable) 2024-02-19T21:43:22.219 INFO:teuthology.orchestra.run.smithi053.stdout:Packager : CentOS BuildSystem 2024-02-19T21:43:22.219 INFO:teuthology.orchestra.run.smithi053.stdout:Vendor : CentOS 2024-02-19T21:43:22.219 INFO:teuthology.orchestra.run.smithi053.stdout:URL : http://www.gnu.org/software/grub/ 2024-02-19T21:43:22.219 INFO:teuthology.orchestra.run.smithi053.stdout:Summary : Support tools for GRUB. 2024-02-19T21:43:22.220 INFO:teuthology.orchestra.run.smithi053.stdout:Description : 2024-02-19T21:43:22.220 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-02-19T21:43:22.220 INFO:teuthology.orchestra.run.smithi053.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-02-19T21:43:22.220 INFO:teuthology.orchestra.run.smithi053.stdout:customizable bootloader with modular architecture. It supports a rich 2024-02-19T21:43:22.220 INFO:teuthology.orchestra.run.smithi053.stdout:variety of kernel formats, file systems, computer architectures and 2024-02-19T21:43:22.220 INFO:teuthology.orchestra.run.smithi053.stdout:hardware devices. 2024-02-19T21:43:22.220 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-02-19T21:43:22.220 INFO:teuthology.orchestra.run.smithi053.stdout:This subpackage provides tools for support of all platforms. 2024-02-19T21:43:22.221 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-02-19T21:43:22.221 INFO:teuthology.task.kernel:Updating grub on smithi053 to boot 4.18.0-539.el8.x86_64 2024-02-19T21:43:22.221 DEBUG:teuthology.orchestra.run.smithi053:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-02-19T21:43:23.458 INFO:teuthology.orchestra.run.smithi053.stderr:Generating grub configuration file ... 2024-02-19T21:43:24.035 INFO:teuthology.orchestra.run.smithi086.stderr:done 2024-02-19T21:43:24.037 DEBUG:teuthology.orchestra.run.smithi086:> mktemp 2024-02-19T21:43:24.069 INFO:teuthology.orchestra.run.smithi086.stdout:/tmp/tmp.QNdoo9VzTu 2024-02-19T21:43:24.069 DEBUG:teuthology.orchestra.run.smithi086:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.QNdoo9VzTu 2024-02-19T21:43:24.143 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod 0666 /tmp/tmp.QNdoo9VzTu 2024-02-19T21:43:24.297 DEBUG:teuthology.orchestra.remote:smithi086:/tmp/tmp.QNdoo9VzTu is 6KB 2024-02-19T21:43:24.346 DEBUG:teuthology.orchestra.run.smithi086:> rm -fr /tmp/tmp.QNdoo9VzTu 2024-02-19T21:43:24.363 DEBUG:teuthology.orchestra.run.smithi086:> sudo /bin/ls /boot/loader/entries || true 2024-02-19T21:43:24.436 INFO:teuthology.orchestra.run.smithi086.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-02-19T21:43:24.436 INFO:teuthology.orchestra.run.smithi086.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-02-19T21:43:24.436 INFO:teuthology.orchestra.run.smithi086.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-02-19T21:43:24.436 INFO:teuthology.orchestra.run.smithi086.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-539.el8.x86_64.conf 2024-02-19T21:43:24.438 DEBUG:teuthology.orchestra.run.smithi086:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-539.el8.x86_64 2024-02-19T21:43:24.582 DEBUG:teuthology.orchestra.run.smithi086:> sudo shutdown -r now 2024-02-19T21:43:25.372 INFO:teuthology.orchestra.run.smithi053.stderr:done 2024-02-19T21:43:25.374 DEBUG:teuthology.orchestra.run.smithi053:> mktemp 2024-02-19T21:43:25.443 INFO:teuthology.orchestra.run.smithi053.stdout:/tmp/tmp.oX5qzJw7X2 2024-02-19T21:43:25.443 DEBUG:teuthology.orchestra.run.smithi053:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.oX5qzJw7X2 2024-02-19T21:43:25.526 DEBUG:teuthology.orchestra.run.smithi053:> sudo chmod 0666 /tmp/tmp.oX5qzJw7X2 2024-02-19T21:43:25.661 DEBUG:teuthology.orchestra.remote:smithi053:/tmp/tmp.oX5qzJw7X2 is 6KB 2024-02-19T21:43:25.709 DEBUG:teuthology.orchestra.run.smithi053:> rm -fr /tmp/tmp.oX5qzJw7X2 2024-02-19T21:43:25.727 DEBUG:teuthology.orchestra.run.smithi053:> sudo /bin/ls /boot/loader/entries || true 2024-02-19T21:43:25.797 INFO:teuthology.orchestra.run.smithi053.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-02-19T21:43:25.797 INFO:teuthology.orchestra.run.smithi053.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-02-19T21:43:25.797 INFO:teuthology.orchestra.run.smithi053.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-02-19T21:43:25.798 INFO:teuthology.orchestra.run.smithi053.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-539.el8.x86_64.conf 2024-02-19T21:43:25.798 DEBUG:teuthology.orchestra.run.smithi053:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-539.el8.x86_64 2024-02-19T21:43:25.946 DEBUG:teuthology.orchestra.run.smithi053:> sudo shutdown -r now 2024-02-19T21:43:54.611 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2024-02-19T21:43:54.612 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-02-19T21:43:54.613 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T21:43:55.950 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2024-02-19T21:43:55.951 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi053.front.sepia.ceph.com' 2024-02-19T21:43:55.951 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi053.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T21:44:13.120 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.86 2024-02-19T21:44:14.400 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.53 2024-02-19T21:44:22.128 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-02-19T21:44:22.129 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T21:44:23.402 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi053.front.sepia.ceph.com' 2024-02-19T21:44:23.403 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi053.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T21:44:25.184 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.86 2024-02-19T21:44:37.197 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-02-19T21:44:37.198 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T21:44:41.788 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.53 2024-02-19T21:44:53.800 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi053.front.sepia.ceph.com' 2024-02-19T21:44:53.801 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi053.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T21:44:54.174 DEBUG:teuthology.orchestra.run.smithi053:> true 2024-02-19T21:44:54.439 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi053.front.sepia.ceph.com' 2024-02-19T21:44:54.439 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "4.18.0-539.el8.x86_64"... 2024-02-19T21:44:54.439 DEBUG:teuthology.orchestra.run.smithi053:> uname -r 2024-02-19T21:44:54.456 INFO:teuthology.orchestra.run.smithi053.stdout:4.18.0-539.el8.x86_64 2024-02-19T21:44:54.457 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-539.el8.x86_64 vs 4.18.0-539.el8.x86_64 2024-02-19T21:44:54.457 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-02-19T21:44:54.457 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-02-19T21:44:55.457 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2024-02-19T21:44:55.458 DEBUG:teuthology.orchestra.run.smithi053:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-02-19T21:44:55.532 INFO:teuthology.orchestra.run.smithi053.stdout:ttyS1 2024-02-19T21:44:55.549 DEBUG:teuthology.parallel:result is None 2024-02-19T21:44:55.612 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.86 2024-02-19T21:45:10.613 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-02-19T21:45:10.614 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T21:45:10.906 DEBUG:teuthology.orchestra.run.smithi086:> true 2024-02-19T21:45:11.151 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-02-19T21:45:11.151 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "4.18.0-539.el8.x86_64"... 2024-02-19T21:45:11.151 DEBUG:teuthology.orchestra.run.smithi086:> uname -r 2024-02-19T21:45:11.167 INFO:teuthology.orchestra.run.smithi086.stdout:4.18.0-539.el8.x86_64 2024-02-19T21:45:11.168 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-539.el8.x86_64 vs 4.18.0-539.el8.x86_64 2024-02-19T21:45:11.168 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-02-19T21:45:11.168 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-02-19T21:45:12.169 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2024-02-19T21:45:12.169 DEBUG:teuthology.orchestra.run.smithi086:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-02-19T21:45:12.919 INFO:teuthology.orchestra.run.smithi086.stdout:ttyS1 2024-02-19T21:45:12.952 DEBUG:teuthology.parallel:result is None 2024-02-19T21:45:12.952 INFO:teuthology.run_tasks:Running task internal.base... 2024-02-19T21:45:12.962 INFO:teuthology.task.internal:Creating test directory... 2024-02-19T21:45:12.962 DEBUG:teuthology.orchestra.run.smithi053:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-02-19T21:45:12.966 DEBUG:teuthology.orchestra.run.smithi086:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-02-19T21:45:13.383 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-02-19T21:45:13.390 INFO:teuthology.run_tasks:Running task internal.archive... 2024-02-19T21:45:13.398 INFO:teuthology.task.internal:Creating archive directory... 2024-02-19T21:45:13.398 DEBUG:teuthology.orchestra.run.smithi053:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-02-19T21:45:13.400 DEBUG:teuthology.orchestra.run.smithi086:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-02-19T21:45:13.524 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-02-19T21:45:13.531 INFO:teuthology.task.internal:Enabling coredump saving... 2024-02-19T21:45:13.531 DEBUG:teuthology.orchestra.run.smithi053:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-02-19T21:45:13.533 DEBUG:teuthology.orchestra.run.smithi086:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-02-19T21:45:13.569 INFO:teuthology.orchestra.run.smithi086.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-19T21:45:13.578 INFO:teuthology.orchestra.run.smithi053.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-19T21:45:13.579 INFO:teuthology.orchestra.run.smithi086.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-19T21:45:13.590 INFO:teuthology.orchestra.run.smithi053.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-19T21:45:13.592 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-02-19T21:45:13.599 INFO:teuthology.task.internal:Configuring sudo... 2024-02-19T21:45:13.599 DEBUG:teuthology.orchestra.run.smithi053:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-02-19T21:45:13.601 DEBUG:teuthology.orchestra.run.smithi086:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-02-19T21:45:13.644 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-02-19T21:45:13.656 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-02-19T21:45:13.656 DEBUG:teuthology.orchestra.run.smithi053:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-02-19T21:45:13.672 DEBUG:teuthology.orchestra.run.smithi086:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-02-19T21:45:13.700 DEBUG:teuthology.orchestra.run.smithi053:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-19T21:45:13.779 DEBUG:teuthology.orchestra.run.smithi053:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-19T21:45:13.829 DEBUG:teuthology.orchestra.run.smithi053:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-19T21:45:13.917 DEBUG:teuthology.orchestra.run.smithi053:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-19T21:45:13.960 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-02-19T21:45:13.961 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-02-19T21:45:14.030 DEBUG:teuthology.orchestra.run.smithi086:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-19T21:45:14.079 DEBUG:teuthology.orchestra.run.smithi086:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-19T21:45:14.185 DEBUG:teuthology.orchestra.run.smithi086:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-19T21:45:14.230 DEBUG:teuthology.orchestra.run.smithi086:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-19T21:45:14.268 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-02-19T21:45:14.268 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-02-19T21:45:14.331 DEBUG:teuthology.orchestra.run.smithi053:> sudo service rsyslog restart 2024-02-19T21:45:14.333 DEBUG:teuthology.orchestra.run.smithi086:> sudo service rsyslog restart 2024-02-19T21:45:14.394 INFO:teuthology.orchestra.run.smithi053.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-19T21:45:14.433 INFO:teuthology.orchestra.run.smithi086.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-19T21:45:14.873 INFO:teuthology.run_tasks:Running task internal.timer... 2024-02-19T21:45:14.883 INFO:teuthology.task.internal:Starting timer... 2024-02-19T21:45:14.883 INFO:teuthology.run_tasks:Running task pcp... 2024-02-19T21:45:14.893 INFO:teuthology.run_tasks:Running task selinux... 2024-02-19T21:45:14.903 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-02-19T21:45:14.903 DEBUG:teuthology.orchestra.run.smithi053:> sudo service auditd rotate 2024-02-19T21:45:15.001 INFO:teuthology.orchestra.run.smithi053.stdout:Rotating logs: 2024-02-19T21:45:15.002 DEBUG:teuthology.orchestra.run.smithi086:> sudo service auditd rotate 2024-02-19T21:45:15.076 INFO:teuthology.orchestra.run.smithi086.stdout:Rotating logs: 2024-02-19T21:45:15.078 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-02-19T21:45:15.078 DEBUG:teuthology.orchestra.run.smithi053:> /usr/sbin/getenforce 2024-02-19T21:45:15.106 INFO:teuthology.orchestra.run.smithi053.stdout:Permissive 2024-02-19T21:45:15.107 DEBUG:teuthology.orchestra.run.smithi086:> /usr/sbin/getenforce 2024-02-19T21:45:15.132 INFO:teuthology.orchestra.run.smithi086.stdout:Permissive 2024-02-19T21:45:15.133 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi053.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi086.front.sepia.ceph.com': 'permissive'} 2024-02-19T21:45:15.133 DEBUG:teuthology.orchestra.run.smithi053:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-19T21:45:15.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-19T21:45:15.178 DEBUG:teuthology.orchestra.run.smithi086:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-19T21:45:15.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-19T21:45:15.204 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-02-19T21:45:15.205 DEBUG:teuthology.orchestra.run.smithi053:> sudo /usr/sbin/setenforce permissive 2024-02-19T21:45:15.246 DEBUG:teuthology.orchestra.run.smithi086:> sudo /usr/sbin/setenforce permissive 2024-02-19T21:45:15.273 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-02-19T21:45:15.283 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-02-19T21:45:15.287 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-02-19T21:45:15.414 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-02-19T21:45:15.429 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-02-19T21:45:15.431 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi053.front.sepia.ceph.com,smithi086.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-02-19T21:51:12.357 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi053.front.sepia.ceph.com'), Remote(name='ubuntu@smithi086.front.sepia.ceph.com')] 2024-02-19T21:51:12.358 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi053.front.sepia.ceph.com' 2024-02-19T21:51:12.359 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi053.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T21:51:12.435 DEBUG:teuthology.orchestra.run.smithi053:> true 2024-02-19T21:51:12.504 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi053.front.sepia.ceph.com' 2024-02-19T21:51:12.505 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-02-19T21:51:12.505 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T21:51:12.579 DEBUG:teuthology.orchestra.run.smithi086:> true 2024-02-19T21:51:12.604 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-02-19T21:51:12.605 INFO:teuthology.run_tasks:Running task clock... 2024-02-19T21:51:12.616 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-02-19T21:51:12.617 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-02-19T21:51:12.617 DEBUG:teuthology.orchestra.run.smithi053:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-19T21:51:12.620 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-02-19T21:51:12.620 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-19T21:51:12.651 INFO:teuthology.orchestra.run.smithi053.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-02-19T21:51:12.665 INFO:teuthology.orchestra.run.smithi053.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-02-19T21:51:12.676 INFO:teuthology.orchestra.run.smithi086.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-02-19T21:51:12.689 INFO:teuthology.orchestra.run.smithi086.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-02-19T21:51:12.731 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: ntpd: command not found 2024-02-19T21:51:12.743 INFO:teuthology.orchestra.run.smithi053.stdout:506 Cannot talk to daemon 2024-02-19T21:51:12.753 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: ntpd: command not found 2024-02-19T21:51:12.757 INFO:teuthology.orchestra.run.smithi053.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-02-19T21:51:12.764 INFO:teuthology.orchestra.run.smithi086.stdout:506 Cannot talk to daemon 2024-02-19T21:51:12.770 INFO:teuthology.orchestra.run.smithi053.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-02-19T21:51:12.777 INFO:teuthology.orchestra.run.smithi086.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-02-19T21:51:12.790 INFO:teuthology.orchestra.run.smithi086.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-02-19T21:51:12.843 INFO:teuthology.orchestra.run.smithi053.stderr:bash: ntpq: command not found 2024-02-19T21:51:12.847 INFO:teuthology.orchestra.run.smithi053.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-19T21:51:12.847 INFO:teuthology.orchestra.run.smithi053.stdout:=============================================================================== 2024-02-19T21:51:12.847 INFO:teuthology.orchestra.run.smithi053.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-19T21:51:12.847 INFO:teuthology.orchestra.run.smithi053.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-19T21:51:12.847 INFO:teuthology.orchestra.run.smithi053.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-19T21:51:12.848 INFO:teuthology.orchestra.run.smithi053.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-19T21:51:12.866 INFO:teuthology.orchestra.run.smithi086.stderr:bash: ntpq: command not found 2024-02-19T21:51:12.870 INFO:teuthology.orchestra.run.smithi086.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-19T21:51:12.870 INFO:teuthology.orchestra.run.smithi086.stdout:=============================================================================== 2024-02-19T21:51:12.870 INFO:teuthology.orchestra.run.smithi086.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-19T21:51:12.870 INFO:teuthology.orchestra.run.smithi086.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-19T21:51:12.870 INFO:teuthology.orchestra.run.smithi086.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-19T21:51:12.870 INFO:teuthology.orchestra.run.smithi086.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-19T21:51:12.871 INFO:teuthology.run_tasks:Running task pexec... 2024-02-19T21:51:12.881 INFO:teuthology.task.pexec:Executing custom commands... 2024-02-19T21:51:12.881 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi053.front.sepia.ceph.com 2024-02-19T21:51:12.881 DEBUG:teuthology.orchestra.run.smithi053:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-02-19T21:51:12.881 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi086.front.sepia.ceph.com 2024-02-19T21:51:12.882 DEBUG:teuthology.orchestra.run.smithi086:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-02-19T21:51:13.698 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:34 ago on Mon 19 Feb 2024 09:49:39 PM UTC. 2024-02-19T21:51:13.705 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:01:27 ago on Mon 19 Feb 2024 09:49:46 PM UTC. 2024-02-19T21:51:15.729 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-02-19T21:51:15.730 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-02-19T21:51:15.730 INFO:teuthology.orchestra.run.smithi086.stdout: Package Architecture Version Repository Size 2024-02-19T21:51:15.731 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-02-19T21:51:15.731 INFO:teuthology.orchestra.run.smithi086.stdout:Resetting modules: 2024-02-19T21:51:15.731 INFO:teuthology.orchestra.run.smithi086.stdout: container-tools 2024-02-19T21:51:15.731 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T21:51:15.731 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-02-19T21:51:15.731 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-02-19T21:51:15.731 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T21:51:15.860 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-02-19T21:51:15.861 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-02-19T21:51:15.861 INFO:teuthology.orchestra.run.smithi053.stdout: Package Architecture Version Repository Size 2024-02-19T21:51:15.861 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-02-19T21:51:15.861 INFO:teuthology.orchestra.run.smithi053.stdout:Resetting modules: 2024-02-19T21:51:15.861 INFO:teuthology.orchestra.run.smithi053.stdout: container-tools 2024-02-19T21:51:15.861 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-02-19T21:51:15.861 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2024-02-19T21:51:15.861 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-02-19T21:51:15.861 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-02-19T21:51:15.987 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-02-19T21:51:16.126 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-02-19T21:51:16.857 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:37 ago on Mon 19 Feb 2024 09:49:39 PM UTC. 2024-02-19T21:51:17.063 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:01:30 ago on Mon 19 Feb 2024 09:49:46 PM UTC. 2024-02-19T21:51:18.874 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-02-19T21:51:18.909 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-02-19T21:51:18.909 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2024-02-19T21:51:18.909 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-02-19T21:51:18.909 INFO:teuthology.orchestra.run.smithi086.stdout:Installing group/module packages: 2024-02-19T21:51:18.909 INFO:teuthology.orchestra.run.smithi086.stdout: aardvark-dns x86_64 2:1.10.0-1.module_el8+871+552be958 appstream 1.0 M 2024-02-19T21:51:18.910 INFO:teuthology.orchestra.run.smithi086.stdout: buildah x86_64 2:1.33.5-1.module_el8+885+7da147f3 appstream 9.7 M 2024-02-19T21:51:18.910 INFO:teuthology.orchestra.run.smithi086.stdout: cockpit-podman noarch 83-1.module_el8+847+7863d4e6 appstream 688 k 2024-02-19T21:51:18.910 INFO:teuthology.orchestra.run.smithi086.stdout: crun x86_64 1.14-1.module_el8+871+552be958 appstream 257 k 2024-02-19T21:51:18.910 INFO:teuthology.orchestra.run.smithi086.stdout: netavark x86_64 2:1.10.2-1.module_el8+885+7da147f3 appstream 4.3 M 2024-02-19T21:51:18.910 INFO:teuthology.orchestra.run.smithi086.stdout: python3-podman noarch 4.9.0-1.module_el8+847+7863d4e6 appstream 180 k 2024-02-19T21:51:18.910 INFO:teuthology.orchestra.run.smithi086.stdout: skopeo x86_64 2:1.14.3-0.1.module_el8+885+7da147f3 appstream 8.8 M 2024-02-19T21:51:18.910 INFO:teuthology.orchestra.run.smithi086.stdout: toolbox x86_64 0.0.99.5-1.module_el8+847+7863d4e6 appstream 2.5 M 2024-02-19T21:51:18.910 INFO:teuthology.orchestra.run.smithi086.stdout: udica noarch 0.2.6-20.module_el8+804+f131391c appstream 53 k 2024-02-19T21:51:18.910 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2024-02-19T21:51:18.910 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2024-02-19T21:51:18.910 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2024-02-19T21:51:18.910 INFO:teuthology.orchestra.run.smithi086.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2024-02-19T21:51:18.910 INFO:teuthology.orchestra.run.smithi086.stdout:Installing weak dependencies: 2024-02-19T21:51:18.911 INFO:teuthology.orchestra.run.smithi086.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2024-02-19T21:51:18.911 INFO:teuthology.orchestra.run.smithi086.stdout:Installing module profiles: 2024-02-19T21:51:18.911 INFO:teuthology.orchestra.run.smithi086.stdout: container-tools/common 2024-02-19T21:51:18.911 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T21:51:18.911 INFO:teuthology.orchestra.run.smithi086.stdout:Enabling module streams: 2024-02-19T21:51:18.911 INFO:teuthology.orchestra.run.smithi086.stdout: container-tools rhel8 2024-02-19T21:51:18.911 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T21:51:18.911 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-02-19T21:51:18.911 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-02-19T21:51:18.911 INFO:teuthology.orchestra.run.smithi086.stdout:Install 13 Packages 2024-02-19T21:51:18.911 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T21:51:18.912 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 28 M 2024-02-19T21:51:18.912 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 92 M 2024-02-19T21:51:18.912 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-02-19T21:51:19.234 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-02-19T21:51:19.273 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-02-19T21:51:19.273 INFO:teuthology.orchestra.run.smithi053.stdout: Package Arch Version Repository Size 2024-02-19T21:51:19.273 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-02-19T21:51:19.273 INFO:teuthology.orchestra.run.smithi053.stdout:Installing group/module packages: 2024-02-19T21:51:19.273 INFO:teuthology.orchestra.run.smithi053.stdout: aardvark-dns x86_64 2:1.10.0-1.module_el8+871+552be958 appstream 1.0 M 2024-02-19T21:51:19.273 INFO:teuthology.orchestra.run.smithi053.stdout: buildah x86_64 2:1.33.5-1.module_el8+885+7da147f3 appstream 9.7 M 2024-02-19T21:51:19.273 INFO:teuthology.orchestra.run.smithi053.stdout: cockpit-podman noarch 83-1.module_el8+847+7863d4e6 appstream 688 k 2024-02-19T21:51:19.273 INFO:teuthology.orchestra.run.smithi053.stdout: crun x86_64 1.14-1.module_el8+871+552be958 appstream 257 k 2024-02-19T21:51:19.274 INFO:teuthology.orchestra.run.smithi053.stdout: netavark x86_64 2:1.10.2-1.module_el8+885+7da147f3 appstream 4.3 M 2024-02-19T21:51:19.274 INFO:teuthology.orchestra.run.smithi053.stdout: python3-podman noarch 4.9.0-1.module_el8+847+7863d4e6 appstream 180 k 2024-02-19T21:51:19.274 INFO:teuthology.orchestra.run.smithi053.stdout: skopeo x86_64 2:1.14.3-0.1.module_el8+885+7da147f3 appstream 8.8 M 2024-02-19T21:51:19.274 INFO:teuthology.orchestra.run.smithi053.stdout: toolbox x86_64 0.0.99.5-1.module_el8+847+7863d4e6 appstream 2.5 M 2024-02-19T21:51:19.274 INFO:teuthology.orchestra.run.smithi053.stdout: udica noarch 0.2.6-20.module_el8+804+f131391c appstream 53 k 2024-02-19T21:51:19.274 INFO:teuthology.orchestra.run.smithi053.stdout:Installing dependencies: 2024-02-19T21:51:19.274 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2024-02-19T21:51:19.274 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2024-02-19T21:51:19.274 INFO:teuthology.orchestra.run.smithi053.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2024-02-19T21:51:19.274 INFO:teuthology.orchestra.run.smithi053.stdout:Installing weak dependencies: 2024-02-19T21:51:19.274 INFO:teuthology.orchestra.run.smithi053.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2024-02-19T21:51:19.274 INFO:teuthology.orchestra.run.smithi053.stdout:Installing module profiles: 2024-02-19T21:51:19.274 INFO:teuthology.orchestra.run.smithi053.stdout: container-tools/common 2024-02-19T21:51:19.275 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-02-19T21:51:19.275 INFO:teuthology.orchestra.run.smithi053.stdout:Enabling module streams: 2024-02-19T21:51:19.275 INFO:teuthology.orchestra.run.smithi053.stdout: container-tools rhel8 2024-02-19T21:51:19.275 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-02-19T21:51:19.275 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2024-02-19T21:51:19.275 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-02-19T21:51:19.275 INFO:teuthology.orchestra.run.smithi053.stdout:Install 13 Packages 2024-02-19T21:51:19.275 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-02-19T21:51:19.275 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 28 M 2024-02-19T21:51:19.276 INFO:teuthology.orchestra.run.smithi053.stdout:Installed size: 92 M 2024-02-19T21:51:19.276 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2024-02-19T21:51:19.290 INFO:teuthology.orchestra.run.smithi086.stdout:(1/13): cockpit-podman-83-1.module_el8+847+7863 3.7 MB/s | 688 kB 00:00 2024-02-19T21:51:19.324 INFO:teuthology.orchestra.run.smithi086.stdout:(2/13): aardvark-dns-1.10.0-1.module_el8+871+55 4.7 MB/s | 1.0 MB 00:00 2024-02-19T21:51:19.349 INFO:teuthology.orchestra.run.smithi086.stdout:(3/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 650 kB/s | 38 kB 00:00 2024-02-19T21:51:19.391 INFO:teuthology.orchestra.run.smithi086.stdout:(4/13): crun-1.14-1.module_el8+871+552be958.x86 3.8 MB/s | 257 kB 00:00 2024-02-19T21:51:19.564 INFO:teuthology.orchestra.run.smithi053.stdout:(1/13): aardvark-dns-1.10.0-1.module_el8+871+55 4.1 MB/s | 1.0 MB 00:00 2024-02-19T21:51:19.597 INFO:teuthology.orchestra.run.smithi053.stdout:(2/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 1.1 MB/s | 38 kB 00:00 2024-02-19T21:51:19.656 INFO:teuthology.orchestra.run.smithi053.stdout:(3/13): crun-1.14-1.module_el8+871+552be958.x86 4.3 MB/s | 257 kB 00:00 2024-02-19T21:51:19.658 INFO:teuthology.orchestra.run.smithi086.stdout:(5/13): python3-podman-4.9.0-1.module_el8+847+7 675 kB/s | 180 kB 00:00 2024-02-19T21:51:19.709 INFO:teuthology.orchestra.run.smithi086.stdout:(6/13): python3-pytoml-0.1.14-5.git7dea353.el8. 496 kB/s | 25 kB 00:00 2024-02-19T21:51:20.009 INFO:teuthology.orchestra.run.smithi086.stdout:(7/13): python3-pyxdg-0.25-16.el8.noarch.rpm 313 kB/s | 94 kB 00:00 2024-02-19T21:51:20.067 INFO:teuthology.orchestra.run.smithi086.stdout:(8/13): netavark-1.10.2-1.module_el8+885+7da147 6.0 MB/s | 4.3 MB 00:00 2024-02-19T21:51:20.140 INFO:teuthology.orchestra.run.smithi053.stdout:(4/13): netavark-1.10.2-1.module_el8+885+7da147 8.9 MB/s | 4.3 MB 00:00 2024-02-19T21:51:20.182 INFO:teuthology.orchestra.run.smithi053.stdout:(5/13): python3-podman-4.9.0-1.module_el8+847+7 4.3 MB/s | 180 kB 00:00 2024-02-19T21:51:20.248 INFO:teuthology.orchestra.run.smithi053.stdout:(6/13): python3-pytoml-0.1.14-5.git7dea353.el8. 373 kB/s | 25 kB 00:00 2024-02-19T21:51:20.274 INFO:teuthology.orchestra.run.smithi053.stdout:(7/13): cockpit-podman-83-1.module_el8+847+7863 720 kB/s | 688 kB 00:00 2024-02-19T21:51:20.308 INFO:teuthology.orchestra.run.smithi053.stdout:(8/13): python3-pyxdg-0.25-16.el8.noarch.rpm 1.6 MB/s | 94 kB 00:00 2024-02-19T21:51:21.000 INFO:teuthology.orchestra.run.smithi053.stdout:(9/13): toolbox-0.0.99.5-1.module_el8+847+7863d 3.7 MB/s | 2.5 MB 00:00 2024-02-19T21:51:21.041 INFO:teuthology.orchestra.run.smithi053.stdout:(10/13): udica-0.2.6-20.module_el8+804+f131391c 1.2 MB/s | 53 kB 00:00 2024-02-19T21:51:21.108 INFO:teuthology.orchestra.run.smithi053.stdout:(11/13): yajl-2.1.0-12.el8.x86_64.rpm 615 kB/s | 41 kB 00:00 2024-02-19T21:51:21.251 INFO:teuthology.orchestra.run.smithi086.stdout:(9/13): toolbox-0.0.99.5-1.module_el8+847+7863d 2.1 MB/s | 2.5 MB 00:01 2024-02-19T21:51:21.343 INFO:teuthology.orchestra.run.smithi086.stdout:(10/13): udica-0.2.6-20.module_el8+804+f131391c 575 kB/s | 53 kB 00:00 2024-02-19T21:51:21.410 INFO:teuthology.orchestra.run.smithi086.stdout:(11/13): yajl-2.1.0-12.el8.x86_64.rpm 615 kB/s | 41 kB 00:00 2024-02-19T21:51:22.110 INFO:teuthology.orchestra.run.smithi086.stdout:(12/13): buildah-1.33.5-1.module_el8+885+7da147 3.2 MB/s | 9.7 MB 00:03 2024-02-19T21:51:24.069 INFO:teuthology.orchestra.run.smithi086.stdout:(13/13): skopeo-1.14.3-0.1.module_el8+885+7da14 2.2 MB/s | 8.8 MB 00:04 2024-02-19T21:51:24.069 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-02-19T21:51:24.069 INFO:teuthology.orchestra.run.smithi086.stdout:Total 5.4 MB/s | 28 MB 00:05 2024-02-19T21:51:24.340 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-02-19T21:51:24.361 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-02-19T21:51:24.361 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-02-19T21:51:24.625 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-02-19T21:51:24.628 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-02-19T21:51:25.293 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-02-19T21:51:26.226 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2024-02-19T21:51:26.427 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 2/13 2024-02-19T21:51:26.631 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 3/13 2024-02-19T21:51:26.815 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 4/13 2024-02-19T21:51:27.025 INFO:teuthology.orchestra.run.smithi053.stdout:(12/13): skopeo-1.14.3-0.1.module_el8+885+7da14 1.3 MB/s | 8.8 MB 00:06 2024-02-19T21:51:27.041 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 5/13 2024-02-19T21:51:27.217 INFO:teuthology.orchestra.run.smithi053.stdout:(13/13): buildah-1.33.5-1.module_el8+885+7da147 1.2 MB/s | 9.7 MB 00:07 2024-02-19T21:51:27.218 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2024-02-19T21:51:27.218 INFO:teuthology.orchestra.run.smithi053.stdout:Total 3.5 MB/s | 28 MB 00:07 2024-02-19T21:51:27.499 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2024-02-19T21:51:27.522 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2024-02-19T21:51:27.522 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2024-02-19T21:51:27.611 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x8 6/13 2024-02-19T21:51:27.799 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2024-02-19T21:51:27.802 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2024-02-19T21:51:28.505 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2024-02-19T21:51:28.579 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : netavark-2:1.10.2-1.module_el8+885+7da147f3.x86_64 7/13 2024-02-19T21:51:28.828 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : buildah-2:1.33.5-1.module_el8+885+7da147f3.x86_64 8/13 2024-02-19T21:51:29.048 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : crun-1.14-1.module_el8+871+552be958.x86_64 9/13 2024-02-19T21:51:29.481 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-podman-4.9.0-1.module_el8+847+7863d4e6.noa 10/13 2024-02-19T21:51:29.503 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2024-02-19T21:51:29.689 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 2/13 2024-02-19T21:51:29.711 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : toolbox-0.0.99.5-1.module_el8+847+7863d4e6.x86_64 11/13 2024-02-19T21:51:29.858 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 3/13 2024-02-19T21:51:29.926 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/13 2024-02-19T21:51:30.052 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 4/13 2024-02-19T21:51:30.138 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : cockpit-podman-83-1.module_el8+847+7863d4e6.noarch 13/13 2024-02-19T21:51:30.281 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 5/13 2024-02-19T21:51:30.563 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: cockpit-podman-83-1.module_el8+847+7863d4e6.noarch 13/13 2024-02-19T21:51:30.564 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x8 1/13 2024-02-19T21:51:30.564 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : buildah-2:1.33.5-1.module_el8+885+7da147f3.x86_64 2/13 2024-02-19T21:51:30.564 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : cockpit-podman-83-1.module_el8+847+7863d4e6.noarch 3/13 2024-02-19T21:51:30.564 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-02-19T21:51:30.564 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : crun-1.14-1.module_el8+871+552be958.x86_64 5/13 2024-02-19T21:51:30.564 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : netavark-2:1.10.2-1.module_el8+885+7da147f3.x86_64 6/13 2024-02-19T21:51:30.564 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-podman-4.9.0-1.module_el8+847+7863d4e6.noa 7/13 2024-02-19T21:51:30.564 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2024-02-19T21:51:30.564 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2024-02-19T21:51:30.564 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 10/13 2024-02-19T21:51:30.565 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : toolbox-0.0.99.5-1.module_el8+847+7863d4e6.x86_64 11/13 2024-02-19T21:51:30.565 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/13 2024-02-19T21:51:30.896 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x8 6/13 2024-02-19T21:51:31.070 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2024-02-19T21:51:31.071 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T21:51:31.071 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-02-19T21:51:31.071 INFO:teuthology.orchestra.run.smithi086.stdout: aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x86_64 2024-02-19T21:51:31.071 INFO:teuthology.orchestra.run.smithi086.stdout: buildah-2:1.33.5-1.module_el8+885+7da147f3.x86_64 2024-02-19T21:51:31.071 INFO:teuthology.orchestra.run.smithi086.stdout: cockpit-podman-83-1.module_el8+847+7863d4e6.noarch 2024-02-19T21:51:31.071 INFO:teuthology.orchestra.run.smithi086.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2024-02-19T21:51:31.071 INFO:teuthology.orchestra.run.smithi086.stdout: crun-1.14-1.module_el8+871+552be958.x86_64 2024-02-19T21:51:31.071 INFO:teuthology.orchestra.run.smithi086.stdout: netavark-2:1.10.2-1.module_el8+885+7da147f3.x86_64 2024-02-19T21:51:31.071 INFO:teuthology.orchestra.run.smithi086.stdout: python3-podman-4.9.0-1.module_el8+847+7863d4e6.noarch 2024-02-19T21:51:31.071 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2024-02-19T21:51:31.071 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pyxdg-0.25-16.el8.noarch 2024-02-19T21:51:31.071 INFO:teuthology.orchestra.run.smithi086.stdout: skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 2024-02-19T21:51:31.071 INFO:teuthology.orchestra.run.smithi086.stdout: toolbox-0.0.99.5-1.module_el8+847+7863d4e6.x86_64 2024-02-19T21:51:31.071 INFO:teuthology.orchestra.run.smithi086.stdout: udica-0.2.6-20.module_el8+804+f131391c.noarch 2024-02-19T21:51:31.072 INFO:teuthology.orchestra.run.smithi086.stdout: yajl-2.1.0-12.el8.x86_64 2024-02-19T21:51:31.072 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T21:51:31.072 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-02-19T21:51:31.509 DEBUG:teuthology.parallel:result is None 2024-02-19T21:51:31.899 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : netavark-2:1.10.2-1.module_el8+885+7da147f3.x86_64 7/13 2024-02-19T21:51:32.109 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : buildah-2:1.33.5-1.module_el8+885+7da147f3.x86_64 8/13 2024-02-19T21:51:32.299 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : crun-1.14-1.module_el8+871+552be958.x86_64 9/13 2024-02-19T21:51:32.746 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-podman-4.9.0-1.module_el8+847+7863d4e6.noa 10/13 2024-02-19T21:51:33.059 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : toolbox-0.0.99.5-1.module_el8+847+7863d4e6.x86_64 11/13 2024-02-19T21:51:33.270 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/13 2024-02-19T21:51:33.465 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : cockpit-podman-83-1.module_el8+847+7863d4e6.noarch 13/13 2024-02-19T21:51:33.896 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: cockpit-podman-83-1.module_el8+847+7863d4e6.noarch 13/13 2024-02-19T21:51:33.896 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x8 1/13 2024-02-19T21:51:33.896 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : buildah-2:1.33.5-1.module_el8+885+7da147f3.x86_64 2/13 2024-02-19T21:51:33.896 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : cockpit-podman-83-1.module_el8+847+7863d4e6.noarch 3/13 2024-02-19T21:51:33.896 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-02-19T21:51:33.896 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : crun-1.14-1.module_el8+871+552be958.x86_64 5/13 2024-02-19T21:51:33.897 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : netavark-2:1.10.2-1.module_el8+885+7da147f3.x86_64 6/13 2024-02-19T21:51:33.897 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-podman-4.9.0-1.module_el8+847+7863d4e6.noa 7/13 2024-02-19T21:51:33.897 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2024-02-19T21:51:33.897 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2024-02-19T21:51:33.897 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 10/13 2024-02-19T21:51:33.897 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : toolbox-0.0.99.5-1.module_el8+847+7863d4e6.x86_64 11/13 2024-02-19T21:51:33.897 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/13 2024-02-19T21:51:34.430 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2024-02-19T21:51:34.431 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-02-19T21:51:34.431 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2024-02-19T21:51:34.431 INFO:teuthology.orchestra.run.smithi053.stdout: aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x86_64 2024-02-19T21:51:34.431 INFO:teuthology.orchestra.run.smithi053.stdout: buildah-2:1.33.5-1.module_el8+885+7da147f3.x86_64 2024-02-19T21:51:34.431 INFO:teuthology.orchestra.run.smithi053.stdout: cockpit-podman-83-1.module_el8+847+7863d4e6.noarch 2024-02-19T21:51:34.431 INFO:teuthology.orchestra.run.smithi053.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2024-02-19T21:51:34.431 INFO:teuthology.orchestra.run.smithi053.stdout: crun-1.14-1.module_el8+871+552be958.x86_64 2024-02-19T21:51:34.431 INFO:teuthology.orchestra.run.smithi053.stdout: netavark-2:1.10.2-1.module_el8+885+7da147f3.x86_64 2024-02-19T21:51:34.431 INFO:teuthology.orchestra.run.smithi053.stdout: python3-podman-4.9.0-1.module_el8+847+7863d4e6.noarch 2024-02-19T21:51:34.431 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2024-02-19T21:51:34.431 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pyxdg-0.25-16.el8.noarch 2024-02-19T21:51:34.431 INFO:teuthology.orchestra.run.smithi053.stdout: skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 2024-02-19T21:51:34.432 INFO:teuthology.orchestra.run.smithi053.stdout: toolbox-0.0.99.5-1.module_el8+847+7863d4e6.x86_64 2024-02-19T21:51:34.432 INFO:teuthology.orchestra.run.smithi053.stdout: udica-0.2.6-20.module_el8+804+f131391c.noarch 2024-02-19T21:51:34.432 INFO:teuthology.orchestra.run.smithi053.stdout: yajl-2.1.0-12.el8.x86_64 2024-02-19T21:51:34.432 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-02-19T21:51:34.432 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-02-19T21:51:34.711 DEBUG:teuthology.parallel:result is None 2024-02-19T21:51:34.711 INFO:teuthology.run_tasks:Running task pexec... 2024-02-19T21:51:34.719 INFO:teuthology.task.pexec:Executing custom commands... 2024-02-19T21:51:34.720 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi053.front.sepia.ceph.com 2024-02-19T21:51:34.720 DEBUG:teuthology.orchestra.run.smithi053:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-02-19T21:51:34.720 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi086.front.sepia.ceph.com 2024-02-19T21:51:34.721 DEBUG:teuthology.orchestra.run.smithi086:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-02-19T21:51:34.894 DEBUG:teuthology.parallel:result is None 2024-02-19T21:51:34.904 DEBUG:teuthology.parallel:result is None 2024-02-19T21:51:34.904 INFO:teuthology.run_tasks:Running task install... 2024-02-19T21:51:34.913 DEBUG:teuthology.task.install:project ceph 2024-02-19T21:51:34.913 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '81bd20d634209c7cb82c18be12b4b5a05643ebf1'}} 2024-02-19T21:51:34.913 DEBUG:teuthology.task.install:config {'exclude_packages': ['ceph-volume'], 'tag': 'v16.2.4', 'flavor': 'default', 'sha1': '81bd20d634209c7cb82c18be12b4b5a05643ebf1'} 2024-02-19T21:51:34.913 INFO:teuthology.task.install:Using flavor: default 2024-02-19T21:51:34.921 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-02-19T21:51:34.921 INFO:teuthology.task.install:extra packages: [] 2024-02-19T21:51:34.922 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '81bd20d634209c7cb82c18be12b4b5a05643ebf1', 'tag': 'v16.2.4', 'wait_for_package': False} 2024-02-19T21:51:34.922 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-02-19T21:51:34.922 INFO:teuthology.packaging:ref: None 2024-02-19T21:51:34.922 INFO:teuthology.packaging:tag: v16.2.4 2024-02-19T21:51:34.922 INFO:teuthology.packaging:branch: None 2024-02-19T21:51:34.922 INFO:teuthology.packaging:sha1: 81bd20d634209c7cb82c18be12b4b5a05643ebf1 2024-02-19T21:51:35.176 DEBUG:teuthology.repo_utils:git ls-remote https://github.com/ceph/ceph-ci v16.2.4^{} -> 3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2024-02-19T21:51:35.176 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2024-02-19T21:51:35.179 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '81bd20d634209c7cb82c18be12b4b5a05643ebf1', 'tag': 'v16.2.4', 'wait_for_package': False} 2024-02-19T21:51:35.180 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-02-19T21:51:35.180 INFO:teuthology.packaging:ref: None 2024-02-19T21:51:35.180 INFO:teuthology.packaging:tag: v16.2.4 2024-02-19T21:51:35.180 INFO:teuthology.packaging:branch: None 2024-02-19T21:51:35.180 INFO:teuthology.packaging:sha1: 81bd20d634209c7cb82c18be12b4b5a05643ebf1 2024-02-19T21:51:35.180 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2024-02-19T21:51:35.337 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/ 2024-02-19T21:51:35.337 INFO:teuthology.task.install.rpm:Package version is 16.2.4-0 2024-02-19T21:51:35.369 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-02-19T21:51:35.369 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-02-19T21:51:35.369 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-02-19T21:51:35.402 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-02-19T21:51:35.402 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-02-19T21:51:35.402 INFO:teuthology.packaging:ref: None 2024-02-19T21:51:35.402 INFO:teuthology.packaging:tag: v16.2.4 2024-02-19T21:51:35.403 INFO:teuthology.packaging:branch: None 2024-02-19T21:51:35.403 INFO:teuthology.packaging:sha1: 81bd20d634209c7cb82c18be12b4b5a05643ebf1 2024-02-19T21:51:35.403 DEBUG:teuthology.orchestra.run.smithi086:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/v16.2.4/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-02-19T21:51:35.473 DEBUG:teuthology.orchestra.run.smithi086:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-02-19T21:51:35.487 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/ 2024-02-19T21:51:35.487 INFO:teuthology.task.install.rpm:Package version is 16.2.4-0 2024-02-19T21:51:35.512 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-02-19T21:51:35.512 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-02-19T21:51:35.512 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-02-19T21:51:35.544 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-02-19T21:51:35.544 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-02-19T21:51:35.544 INFO:teuthology.packaging:ref: None 2024-02-19T21:51:35.544 INFO:teuthology.packaging:tag: v16.2.4 2024-02-19T21:51:35.545 INFO:teuthology.packaging:branch: None 2024-02-19T21:51:35.545 INFO:teuthology.packaging:sha1: 81bd20d634209c7cb82c18be12b4b5a05643ebf1 2024-02-19T21:51:35.545 DEBUG:teuthology.orchestra.run.smithi053:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/v16.2.4/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-02-19T21:51:35.553 DEBUG:teuthology.orchestra.run.smithi086:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-02-19T21:51:35.615 DEBUG:teuthology.orchestra.run.smithi053:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-02-19T21:51:35.621 INFO:teuthology.orchestra.run.smithi086.stdout:check_obsoletes = 1 2024-02-19T21:51:35.622 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum clean all 2024-02-19T21:51:35.697 DEBUG:teuthology.orchestra.run.smithi053:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-02-19T21:51:35.765 INFO:teuthology.orchestra.run.smithi053.stdout:check_obsoletes = 1 2024-02-19T21:51:35.767 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum clean all 2024-02-19T21:51:35.888 INFO:teuthology.orchestra.run.smithi086.stdout:76 files removed 2024-02-19T21:51:35.915 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-radosgw 2024-02-19T21:51:36.045 INFO:teuthology.orchestra.run.smithi053.stdout:76 files removed 2024-02-19T21:51:36.074 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install ceph-radosgw 2024-02-19T21:51:36.395 INFO:teuthology.orchestra.run.smithi086.stdout:ceph packages for x86_64 953 kB/s | 74 kB 00:00 2024-02-19T21:51:36.512 INFO:teuthology.orchestra.run.smithi086.stdout:ceph noarch packages 192 kB/s | 15 kB 00:00 2024-02-19T21:51:36.577 INFO:teuthology.orchestra.run.smithi053.stdout:ceph packages for x86_64 991 kB/s | 74 kB 00:00 2024-02-19T21:51:36.604 INFO:teuthology.orchestra.run.smithi086.stdout:ceph source packages 28 kB/s | 1.7 kB 00:00 2024-02-19T21:51:36.695 INFO:teuthology.orchestra.run.smithi053.stdout:ceph noarch packages 198 kB/s | 15 kB 00:00 2024-02-19T21:51:36.787 INFO:teuthology.orchestra.run.smithi053.stdout:ceph source packages 29 kB/s | 1.7 kB 00:00 2024-02-19T21:51:48.055 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS Stream 8 - AppStream 2.4 MB/s | 27 MB 00:11 2024-02-19T21:51:48.212 INFO:teuthology.orchestra.run.smithi053.stdout:CentOS Stream 8 - AppStream 2.4 MB/s | 27 MB 00:11 2024-02-19T21:51:53.605 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS Stream 8 - BaseOS 17 MB/s | 10 MB 00:00 2024-02-19T21:51:53.871 INFO:teuthology.orchestra.run.smithi053.stdout:CentOS Stream 8 - BaseOS 23 MB/s | 10 MB 00:00 2024-02-19T21:51:55.372 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS Stream 8 - Extras 142 kB/s | 18 kB 00:00 2024-02-19T21:51:55.782 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS Stream 8 - Extras common packages 23 kB/s | 6.9 kB 00:00 2024-02-19T21:51:55.905 INFO:teuthology.orchestra.run.smithi053.stdout:CentOS Stream 8 - Extras 65 kB/s | 18 kB 00:00 2024-02-19T21:51:56.347 INFO:teuthology.orchestra.run.smithi053.stdout:CentOS Stream 8 - Extras common packages 21 kB/s | 6.9 kB 00:00 2024-02-19T21:51:56.808 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS Stream 8 - PowerTools 5.2 MB/s | 4.8 MB 00:00 2024-02-19T21:51:57.473 INFO:teuthology.orchestra.run.smithi053.stdout:CentOS Stream 8 - PowerTools 4.7 MB/s | 4.8 MB 00:01 2024-02-19T21:51:57.885 INFO:teuthology.orchestra.run.smithi086.stdout:Copr repo for python3-asyncssh owned by ceph 26 kB/s | 3.5 kB 00:00 2024-02-19T21:51:58.612 INFO:teuthology.orchestra.run.smithi053.stdout:Copr repo for python3-asyncssh owned by ceph 23 kB/s | 3.5 kB 00:00 2024-02-19T21:51:59.829 INFO:teuthology.orchestra.run.smithi086.stdout:Extra Packages for Enterprise Linux 8.8 MB/s | 16 MB 00:01 2024-02-19T21:52:00.167 INFO:teuthology.orchestra.run.smithi053.stdout:Extra Packages for Enterprise Linux 11 MB/s | 16 MB 00:01 2024-02-19T21:52:03.661 INFO:teuthology.orchestra.run.smithi086.stdout:lab-extras 467 kB/s | 24 kB 00:00 2024-02-19T21:52:04.118 INFO:teuthology.orchestra.run.smithi053.stdout:lab-extras 464 kB/s | 24 kB 00:00 2024-02-19T21:52:07.182 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-02-19T21:52:07.185 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-02-19T21:52:07.185 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2024-02-19T21:52:07.185 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-02-19T21:52:07.185 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-02-19T21:52:07.185 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-radosgw x86_64 2:16.2.4-0.el8 ceph 13 M 2024-02-19T21:52:07.186 INFO:teuthology.orchestra.run.smithi086.stdout:Upgrading: 2024-02-19T21:52:07.186 INFO:teuthology.orchestra.run.smithi086.stdout: librados2 x86_64 2:16.2.4-0.el8 ceph 3.7 M 2024-02-19T21:52:07.186 INFO:teuthology.orchestra.run.smithi086.stdout: librbd1 x86_64 2:16.2.4-0.el8 ceph 4.0 M 2024-02-19T21:52:07.186 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2024-02-19T21:52:07.186 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-base x86_64 2:16.2.4-0.el8 ceph 6.0 M 2024-02-19T21:52:07.186 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-common x86_64 2:16.2.4-0.el8 ceph 24 M 2024-02-19T21:52:07.186 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-selinux x86_64 2:16.2.4-0.el8 ceph 25 k 2024-02-19T21:52:07.186 INFO:teuthology.orchestra.run.smithi086.stdout: daxctl-libs x86_64 71.1-7.el8 baseos 42 k 2024-02-19T21:52:07.186 INFO:teuthology.orchestra.run.smithi086.stdout: fmt x86_64 6.2.1-1.el8 epel 116 k 2024-02-19T21:52:07.186 INFO:teuthology.orchestra.run.smithi086.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-02-19T21:52:07.186 INFO:teuthology.orchestra.run.smithi086.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2024-02-19T21:52:07.186 INFO:teuthology.orchestra.run.smithi086.stdout: libcephfs2 x86_64 2:16.2.4-0.el8 ceph 801 k 2024-02-19T21:52:07.186 INFO:teuthology.orchestra.run.smithi086.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-02-19T21:52:07.186 INFO:teuthology.orchestra.run.smithi086.stdout: libpmemobj x86_64 1.6.1-1.el8 appstream 145 k 2024-02-19T21:52:07.187 INFO:teuthology.orchestra.run.smithi086.stdout: librabbitmq x86_64 0.9.0-5.el8 baseos 47 k 2024-02-19T21:52:07.187 INFO:teuthology.orchestra.run.smithi086.stdout: libradosstriper1 x86_64 2:16.2.4-0.el8 ceph 486 k 2024-02-19T21:52:07.187 INFO:teuthology.orchestra.run.smithi086.stdout: librdkafka x86_64 1.6.1-1.el8 appstream 677 k 2024-02-19T21:52:07.187 INFO:teuthology.orchestra.run.smithi086.stdout: librgw2 x86_64 2:16.2.4-0.el8 ceph 4.1 M 2024-02-19T21:52:07.187 INFO:teuthology.orchestra.run.smithi086.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-02-19T21:52:07.187 INFO:teuthology.orchestra.run.smithi086.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2024-02-19T21:52:07.187 INFO:teuthology.orchestra.run.smithi086.stdout: ndctl-libs x86_64 71.1-7.el8 baseos 80 k 2024-02-19T21:52:07.187 INFO:teuthology.orchestra.run.smithi086.stdout: python3-ceph-argparse x86_64 2:16.2.4-0.el8 ceph 44 k 2024-02-19T21:52:07.187 INFO:teuthology.orchestra.run.smithi086.stdout: python3-ceph-common x86_64 2:16.2.4-0.el8 ceph 80 k 2024-02-19T21:52:07.187 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cephfs x86_64 2:16.2.4-0.el8 ceph 213 k 2024-02-19T21:52:07.187 INFO:teuthology.orchestra.run.smithi086.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2024-02-19T21:52:07.187 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rados x86_64 2:16.2.4-0.el8 ceph 387 k 2024-02-19T21:52:07.187 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rbd x86_64 2:16.2.4-0.el8 ceph 368 k 2024-02-19T21:52:07.187 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rgw x86_64 2:16.2.4-0.el8 ceph 114 k 2024-02-19T21:52:07.187 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T21:52:07.187 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-02-19T21:52:07.188 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-02-19T21:52:07.188 INFO:teuthology.orchestra.run.smithi086.stdout:Install 25 Packages 2024-02-19T21:52:07.188 INFO:teuthology.orchestra.run.smithi086.stdout:Upgrade 2 Packages 2024-02-19T21:52:07.188 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T21:52:07.188 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 59 M 2024-02-19T21:52:07.188 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-02-19T21:52:07.604 INFO:teuthology.orchestra.run.smithi086.stdout:(1/27): ceph-base-16.2.4-0.el8.x86_64.rpm 42 MB/s | 6.0 MB 00:00 2024-02-19T21:52:07.637 INFO:teuthology.orchestra.run.smithi086.stdout:(2/27): ceph-selinux-16.2.4-0.el8.x86_64.rpm 745 kB/s | 25 kB 00:00 2024-02-19T21:52:07.679 INFO:teuthology.orchestra.run.smithi086.stdout:(3/27): libcephfs2-16.2.4-0.el8.x86_64.rpm 19 MB/s | 801 kB 00:00 2024-02-19T21:52:07.712 INFO:teuthology.orchestra.run.smithi086.stdout:(4/27): libradosstriper1-16.2.4-0.el8.x86_64.rp 14 MB/s | 486 kB 00:00 2024-02-19T21:52:07.805 INFO:teuthology.orchestra.run.smithi086.stdout:(5/27): librgw2-16.2.4-0.el8.x86_64.rpm 45 MB/s | 4.1 MB 00:00 2024-02-19T21:52:07.838 INFO:teuthology.orchestra.run.smithi086.stdout:(6/27): python3-ceph-argparse-16.2.4-0.el8.x86_ 1.3 MB/s | 44 kB 00:00 2024-02-19T21:52:07.864 INFO:teuthology.orchestra.run.smithi086.stdout:(7/27): python3-ceph-common-16.2.4-0.el8.x86_64 3.1 MB/s | 80 kB 00:00 2024-02-19T21:52:07.889 INFO:teuthology.orchestra.run.smithi086.stdout:(8/27): python3-cephfs-16.2.4-0.el8.x86_64.rpm 8.3 MB/s | 213 kB 00:00 2024-02-19T21:52:07.904 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-02-19T21:52:07.907 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-02-19T21:52:07.908 INFO:teuthology.orchestra.run.smithi053.stdout: Package Arch Version Repository Size 2024-02-19T21:52:07.908 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-02-19T21:52:07.908 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2024-02-19T21:52:07.908 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-radosgw x86_64 2:16.2.4-0.el8 ceph 13 M 2024-02-19T21:52:07.908 INFO:teuthology.orchestra.run.smithi053.stdout:Upgrading: 2024-02-19T21:52:07.908 INFO:teuthology.orchestra.run.smithi053.stdout: librados2 x86_64 2:16.2.4-0.el8 ceph 3.7 M 2024-02-19T21:52:07.908 INFO:teuthology.orchestra.run.smithi053.stdout: librbd1 x86_64 2:16.2.4-0.el8 ceph 4.0 M 2024-02-19T21:52:07.908 INFO:teuthology.orchestra.run.smithi053.stdout:Installing dependencies: 2024-02-19T21:52:07.908 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-base x86_64 2:16.2.4-0.el8 ceph 6.0 M 2024-02-19T21:52:07.908 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-common x86_64 2:16.2.4-0.el8 ceph 24 M 2024-02-19T21:52:07.908 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-selinux x86_64 2:16.2.4-0.el8 ceph 25 k 2024-02-19T21:52:07.908 INFO:teuthology.orchestra.run.smithi053.stdout: daxctl-libs x86_64 71.1-7.el8 baseos 42 k 2024-02-19T21:52:07.909 INFO:teuthology.orchestra.run.smithi053.stdout: fmt x86_64 6.2.1-1.el8 epel 116 k 2024-02-19T21:52:07.909 INFO:teuthology.orchestra.run.smithi053.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-02-19T21:52:07.909 INFO:teuthology.orchestra.run.smithi053.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2024-02-19T21:52:07.909 INFO:teuthology.orchestra.run.smithi053.stdout: libcephfs2 x86_64 2:16.2.4-0.el8 ceph 801 k 2024-02-19T21:52:07.909 INFO:teuthology.orchestra.run.smithi053.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-02-19T21:52:07.909 INFO:teuthology.orchestra.run.smithi053.stdout: libpmemobj x86_64 1.6.1-1.el8 appstream 145 k 2024-02-19T21:52:07.909 INFO:teuthology.orchestra.run.smithi053.stdout: librabbitmq x86_64 0.9.0-5.el8 baseos 47 k 2024-02-19T21:52:07.909 INFO:teuthology.orchestra.run.smithi053.stdout: libradosstriper1 x86_64 2:16.2.4-0.el8 ceph 486 k 2024-02-19T21:52:07.909 INFO:teuthology.orchestra.run.smithi053.stdout: librdkafka x86_64 1.6.1-1.el8 appstream 677 k 2024-02-19T21:52:07.909 INFO:teuthology.orchestra.run.smithi053.stdout: librgw2 x86_64 2:16.2.4-0.el8 ceph 4.1 M 2024-02-19T21:52:07.909 INFO:teuthology.orchestra.run.smithi053.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-02-19T21:52:07.909 INFO:teuthology.orchestra.run.smithi053.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2024-02-19T21:52:07.909 INFO:teuthology.orchestra.run.smithi053.stdout: ndctl-libs x86_64 71.1-7.el8 baseos 80 k 2024-02-19T21:52:07.910 INFO:teuthology.orchestra.run.smithi053.stdout: python3-ceph-argparse x86_64 2:16.2.4-0.el8 ceph 44 k 2024-02-19T21:52:07.910 INFO:teuthology.orchestra.run.smithi053.stdout: python3-ceph-common x86_64 2:16.2.4-0.el8 ceph 80 k 2024-02-19T21:52:07.910 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cephfs x86_64 2:16.2.4-0.el8 ceph 213 k 2024-02-19T21:52:07.910 INFO:teuthology.orchestra.run.smithi053.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2024-02-19T21:52:07.910 INFO:teuthology.orchestra.run.smithi053.stdout: python3-rados x86_64 2:16.2.4-0.el8 ceph 387 k 2024-02-19T21:52:07.910 INFO:teuthology.orchestra.run.smithi053.stdout: python3-rbd x86_64 2:16.2.4-0.el8 ceph 368 k 2024-02-19T21:52:07.910 INFO:teuthology.orchestra.run.smithi053.stdout: python3-rgw x86_64 2:16.2.4-0.el8 ceph 114 k 2024-02-19T21:52:07.910 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-02-19T21:52:07.910 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2024-02-19T21:52:07.910 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-02-19T21:52:07.910 INFO:teuthology.orchestra.run.smithi053.stdout:Install 25 Packages 2024-02-19T21:52:07.910 INFO:teuthology.orchestra.run.smithi053.stdout:Upgrade 2 Packages 2024-02-19T21:52:07.910 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-02-19T21:52:07.911 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 59 M 2024-02-19T21:52:07.911 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2024-02-19T21:52:08.031 INFO:teuthology.orchestra.run.smithi086.stdout:(9/27): ceph-radosgw-16.2.4-0.el8.x86_64.rpm 22 MB/s | 13 MB 00:00 2024-02-19T21:52:08.056 INFO:teuthology.orchestra.run.smithi086.stdout:(10/27): python3-rados-16.2.4-0.el8.x86_64.rpm 2.3 MB/s | 387 kB 00:00 2024-02-19T21:52:08.073 INFO:teuthology.orchestra.run.smithi086.stdout:(11/27): python3-rgw-16.2.4-0.el8.x86_64.rpm 6.7 MB/s | 114 kB 00:00 2024-02-19T21:52:08.098 INFO:teuthology.orchestra.run.smithi086.stdout:(12/27): python3-rbd-16.2.4-0.el8.x86_64.rpm 5.4 MB/s | 368 kB 00:00 2024-02-19T21:52:08.358 INFO:teuthology.orchestra.run.smithi086.stdout:(13/27): ceph-common-16.2.4-0.el8.x86_64.rpm 27 MB/s | 24 MB 00:00 2024-02-19T21:52:08.383 INFO:teuthology.orchestra.run.smithi086.stdout:(14/27): libpmemobj-1.6.1-1.el8.x86_64.rpm 470 kB/s | 145 kB 00:00 2024-02-19T21:52:08.483 INFO:teuthology.orchestra.run.smithi086.stdout:(15/27): python3-prettytable-0.7.2-14.el8.noarc 439 kB/s | 44 kB 00:00 2024-02-19T21:52:08.533 INFO:teuthology.orchestra.run.smithi053.stdout:(1/27): ceph-base-16.2.4-0.el8.x86_64.rpm 25 MB/s | 6.0 MB 00:00 2024-02-19T21:52:08.551 INFO:teuthology.orchestra.run.smithi086.stdout:(16/27): daxctl-libs-71.1-7.el8.x86_64.rpm 634 kB/s | 42 kB 00:00 2024-02-19T21:52:08.562 INFO:teuthology.orchestra.run.smithi053.stdout:(2/27): ceph-selinux-16.2.4-0.el8.x86_64.rpm 877 kB/s | 25 kB 00:00 2024-02-19T21:52:08.575 INFO:teuthology.orchestra.run.smithi086.stdout:(17/27): lttng-ust-2.8.1-11.el8.x86_64.rpm 1.2 MB/s | 259 kB 00:00 2024-02-19T21:52:08.600 INFO:teuthology.orchestra.run.smithi086.stdout:(18/27): librabbitmq-0.9.0-5.el8.x86_64.rpm 938 kB/s | 47 kB 00:00 2024-02-19T21:52:08.604 INFO:teuthology.orchestra.run.smithi053.stdout:(3/27): libcephfs2-16.2.4-0.el8.x86_64.rpm 19 MB/s | 801 kB 00:00 2024-02-19T21:52:08.634 INFO:teuthology.orchestra.run.smithi086.stdout:(19/27): librdkafka-1.6.1-1.el8.x86_64.rpm 1.2 MB/s | 677 kB 00:00 2024-02-19T21:52:08.637 INFO:teuthology.orchestra.run.smithi053.stdout:(4/27): libradosstriper1-16.2.4-0.el8.x86_64.rp 14 MB/s | 486 kB 00:00 2024-02-19T21:52:08.659 INFO:teuthology.orchestra.run.smithi086.stdout:(20/27): ndctl-libs-71.1-7.el8.x86_64.rpm 956 kB/s | 80 kB 00:00 2024-02-19T21:52:08.738 INFO:teuthology.orchestra.run.smithi053.stdout:(5/27): librgw2-16.2.4-0.el8.x86_64.rpm 41 MB/s | 4.1 MB 00:00 2024-02-19T21:52:08.763 INFO:teuthology.orchestra.run.smithi053.stdout:(6/27): python3-ceph-argparse-16.2.4-0.el8.x86_ 1.7 MB/s | 44 kB 00:00 2024-02-19T21:52:08.788 INFO:teuthology.orchestra.run.smithi053.stdout:(7/27): python3-ceph-common-16.2.4-0.el8.x86_64 3.1 MB/s | 80 kB 00:00 2024-02-19T21:52:08.813 INFO:teuthology.orchestra.run.smithi053.stdout:(8/27): python3-cephfs-16.2.4-0.el8.x86_64.rpm 8.4 MB/s | 213 kB 00:00 2024-02-19T21:52:08.839 INFO:teuthology.orchestra.run.smithi053.stdout:(9/27): python3-rados-16.2.4-0.el8.x86_64.rpm 15 MB/s | 387 kB 00:00 2024-02-19T21:52:08.864 INFO:teuthology.orchestra.run.smithi053.stdout:(10/27): python3-rbd-16.2.4-0.el8.x86_64.rpm 14 MB/s | 368 kB 00:00 2024-02-19T21:52:08.889 INFO:teuthology.orchestra.run.smithi053.stdout:(11/27): python3-rgw-16.2.4-0.el8.x86_64.rpm 4.5 MB/s | 114 kB 00:00 2024-02-19T21:52:09.010 INFO:teuthology.orchestra.run.smithi086.stdout:(21/27): fmt-6.2.1-1.el8.x86_64.rpm 284 kB/s | 116 kB 00:00 2024-02-19T21:52:09.023 INFO:teuthology.orchestra.run.smithi053.stdout:(12/27): ceph-radosgw-16.2.4-0.el8.x86_64.rpm 17 MB/s | 13 MB 00:00 2024-02-19T21:52:09.068 INFO:teuthology.orchestra.run.smithi086.stdout:(22/27): leveldb-1.22-1.el8.x86_64.rpm 443 kB/s | 181 kB 00:00 2024-02-19T21:52:09.093 INFO:teuthology.orchestra.run.smithi086.stdout:(23/27): gperftools-libs-2.7-9.el8.x86_64.rpm 667 kB/s | 306 kB 00:00 2024-02-19T21:52:09.110 INFO:teuthology.orchestra.run.smithi086.stdout:(24/27): liboath-2.6.2-3.el8.x86_64.rpm 585 kB/s | 59 kB 00:00 2024-02-19T21:52:09.194 INFO:teuthology.orchestra.run.smithi086.stdout:(25/27): librados2-16.2.4-0.el8.x86_64.rpm 37 MB/s | 3.7 MB 00:00 2024-02-19T21:52:09.211 INFO:teuthology.orchestra.run.smithi086.stdout:(26/27): libunwind-1.3.1-3.el8.x86_64.rpm 526 kB/s | 75 kB 00:00 2024-02-19T21:52:09.282 INFO:teuthology.orchestra.run.smithi053.stdout:(13/27): ceph-common-16.2.4-0.el8.x86_64.rpm 24 MB/s | 24 MB 00:00 2024-02-19T21:52:09.286 INFO:teuthology.orchestra.run.smithi086.stdout:(27/27): librbd1-16.2.4-0.el8.x86_64.rpm 23 MB/s | 4.0 MB 00:00 2024-02-19T21:52:09.288 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-02-19T21:52:09.288 INFO:teuthology.orchestra.run.smithi086.stdout:Total 28 MB/s | 59 MB 00:02 2024-02-19T21:52:09.299 INFO:teuthology.orchestra.run.smithi053.stdout:(14/27): libpmemobj-1.6.1-1.el8.x86_64.rpm 355 kB/s | 145 kB 00:00 2024-02-19T21:52:09.355 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-02-19T21:52:09.357 INFO:teuthology.orchestra.run.smithi053.stdout:(15/27): python3-prettytable-0.7.2-14.el8.noarc 755 kB/s | 44 kB 00:00 2024-02-19T21:52:09.400 INFO:teuthology.orchestra.run.smithi053.stdout:(16/27): librdkafka-1.6.1-1.el8.x86_64.rpm 1.8 MB/s | 677 kB 00:00 2024-02-19T21:52:09.403 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-02-19T21:52:09.403 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-02-19T21:52:09.417 INFO:teuthology.orchestra.run.smithi053.stdout:(17/27): daxctl-libs-71.1-7.el8.x86_64.rpm 718 kB/s | 42 kB 00:00 2024-02-19T21:52:09.450 INFO:teuthology.orchestra.run.smithi053.stdout:(18/27): ndctl-libs-71.1-7.el8.x86_64.rpm 2.4 MB/s | 80 kB 00:00 2024-02-19T21:52:09.501 INFO:teuthology.orchestra.run.smithi053.stdout:(19/27): lttng-ust-2.8.1-11.el8.x86_64.rpm 1.2 MB/s | 259 kB 00:00 2024-02-19T21:52:09.517 INFO:teuthology.orchestra.run.smithi053.stdout:(20/27): librabbitmq-0.9.0-5.el8.x86_64.rpm 400 kB/s | 47 kB 00:00 2024-02-19T21:52:09.742 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-02-19T21:52:09.745 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-02-19T21:52:09.993 INFO:teuthology.orchestra.run.smithi053.stdout:(21/27): fmt-6.2.1-1.el8.x86_64.rpm 214 kB/s | 116 kB 00:00 2024-02-19T21:52:10.035 INFO:teuthology.orchestra.run.smithi053.stdout:(22/27): leveldb-1.22-1.el8.x86_64.rpm 350 kB/s | 181 kB 00:00 2024-02-19T21:52:10.102 INFO:teuthology.orchestra.run.smithi053.stdout:(23/27): gperftools-libs-2.7-9.el8.x86_64.rpm 510 kB/s | 306 kB 00:00 2024-02-19T21:52:10.127 INFO:teuthology.orchestra.run.smithi053.stdout:(24/27): liboath-2.6.2-3.el8.x86_64.rpm 439 kB/s | 59 kB 00:00 2024-02-19T21:52:10.153 INFO:teuthology.orchestra.run.smithi053.stdout:(25/27): libunwind-1.3.1-3.el8.x86_64.rpm 638 kB/s | 75 kB 00:00 2024-02-19T21:52:10.236 INFO:teuthology.orchestra.run.smithi053.stdout:(26/27): librados2-16.2.4-0.el8.x86_64.rpm 28 MB/s | 3.7 MB 00:00 2024-02-19T21:52:10.303 INFO:teuthology.orchestra.run.smithi053.stdout:(27/27): librbd1-16.2.4-0.el8.x86_64.rpm 23 MB/s | 4.0 MB 00:00 2024-02-19T21:52:10.307 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2024-02-19T21:52:10.308 INFO:teuthology.orchestra.run.smithi053.stdout:Total 25 MB/s | 59 MB 00:02 2024-02-19T21:52:10.377 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2024-02-19T21:52:10.430 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2024-02-19T21:52:10.430 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2024-02-19T21:52:10.794 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2024-02-19T21:52:10.799 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2024-02-19T21:52:10.863 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-02-19T21:52:11.095 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : fmt-6.2.1-1.el8.x86_64 1/29 2024-02-19T21:52:11.274 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/29 2024-02-19T21:52:11.425 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/29 2024-02-19T21:52:11.828 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/29 2024-02-19T21:52:11.944 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2024-02-19T21:52:12.000 INFO:teuthology.orchestra.run.smithi086.stdout: Upgrading : librados2-2:16.2.4-0.el8.x86_64 4/29 2024-02-19T21:52:12.096 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: librados2-2:16.2.4-0.el8.x86_64 4/29 2024-02-19T21:52:12.148 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : fmt-6.2.1-1.el8.x86_64 1/29 2024-02-19T21:52:12.331 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/29 2024-02-19T21:52:12.349 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-rados-2:16.2.4-0.el8.x86_64 5/29 2024-02-19T21:52:12.510 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/29 2024-02-19T21:52:12.525 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2024-02-19T21:52:12.591 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2024-02-19T21:52:12.809 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : librabbitmq-0.9.0-5.el8.x86_64 7/29 2024-02-19T21:52:12.926 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/29 2024-02-19T21:52:13.093 INFO:teuthology.orchestra.run.smithi053.stdout: Upgrading : librados2-2:16.2.4-0.el8.x86_64 4/29 2024-02-19T21:52:13.180 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: librados2-2:16.2.4-0.el8.x86_64 4/29 2024-02-19T21:52:13.351 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : librdkafka-1.6.1-1.el8.x86_64 8/29 2024-02-19T21:52:13.414 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-rados-2:16.2.4-0.el8.x86_64 5/29 2024-02-19T21:52:13.517 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : librgw2-2:16.2.4-0.el8.x86_64 9/29 2024-02-19T21:52:13.576 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2024-02-19T21:52:13.606 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: librgw2-2:16.2.4-0.el8.x86_64 9/29 2024-02-19T21:52:13.616 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2024-02-19T21:52:13.832 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : librabbitmq-0.9.0-5.el8.x86_64 7/29 2024-02-19T21:52:13.854 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/29 2024-02-19T21:52:14.008 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : daxctl-libs-71.1-7.el8.x86_64 11/29 2024-02-19T21:52:14.071 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: daxctl-libs-71.1-7.el8.x86_64 11/29 2024-02-19T21:52:14.231 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 12/29 2024-02-19T21:52:14.392 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : librdkafka-1.6.1-1.el8.x86_64 8/29 2024-02-19T21:52:14.401 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-cephfs-2:16.2.4-0.el8.x86_64 13/29 2024-02-19T21:52:14.527 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : librgw2-2:16.2.4-0.el8.x86_64 9/29 2024-02-19T21:52:14.600 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ndctl-libs-71.1-7.el8.x86_64 14/29 2024-02-19T21:52:14.615 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: librgw2-2:16.2.4-0.el8.x86_64 9/29 2024-02-19T21:52:14.673 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ndctl-libs-71.1-7.el8.x86_64 14/29 2024-02-19T21:52:14.798 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/29 2024-02-19T21:52:14.808 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libpmemobj-1.6.1-1.el8.x86_64 15/29 2024-02-19T21:52:14.976 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : daxctl-libs-71.1-7.el8.x86_64 11/29 2024-02-19T21:52:15.047 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: daxctl-libs-71.1-7.el8.x86_64 11/29 2024-02-19T21:52:15.200 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 12/29 2024-02-19T21:52:15.239 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: libpmemobj-1.6.1-1.el8.x86_64 15/29 2024-02-19T21:52:15.370 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-cephfs-2:16.2.4-0.el8.x86_64 13/29 2024-02-19T21:52:15.432 INFO:teuthology.orchestra.run.smithi086.stdout: Upgrading : librbd1-2:16.2.4-0.el8.x86_64 16/29 2024-02-19T21:52:15.530 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: librbd1-2:16.2.4-0.el8.x86_64 16/29 2024-02-19T21:52:15.534 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ndctl-libs-71.1-7.el8.x86_64 14/29 2024-02-19T21:52:15.608 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ndctl-libs-71.1-7.el8.x86_64 14/29 2024-02-19T21:52:15.720 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-rbd-2:16.2.4-0.el8.x86_64 17/29 2024-02-19T21:52:15.768 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libpmemobj-1.6.1-1.el8.x86_64 15/29 2024-02-19T21:52:15.951 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-rgw-2:16.2.4-0.el8.x86_64 18/29 2024-02-19T21:52:16.091 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2024-02-19T21:52:16.157 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2024-02-19T21:52:16.233 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: libpmemobj-1.6.1-1.el8.x86_64 15/29 2024-02-19T21:52:16.349 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 20/29 2024-02-19T21:52:16.410 INFO:teuthology.orchestra.run.smithi053.stdout: Upgrading : librbd1-2:16.2.4-0.el8.x86_64 16/29 2024-02-19T21:52:16.513 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: librbd1-2:16.2.4-0.el8.x86_64 16/29 2024-02-19T21:52:16.555 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 21/29 2024-02-19T21:52:16.680 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-rbd-2:16.2.4-0.el8.x86_64 17/29 2024-02-19T21:52:16.748 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 22/29 2024-02-19T21:52:16.881 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-rgw-2:16.2.4-0.el8.x86_64 18/29 2024-02-19T21:52:16.974 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-ceph-common-2:16.2.4-0.el8.x86_64 23/29 2024-02-19T21:52:17.018 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2024-02-19T21:52:17.160 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2024-02-19T21:52:17.338 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 20/29 2024-02-19T21:52:17.532 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 21/29 2024-02-19T21:52:17.706 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 22/29 2024-02-19T21:52:17.893 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-ceph-common-2:16.2.4-0.el8.x86_64 23/29 2024-02-19T21:52:19.539 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-02-19T21:52:19.758 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-02-19T21:52:20.455 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-02-19T21:52:20.595 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-02-19T21:52:20.734 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-base-2:16.2.4-0.el8.x86_64 25/29 2024-02-19T21:52:20.801 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-02-19T21:52:20.850 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-base-2:16.2.4-0.el8.x86_64 25/29 2024-02-19T21:52:21.007 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2024-02-19T21:52:21.554 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-02-19T21:52:21.831 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-base-2:16.2.4-0.el8.x86_64 25/29 2024-02-19T21:52:21.929 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-base-2:16.2.4-0.el8.x86_64 25/29 2024-02-19T21:52:22.084 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2024-02-19T21:52:40.050 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2024-02-19T21:52:40.050 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /sys 2024-02-19T21:52:40.050 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /proc 2024-02-19T21:52:40.050 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /dev 2024-02-19T21:52:40.050 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /run 2024-02-19T21:52:40.050 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /mnt 2024-02-19T21:52:40.051 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /var/tmp 2024-02-19T21:52:40.051 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /home 2024-02-19T21:52:40.051 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /tmp 2024-02-19T21:52:40.051 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /dev 2024-02-19T21:52:40.051 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T21:52:41.406 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2024-02-19T21:52:41.487 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2024-02-19T21:52:41.487 INFO:teuthology.orchestra.run.smithi086.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 28/29 2024-02-19T21:52:41.690 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 28/29 2024-02-19T21:52:41.690 INFO:teuthology.orchestra.run.smithi086.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 29/29 2024-02-19T21:52:41.886 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2024-02-19T21:52:41.886 INFO:teuthology.orchestra.run.smithi053.stdout:skipping the directory /sys 2024-02-19T21:52:41.886 INFO:teuthology.orchestra.run.smithi053.stdout:skipping the directory /proc 2024-02-19T21:52:41.886 INFO:teuthology.orchestra.run.smithi053.stdout:skipping the directory /dev 2024-02-19T21:52:41.886 INFO:teuthology.orchestra.run.smithi053.stdout:skipping the directory /run 2024-02-19T21:52:41.886 INFO:teuthology.orchestra.run.smithi053.stdout:skipping the directory /mnt 2024-02-19T21:52:41.886 INFO:teuthology.orchestra.run.smithi053.stdout:skipping the directory /var/tmp 2024-02-19T21:52:41.886 INFO:teuthology.orchestra.run.smithi053.stdout:skipping the directory /home 2024-02-19T21:52:41.886 INFO:teuthology.orchestra.run.smithi053.stdout:skipping the directory /tmp 2024-02-19T21:52:41.886 INFO:teuthology.orchestra.run.smithi053.stdout:skipping the directory /dev 2024-02-19T21:52:41.886 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-02-19T21:52:42.397 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 29/29 2024-02-19T21:52:42.398 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-base-2:16.2.4-0.el8.x86_64 1/29 2024-02-19T21:52:42.398 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-common-2:16.2.4-0.el8.x86_64 2/29 2024-02-19T21:52:42.398 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-radosgw-2:16.2.4-0.el8.x86_64 3/29 2024-02-19T21:52:42.398 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-selinux-2:16.2.4-0.el8.x86_64 4/29 2024-02-19T21:52:42.398 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libcephfs2-2:16.2.4-0.el8.x86_64 5/29 2024-02-19T21:52:42.398 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libradosstriper1-2:16.2.4-0.el8.x86_64 6/29 2024-02-19T21:52:42.398 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librgw2-2:16.2.4-0.el8.x86_64 7/29 2024-02-19T21:52:42.398 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 8/29 2024-02-19T21:52:42.398 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-ceph-common-2:16.2.4-0.el8.x86_64 9/29 2024-02-19T21:52:42.398 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-cephfs-2:16.2.4-0.el8.x86_64 10/29 2024-02-19T21:52:42.398 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-rados-2:16.2.4-0.el8.x86_64 11/29 2024-02-19T21:52:42.398 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-rbd-2:16.2.4-0.el8.x86_64 12/29 2024-02-19T21:52:42.399 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-rgw-2:16.2.4-0.el8.x86_64 13/29 2024-02-19T21:52:42.399 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libpmemobj-1.6.1-1.el8.x86_64 14/29 2024-02-19T21:52:42.399 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librdkafka-1.6.1-1.el8.x86_64 15/29 2024-02-19T21:52:42.399 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 16/29 2024-02-19T21:52:42.399 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 17/29 2024-02-19T21:52:42.399 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : daxctl-libs-71.1-7.el8.x86_64 18/29 2024-02-19T21:52:42.399 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librabbitmq-0.9.0-5.el8.x86_64 19/29 2024-02-19T21:52:42.399 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ndctl-libs-71.1-7.el8.x86_64 20/29 2024-02-19T21:52:42.399 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : fmt-6.2.1-1.el8.x86_64 21/29 2024-02-19T21:52:42.399 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 22/29 2024-02-19T21:52:42.399 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : leveldb-1.22-1.el8.x86_64 23/29 2024-02-19T21:52:42.399 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 24/29 2024-02-19T21:52:42.399 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 25/29 2024-02-19T21:52:42.399 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librados2-2:16.2.4-0.el8.x86_64 26/29 2024-02-19T21:52:42.400 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 27/29 2024-02-19T21:52:42.400 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librbd1-2:16.2.4-0.el8.x86_64 28/29 2024-02-19T21:52:43.232 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 29/29 2024-02-19T21:52:43.232 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T21:52:43.232 INFO:teuthology.orchestra.run.smithi086.stdout:Upgraded: 2024-02-19T21:52:43.232 INFO:teuthology.orchestra.run.smithi086.stdout: librados2-2:16.2.4-0.el8.x86_64 librbd1-2:16.2.4-0.el8.x86_64 2024-02-19T21:52:43.233 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-02-19T21:52:43.233 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-base-2:16.2.4-0.el8.x86_64 2024-02-19T21:52:43.233 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-common-2:16.2.4-0.el8.x86_64 2024-02-19T21:52:43.233 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-radosgw-2:16.2.4-0.el8.x86_64 2024-02-19T21:52:43.233 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-selinux-2:16.2.4-0.el8.x86_64 2024-02-19T21:52:43.233 INFO:teuthology.orchestra.run.smithi086.stdout: daxctl-libs-71.1-7.el8.x86_64 2024-02-19T21:52:43.233 INFO:teuthology.orchestra.run.smithi086.stdout: fmt-6.2.1-1.el8.x86_64 2024-02-19T21:52:43.233 INFO:teuthology.orchestra.run.smithi086.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-02-19T21:52:43.233 INFO:teuthology.orchestra.run.smithi086.stdout: leveldb-1.22-1.el8.x86_64 2024-02-19T21:52:43.233 INFO:teuthology.orchestra.run.smithi086.stdout: libcephfs2-2:16.2.4-0.el8.x86_64 2024-02-19T21:52:43.233 INFO:teuthology.orchestra.run.smithi086.stdout: liboath-2.6.2-3.el8.x86_64 2024-02-19T21:52:43.233 INFO:teuthology.orchestra.run.smithi086.stdout: libpmemobj-1.6.1-1.el8.x86_64 2024-02-19T21:52:43.233 INFO:teuthology.orchestra.run.smithi086.stdout: librabbitmq-0.9.0-5.el8.x86_64 2024-02-19T21:52:43.233 INFO:teuthology.orchestra.run.smithi086.stdout: libradosstriper1-2:16.2.4-0.el8.x86_64 2024-02-19T21:52:43.234 INFO:teuthology.orchestra.run.smithi086.stdout: librdkafka-1.6.1-1.el8.x86_64 2024-02-19T21:52:43.234 INFO:teuthology.orchestra.run.smithi086.stdout: librgw2-2:16.2.4-0.el8.x86_64 2024-02-19T21:52:43.234 INFO:teuthology.orchestra.run.smithi086.stdout: libunwind-1.3.1-3.el8.x86_64 2024-02-19T21:52:43.234 INFO:teuthology.orchestra.run.smithi086.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-02-19T21:52:43.234 INFO:teuthology.orchestra.run.smithi086.stdout: ndctl-libs-71.1-7.el8.x86_64 2024-02-19T21:52:43.234 INFO:teuthology.orchestra.run.smithi086.stdout: python3-ceph-argparse-2:16.2.4-0.el8.x86_64 2024-02-19T21:52:43.234 INFO:teuthology.orchestra.run.smithi086.stdout: python3-ceph-common-2:16.2.4-0.el8.x86_64 2024-02-19T21:52:43.234 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cephfs-2:16.2.4-0.el8.x86_64 2024-02-19T21:52:43.234 INFO:teuthology.orchestra.run.smithi086.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-02-19T21:52:43.234 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rados-2:16.2.4-0.el8.x86_64 2024-02-19T21:52:43.234 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rbd-2:16.2.4-0.el8.x86_64 2024-02-19T21:52:43.234 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rgw-2:16.2.4-0.el8.x86_64 2024-02-19T21:52:43.234 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T21:52:43.234 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-02-19T21:52:43.311 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2024-02-19T21:52:43.391 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2024-02-19T21:52:43.391 INFO:teuthology.orchestra.run.smithi053.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 28/29 2024-02-19T21:52:43.519 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-test 2024-02-19T21:52:43.583 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 28/29 2024-02-19T21:52:43.583 INFO:teuthology.orchestra.run.smithi053.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 29/29 2024-02-19T21:52:44.249 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 29/29 2024-02-19T21:52:44.249 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-base-2:16.2.4-0.el8.x86_64 1/29 2024-02-19T21:52:44.250 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-common-2:16.2.4-0.el8.x86_64 2/29 2024-02-19T21:52:44.250 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-radosgw-2:16.2.4-0.el8.x86_64 3/29 2024-02-19T21:52:44.250 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-selinux-2:16.2.4-0.el8.x86_64 4/29 2024-02-19T21:52:44.250 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libcephfs2-2:16.2.4-0.el8.x86_64 5/29 2024-02-19T21:52:44.250 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libradosstriper1-2:16.2.4-0.el8.x86_64 6/29 2024-02-19T21:52:44.250 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : librgw2-2:16.2.4-0.el8.x86_64 7/29 2024-02-19T21:52:44.250 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 8/29 2024-02-19T21:52:44.250 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-ceph-common-2:16.2.4-0.el8.x86_64 9/29 2024-02-19T21:52:44.250 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-cephfs-2:16.2.4-0.el8.x86_64 10/29 2024-02-19T21:52:44.250 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-rados-2:16.2.4-0.el8.x86_64 11/29 2024-02-19T21:52:44.250 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-rbd-2:16.2.4-0.el8.x86_64 12/29 2024-02-19T21:52:44.250 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-rgw-2:16.2.4-0.el8.x86_64 13/29 2024-02-19T21:52:44.250 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libpmemobj-1.6.1-1.el8.x86_64 14/29 2024-02-19T21:52:44.251 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : librdkafka-1.6.1-1.el8.x86_64 15/29 2024-02-19T21:52:44.251 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 16/29 2024-02-19T21:52:44.251 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 17/29 2024-02-19T21:52:44.251 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : daxctl-libs-71.1-7.el8.x86_64 18/29 2024-02-19T21:52:44.251 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : librabbitmq-0.9.0-5.el8.x86_64 19/29 2024-02-19T21:52:44.251 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ndctl-libs-71.1-7.el8.x86_64 20/29 2024-02-19T21:52:44.251 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : fmt-6.2.1-1.el8.x86_64 21/29 2024-02-19T21:52:44.251 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 22/29 2024-02-19T21:52:44.251 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : leveldb-1.22-1.el8.x86_64 23/29 2024-02-19T21:52:44.251 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 24/29 2024-02-19T21:52:44.251 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 25/29 2024-02-19T21:52:44.251 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : librados2-2:16.2.4-0.el8.x86_64 26/29 2024-02-19T21:52:44.251 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 27/29 2024-02-19T21:52:44.251 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : librbd1-2:16.2.4-0.el8.x86_64 28/29 2024-02-19T21:52:44.272 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:00:41 ago on Mon 19 Feb 2024 09:52:03 PM UTC. 2024-02-19T21:52:45.255 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 29/29 2024-02-19T21:52:45.255 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-02-19T21:52:45.255 INFO:teuthology.orchestra.run.smithi053.stdout:Upgraded: 2024-02-19T21:52:45.255 INFO:teuthology.orchestra.run.smithi053.stdout: librados2-2:16.2.4-0.el8.x86_64 librbd1-2:16.2.4-0.el8.x86_64 2024-02-19T21:52:45.255 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2024-02-19T21:52:45.255 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-base-2:16.2.4-0.el8.x86_64 2024-02-19T21:52:45.255 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-common-2:16.2.4-0.el8.x86_64 2024-02-19T21:52:45.255 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-radosgw-2:16.2.4-0.el8.x86_64 2024-02-19T21:52:45.255 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-selinux-2:16.2.4-0.el8.x86_64 2024-02-19T21:52:45.255 INFO:teuthology.orchestra.run.smithi053.stdout: daxctl-libs-71.1-7.el8.x86_64 2024-02-19T21:52:45.256 INFO:teuthology.orchestra.run.smithi053.stdout: fmt-6.2.1-1.el8.x86_64 2024-02-19T21:52:45.256 INFO:teuthology.orchestra.run.smithi053.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-02-19T21:52:45.256 INFO:teuthology.orchestra.run.smithi053.stdout: leveldb-1.22-1.el8.x86_64 2024-02-19T21:52:45.256 INFO:teuthology.orchestra.run.smithi053.stdout: libcephfs2-2:16.2.4-0.el8.x86_64 2024-02-19T21:52:45.256 INFO:teuthology.orchestra.run.smithi053.stdout: liboath-2.6.2-3.el8.x86_64 2024-02-19T21:52:45.256 INFO:teuthology.orchestra.run.smithi053.stdout: libpmemobj-1.6.1-1.el8.x86_64 2024-02-19T21:52:45.256 INFO:teuthology.orchestra.run.smithi053.stdout: librabbitmq-0.9.0-5.el8.x86_64 2024-02-19T21:52:45.256 INFO:teuthology.orchestra.run.smithi053.stdout: libradosstriper1-2:16.2.4-0.el8.x86_64 2024-02-19T21:52:45.256 INFO:teuthology.orchestra.run.smithi053.stdout: librdkafka-1.6.1-1.el8.x86_64 2024-02-19T21:52:45.256 INFO:teuthology.orchestra.run.smithi053.stdout: librgw2-2:16.2.4-0.el8.x86_64 2024-02-19T21:52:45.256 INFO:teuthology.orchestra.run.smithi053.stdout: libunwind-1.3.1-3.el8.x86_64 2024-02-19T21:52:45.256 INFO:teuthology.orchestra.run.smithi053.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-02-19T21:52:45.256 INFO:teuthology.orchestra.run.smithi053.stdout: ndctl-libs-71.1-7.el8.x86_64 2024-02-19T21:52:45.256 INFO:teuthology.orchestra.run.smithi053.stdout: python3-ceph-argparse-2:16.2.4-0.el8.x86_64 2024-02-19T21:52:45.257 INFO:teuthology.orchestra.run.smithi053.stdout: python3-ceph-common-2:16.2.4-0.el8.x86_64 2024-02-19T21:52:45.257 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cephfs-2:16.2.4-0.el8.x86_64 2024-02-19T21:52:45.257 INFO:teuthology.orchestra.run.smithi053.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-02-19T21:52:45.257 INFO:teuthology.orchestra.run.smithi053.stdout: python3-rados-2:16.2.4-0.el8.x86_64 2024-02-19T21:52:45.257 INFO:teuthology.orchestra.run.smithi053.stdout: python3-rbd-2:16.2.4-0.el8.x86_64 2024-02-19T21:52:45.257 INFO:teuthology.orchestra.run.smithi053.stdout: python3-rgw-2:16.2.4-0.el8.x86_64 2024-02-19T21:52:45.257 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-02-19T21:52:45.257 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-02-19T21:52:45.567 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install ceph-test 2024-02-19T21:52:45.829 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-02-19T21:52:45.830 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-02-19T21:52:45.830 INFO:teuthology.orchestra.run.smithi086.stdout: Package Architecture Version Repository Size 2024-02-19T21:52:45.830 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-02-19T21:52:45.830 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-02-19T21:52:45.830 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-test x86_64 2:16.2.4-0.el8 ceph 52 M 2024-02-19T21:52:45.830 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2024-02-19T21:52:45.830 INFO:teuthology.orchestra.run.smithi086.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2024-02-19T21:52:45.830 INFO:teuthology.orchestra.run.smithi086.stdout: libcephsqlite x86_64 2:16.2.4-0.el8 ceph 169 k 2024-02-19T21:52:45.831 INFO:teuthology.orchestra.run.smithi086.stdout: oniguruma x86_64 6.8.2-3.el8 appstream 188 k 2024-02-19T21:52:45.831 INFO:teuthology.orchestra.run.smithi086.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2024-02-19T21:52:45.831 INFO:teuthology.orchestra.run.smithi086.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2024-02-19T21:52:45.831 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T21:52:45.831 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-02-19T21:52:45.831 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-02-19T21:52:45.831 INFO:teuthology.orchestra.run.smithi086.stdout:Install 6 Packages 2024-02-19T21:52:45.831 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T21:52:45.832 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 53 M 2024-02-19T21:52:45.832 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 229 M 2024-02-19T21:52:45.832 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-02-19T21:52:45.953 INFO:teuthology.orchestra.run.smithi086.stdout:(1/6): libcephsqlite-16.2.4-0.el8.x86_64.rpm 3.5 MB/s | 169 kB 00:00 2024-02-19T21:52:46.053 INFO:teuthology.orchestra.run.smithi086.stdout:(2/6): jq-1.6-8.el8.x86_64.rpm 1.3 MB/s | 203 kB 00:00 2024-02-19T21:52:46.103 INFO:teuthology.orchestra.run.smithi086.stdout:(3/6): oniguruma-6.8.2-3.el8.x86_64.rpm 1.2 MB/s | 188 kB 00:00 2024-02-19T21:52:46.170 INFO:teuthology.orchestra.run.smithi086.stdout:(4/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 1.0 MB/s | 71 kB 00:00 2024-02-19T21:52:46.213 INFO:teuthology.orchestra.run.smithi086.stdout:(5/6): socat-1.7.4.1-1.el8.x86_64.rpm 2.0 MB/s | 323 kB 00:00 2024-02-19T21:52:46.359 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:00:42 ago on Mon 19 Feb 2024 09:52:04 PM UTC. 2024-02-19T21:52:46.688 INFO:teuthology.orchestra.run.smithi086.stdout:(6/6): ceph-test-16.2.4-0.el8.x86_64.rpm 66 MB/s | 52 MB 00:00 2024-02-19T21:52:46.689 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-02-19T21:52:46.690 INFO:teuthology.orchestra.run.smithi086.stdout:Total 62 MB/s | 53 MB 00:00 2024-02-19T21:52:46.731 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-02-19T21:52:46.752 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-02-19T21:52:46.752 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-02-19T21:52:47.069 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-02-19T21:52:47.072 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-02-19T21:52:47.549 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-02-19T21:52:47.790 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2024-02-19T21:52:47.972 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2024-02-19T21:52:48.029 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-02-19T21:52:48.030 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-02-19T21:52:48.030 INFO:teuthology.orchestra.run.smithi053.stdout: Package Architecture Version Repository Size 2024-02-19T21:52:48.030 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-02-19T21:52:48.030 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2024-02-19T21:52:48.030 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-test x86_64 2:16.2.4-0.el8 ceph 52 M 2024-02-19T21:52:48.030 INFO:teuthology.orchestra.run.smithi053.stdout:Installing dependencies: 2024-02-19T21:52:48.030 INFO:teuthology.orchestra.run.smithi053.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2024-02-19T21:52:48.031 INFO:teuthology.orchestra.run.smithi053.stdout: libcephsqlite x86_64 2:16.2.4-0.el8 ceph 169 k 2024-02-19T21:52:48.031 INFO:teuthology.orchestra.run.smithi053.stdout: oniguruma x86_64 6.8.2-3.el8 appstream 188 k 2024-02-19T21:52:48.031 INFO:teuthology.orchestra.run.smithi053.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2024-02-19T21:52:48.031 INFO:teuthology.orchestra.run.smithi053.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2024-02-19T21:52:48.031 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-02-19T21:52:48.031 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2024-02-19T21:52:48.031 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-02-19T21:52:48.031 INFO:teuthology.orchestra.run.smithi053.stdout:Install 6 Packages 2024-02-19T21:52:48.031 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-02-19T21:52:48.032 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 53 M 2024-02-19T21:52:48.032 INFO:teuthology.orchestra.run.smithi053.stdout:Installed size: 229 M 2024-02-19T21:52:48.032 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2024-02-19T21:52:48.128 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : oniguruma-6.8.2-3.el8.x86_64 3/6 2024-02-19T21:52:48.192 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: oniguruma-6.8.2-3.el8.x86_64 3/6 2024-02-19T21:52:48.281 INFO:teuthology.orchestra.run.smithi053.stdout:(1/6): libcephsqlite-16.2.4-0.el8.x86_64.rpm 3.2 MB/s | 169 kB 00:00 2024-02-19T21:52:48.394 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2024-02-19T21:52:48.407 INFO:teuthology.orchestra.run.smithi053.stdout:(2/6): jq-1.6-8.el8.x86_64.rpm 1.1 MB/s | 203 kB 00:00 2024-02-19T21:52:48.465 INFO:teuthology.orchestra.run.smithi053.stdout:(3/6): oniguruma-6.8.2-3.el8.x86_64.rpm 1.0 MB/s | 188 kB 00:00 2024-02-19T21:52:48.524 INFO:teuthology.orchestra.run.smithi053.stdout:(4/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 1.2 MB/s | 71 kB 00:00 2024-02-19T21:52:48.528 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2024-02-19T21:52:48.557 INFO:teuthology.orchestra.run.smithi053.stdout:(5/6): socat-1.7.4.1-1.el8.x86_64.rpm 2.1 MB/s | 323 kB 00:00 2024-02-19T21:52:49.050 INFO:teuthology.orchestra.run.smithi053.stdout:(6/6): ceph-test-16.2.4-0.el8.x86_64.rpm 63 MB/s | 52 MB 00:00 2024-02-19T21:52:49.052 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2024-02-19T21:52:49.052 INFO:teuthology.orchestra.run.smithi053.stdout:Total 52 MB/s | 53 MB 00:01 2024-02-19T21:52:49.096 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2024-02-19T21:52:49.119 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2024-02-19T21:52:49.119 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2024-02-19T21:52:49.447 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2024-02-19T21:52:49.450 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2024-02-19T21:52:49.938 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2024-02-19T21:52:50.136 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2024-02-19T21:52:50.326 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2024-02-19T21:52:50.480 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : oniguruma-6.8.2-3.el8.x86_64 3/6 2024-02-19T21:52:50.563 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: oniguruma-6.8.2-3.el8.x86_64 3/6 2024-02-19T21:52:50.756 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2024-02-19T21:52:50.899 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2024-02-19T21:52:53.343 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2024-02-19T21:52:53.585 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-test-2:16.2.4-0.el8.x86_64 6/6 2024-02-19T21:52:53.872 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-test-2:16.2.4-0.el8.x86_64 6/6 2024-02-19T21:52:53.873 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-test-2:16.2.4-0.el8.x86_64 1/6 2024-02-19T21:52:53.873 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libcephsqlite-2:16.2.4-0.el8.x86_64 2/6 2024-02-19T21:52:53.873 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2024-02-19T21:52:53.873 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : oniguruma-6.8.2-3.el8.x86_64 4/6 2024-02-19T21:52:53.873 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2024-02-19T21:52:54.338 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2024-02-19T21:52:54.338 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T21:52:54.338 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-02-19T21:52:54.338 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-test-2:16.2.4-0.el8.x86_64 jq-1.6-8.el8.x86_64 2024-02-19T21:52:54.338 INFO:teuthology.orchestra.run.smithi086.stdout: libcephsqlite-2:16.2.4-0.el8.x86_64 oniguruma-6.8.2-3.el8.x86_64 2024-02-19T21:52:54.338 INFO:teuthology.orchestra.run.smithi086.stdout: socat-1.7.4.1-1.el8.x86_64 xmlstarlet-1.6.1-20.el8.x86_64 2024-02-19T21:52:54.338 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T21:52:54.338 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-02-19T21:52:54.565 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph 2024-02-19T21:52:55.321 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:00:52 ago on Mon 19 Feb 2024 09:52:03 PM UTC. 2024-02-19T21:52:56.039 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2024-02-19T21:52:56.253 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-test-2:16.2.4-0.el8.x86_64 6/6 2024-02-19T21:52:56.578 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-test-2:16.2.4-0.el8.x86_64 6/6 2024-02-19T21:52:56.578 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-test-2:16.2.4-0.el8.x86_64 1/6 2024-02-19T21:52:56.579 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libcephsqlite-2:16.2.4-0.el8.x86_64 2/6 2024-02-19T21:52:56.579 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2024-02-19T21:52:56.579 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : oniguruma-6.8.2-3.el8.x86_64 4/6 2024-02-19T21:52:56.579 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2024-02-19T21:52:56.861 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-02-19T21:52:56.864 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-02-19T21:52:56.865 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2024-02-19T21:52:56.865 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-02-19T21:52:56.865 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-02-19T21:52:56.865 INFO:teuthology.orchestra.run.smithi086.stdout: ceph x86_64 2:16.2.4-0.el8 ceph 6.3 k 2024-02-19T21:52:56.865 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2024-02-19T21:52:56.865 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mds x86_64 2:16.2.4-0.el8 ceph 2.4 M 2024-02-19T21:52:56.865 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr x86_64 2:16.2.4-0.el8 ceph 1.6 M 2024-02-19T21:52:56.865 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-modules-core noarch 2:16.2.4-0.el8 ceph-noarch 220 k 2024-02-19T21:52:56.865 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mon x86_64 2:16.2.4-0.el8 ceph 5.0 M 2024-02-19T21:52:56.865 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-osd x86_64 2:16.2.4-0.el8 ceph 18 M 2024-02-19T21:52:56.865 INFO:teuthology.orchestra.run.smithi086.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-02-19T21:52:56.865 INFO:teuthology.orchestra.run.smithi086.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-02-19T21:52:56.865 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2024-02-19T21:52:56.866 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-02-19T21:52:56.866 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-02-19T21:52:56.866 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cryptography x86_64 3.2.1-7.el8 baseos 641 k 2024-02-19T21:52:56.866 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-02-19T21:52:56.866 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-02-19T21:52:56.866 INFO:teuthology.orchestra.run.smithi086.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-02-19T21:52:56.866 INFO:teuthology.orchestra.run.smithi086.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2024-02-19T21:52:56.866 INFO:teuthology.orchestra.run.smithi086.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2024-02-19T21:52:56.866 INFO:teuthology.orchestra.run.smithi086.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-02-19T21:52:56.866 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-02-19T21:52:56.866 INFO:teuthology.orchestra.run.smithi086.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-02-19T21:52:56.867 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2024-02-19T21:52:56.867 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2024-02-19T21:52:56.867 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2024-02-19T21:52:56.867 INFO:teuthology.orchestra.run.smithi086.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-02-19T21:52:56.867 INFO:teuthology.orchestra.run.smithi086.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-02-19T21:52:56.867 INFO:teuthology.orchestra.run.smithi086.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-02-19T21:52:56.867 INFO:teuthology.orchestra.run.smithi086.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-02-19T21:52:56.867 INFO:teuthology.orchestra.run.smithi086.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-02-19T21:52:56.867 INFO:teuthology.orchestra.run.smithi086.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-02-19T21:52:56.867 INFO:teuthology.orchestra.run.smithi086.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-02-19T21:52:56.867 INFO:teuthology.orchestra.run.smithi086.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2024-02-19T21:52:56.867 INFO:teuthology.orchestra.run.smithi086.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-02-19T21:52:56.867 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T21:52:56.867 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-02-19T21:52:56.867 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-02-19T21:52:56.868 INFO:teuthology.orchestra.run.smithi086.stdout:Install 32 Packages 2024-02-19T21:52:56.868 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T21:52:56.868 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 31 M 2024-02-19T21:52:56.868 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 111 M 2024-02-19T21:52:56.869 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-02-19T21:52:57.031 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2024-02-19T21:52:57.031 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-02-19T21:52:57.031 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2024-02-19T21:52:57.031 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-test-2:16.2.4-0.el8.x86_64 jq-1.6-8.el8.x86_64 2024-02-19T21:52:57.031 INFO:teuthology.orchestra.run.smithi053.stdout: libcephsqlite-2:16.2.4-0.el8.x86_64 oniguruma-6.8.2-3.el8.x86_64 2024-02-19T21:52:57.031 INFO:teuthology.orchestra.run.smithi053.stdout: socat-1.7.4.1-1.el8.x86_64 xmlstarlet-1.6.1-20.el8.x86_64 2024-02-19T21:52:57.032 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-02-19T21:52:57.032 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-02-19T21:52:57.261 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install ceph 2024-02-19T21:52:57.672 INFO:teuthology.orchestra.run.smithi086.stdout:(1/32): ceph-16.2.4-0.el8.x86_64.rpm 117 kB/s | 6.3 kB 00:00 2024-02-19T21:52:57.723 INFO:teuthology.orchestra.run.smithi086.stdout:(2/32): ceph-mgr-16.2.4-0.el8.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2024-02-19T21:52:57.790 INFO:teuthology.orchestra.run.smithi086.stdout:(3/32): ceph-mds-16.2.4-0.el8.x86_64.rpm 14 MB/s | 2.4 MB 00:00 2024-02-19T21:52:57.823 INFO:teuthology.orchestra.run.smithi086.stdout:(4/32): ceph-mgr-modules-core-16.2.4-0.el8.noar 6.5 MB/s | 220 kB 00:00 2024-02-19T21:52:57.915 INFO:teuthology.orchestra.run.smithi086.stdout:(5/32): ceph-mon-16.2.4-0.el8.x86_64.rpm 21 MB/s | 5.0 MB 00:00 2024-02-19T21:52:57.941 INFO:teuthology.orchestra.run.smithi086.stdout:(6/32): python3-mako-1.0.6-14.el8.noarch.rpm 1.3 MB/s | 157 kB 00:00 2024-02-19T21:52:57.974 INFO:teuthology.orchestra.run.smithi086.stdout:(7/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 3.0 MB/s | 103 kB 00:00 2024-02-19T21:52:58.008 INFO:teuthology.orchestra.run.smithi086.stdout:(8/32): python3-markupsafe-0.23-19.el8.x86_64.r 421 kB/s | 39 kB 00:00 2024-02-19T21:52:58.041 INFO:teuthology.orchestra.run.smithi086.stdout:(9/32): python3-pytz-2017.2-11.el8.noarch.rpm 854 kB/s | 57 kB 00:00 2024-02-19T21:52:58.045 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:00:54 ago on Mon 19 Feb 2024 09:52:04 PM UTC. 2024-02-19T21:52:58.284 INFO:teuthology.orchestra.run.smithi086.stdout:(10/32): ceph-osd-16.2.4-0.el8.x86_64.rpm 32 MB/s | 18 MB 00:00 2024-02-19T21:52:58.309 INFO:teuthology.orchestra.run.smithi086.stdout:(11/32): python3-werkzeug-0.12.2-4.el8.noarch.r 1.5 MB/s | 457 kB 00:00 2024-02-19T21:52:58.334 INFO:teuthology.orchestra.run.smithi086.stdout:(12/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 851 kB/s | 249 kB 00:00 2024-02-19T21:52:58.418 INFO:teuthology.orchestra.run.smithi086.stdout:(13/32): python3-pycparser-2.14-14.el8.noarch.r 1.0 MB/s | 109 kB 00:00 2024-02-19T21:52:58.484 INFO:teuthology.orchestra.run.smithi086.stdout:(14/32): python3-cryptography-3.2.1-7.el8.x86_6 3.1 MB/s | 641 kB 00:00 2024-02-19T21:52:58.693 INFO:teuthology.orchestra.run.smithi086.stdout:(15/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 121 kB/s | 44 kB 00:00 2024-02-19T21:52:58.883 INFO:teuthology.orchestra.run.smithi086.stdout:(16/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 398 kB/s | 185 kB 00:00 2024-02-19T21:52:58.910 INFO:teuthology.orchestra.run.smithi086.stdout:(17/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 405 kB/s | 173 kB 00:00 2024-02-19T21:52:59.003 INFO:teuthology.orchestra.run.smithi086.stdout:(18/32): python3-jaraco-6.2-6.el8.noarch.rpm 93 kB/s | 11 kB 00:00 2024-02-19T21:52:59.019 INFO:teuthology.orchestra.run.smithi086.stdout:(19/32): python3-jaraco-functools-2.0-4.el8.noa 162 kB/s | 18 kB 00:00 2024-02-19T21:52:59.070 INFO:teuthology.orchestra.run.smithi086.stdout:(20/32): python3-cherrypy-18.4.0-1.el8.noarch.r 1.0 MB/s | 384 kB 00:00 2024-02-19T21:52:59.137 INFO:teuthology.orchestra.run.smithi086.stdout:(21/32): python3-logutils-0.3.5-11.el8.noarch.r 364 kB/s | 49 kB 00:00 2024-02-19T21:52:59.162 INFO:teuthology.orchestra.run.smithi086.stdout:(22/32): python3-more-itertools-7.2.0-3.el8.noa 413 kB/s | 59 kB 00:00 2024-02-19T21:52:59.204 INFO:teuthology.orchestra.run.smithi086.stdout:(23/32): python3-pecan-1.3.2-9.el8.noarch.rpm 2.1 MB/s | 283 kB 00:00 2024-02-19T21:52:59.237 INFO:teuthology.orchestra.run.smithi086.stdout:(24/32): python3-portend-2.6-1.el8.noarch.rpm 155 kB/s | 16 kB 00:00 2024-02-19T21:52:59.263 INFO:teuthology.orchestra.run.smithi086.stdout:(25/32): python3-simplegeneric-0.8.1-17.el8.noa 186 kB/s | 19 kB 00:00 2024-02-19T21:52:59.330 INFO:teuthology.orchestra.run.smithi086.stdout:(26/32): python3-singledispatch-3.4.0.3-18.el8. 190 kB/s | 24 kB 00:00 2024-02-19T21:52:59.363 INFO:teuthology.orchestra.run.smithi086.stdout:(27/32): python3-tempora-1.14.1-5.el8.noarch.rp 233 kB/s | 29 kB 00:00 2024-02-19T21:52:59.388 INFO:teuthology.orchestra.run.smithi086.stdout:(28/32): python3-trustme-0.6.0-4.el8.noarch.rpm 219 kB/s | 27 kB 00:00 2024-02-19T21:52:59.455 INFO:teuthology.orchestra.run.smithi086.stdout:(29/32): python3-waitress-1.4.3-1.el8.noarch.rp 2.0 MB/s | 255 kB 00:00 2024-02-19T21:52:59.514 INFO:teuthology.orchestra.run.smithi086.stdout:(30/32): python3-webtest-2.0.33-1.el8.noarch.rp 682 kB/s | 85 kB 00:00 2024-02-19T21:52:59.585 INFO:teuthology.orchestra.run.smithi086.stdout:(31/32): python3-zc-lockfile-2.0-2.el8.noarch.r 177 kB/s | 23 kB 00:00 2024-02-19T21:52:59.614 INFO:teuthology.orchestra.run.smithi086.stdout:(32/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 1.0 MB/s | 251 kB 00:00 2024-02-19T21:52:59.616 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-02-19T21:52:59.616 INFO:teuthology.orchestra.run.smithi086.stdout:Total 11 MB/s | 31 MB 00:02 2024-02-19T21:52:59.697 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-02-19T21:52:59.718 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-02-19T21:52:59.721 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-02-19T21:52:59.722 INFO:teuthology.orchestra.run.smithi053.stdout: Package Arch Version Repository Size 2024-02-19T21:52:59.722 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-02-19T21:52:59.722 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2024-02-19T21:52:59.722 INFO:teuthology.orchestra.run.smithi053.stdout: ceph x86_64 2:16.2.4-0.el8 ceph 6.3 k 2024-02-19T21:52:59.722 INFO:teuthology.orchestra.run.smithi053.stdout:Installing dependencies: 2024-02-19T21:52:59.722 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mds x86_64 2:16.2.4-0.el8 ceph 2.4 M 2024-02-19T21:52:59.722 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr x86_64 2:16.2.4-0.el8 ceph 1.6 M 2024-02-19T21:52:59.722 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-modules-core noarch 2:16.2.4-0.el8 ceph-noarch 220 k 2024-02-19T21:52:59.722 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mon x86_64 2:16.2.4-0.el8 ceph 5.0 M 2024-02-19T21:52:59.722 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-osd x86_64 2:16.2.4-0.el8 ceph 18 M 2024-02-19T21:52:59.722 INFO:teuthology.orchestra.run.smithi053.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-02-19T21:52:59.723 INFO:teuthology.orchestra.run.smithi053.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-02-19T21:52:59.723 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2024-02-19T21:52:59.723 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-02-19T21:52:59.723 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-02-19T21:52:59.723 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cryptography x86_64 3.2.1-7.el8 baseos 641 k 2024-02-19T21:52:59.723 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-02-19T21:52:59.723 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-02-19T21:52:59.723 INFO:teuthology.orchestra.run.smithi053.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-02-19T21:52:59.723 INFO:teuthology.orchestra.run.smithi053.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2024-02-19T21:52:59.723 INFO:teuthology.orchestra.run.smithi053.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2024-02-19T21:52:59.723 INFO:teuthology.orchestra.run.smithi053.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-02-19T21:52:59.723 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-02-19T21:52:59.723 INFO:teuthology.orchestra.run.smithi053.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-02-19T21:52:59.724 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2024-02-19T21:52:59.724 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2024-02-19T21:52:59.724 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2024-02-19T21:52:59.724 INFO:teuthology.orchestra.run.smithi053.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-02-19T21:52:59.724 INFO:teuthology.orchestra.run.smithi053.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-02-19T21:52:59.724 INFO:teuthology.orchestra.run.smithi053.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-02-19T21:52:59.724 INFO:teuthology.orchestra.run.smithi053.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-02-19T21:52:59.724 INFO:teuthology.orchestra.run.smithi053.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-02-19T21:52:59.724 INFO:teuthology.orchestra.run.smithi053.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-02-19T21:52:59.724 INFO:teuthology.orchestra.run.smithi053.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-02-19T21:52:59.725 INFO:teuthology.orchestra.run.smithi053.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2024-02-19T21:52:59.725 INFO:teuthology.orchestra.run.smithi053.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-02-19T21:52:59.725 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-02-19T21:52:59.725 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2024-02-19T21:52:59.725 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-02-19T21:52:59.725 INFO:teuthology.orchestra.run.smithi053.stdout:Install 32 Packages 2024-02-19T21:52:59.725 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-02-19T21:52:59.726 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 31 M 2024-02-19T21:52:59.726 INFO:teuthology.orchestra.run.smithi053.stdout:Installed size: 111 M 2024-02-19T21:52:59.726 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2024-02-19T21:52:59.736 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-02-19T21:52:59.736 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-02-19T21:53:00.050 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-02-19T21:53:00.054 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-02-19T21:53:00.519 INFO:teuthology.orchestra.run.smithi053.stdout:(1/32): ceph-16.2.4-0.el8.x86_64.rpm 111 kB/s | 6.3 kB 00:00 2024-02-19T21:53:00.577 INFO:teuthology.orchestra.run.smithi053.stdout:(2/32): ceph-mgr-16.2.4-0.el8.x86_64.rpm 14 MB/s | 1.6 MB 00:00 2024-02-19T21:53:00.627 INFO:teuthology.orchestra.run.smithi053.stdout:(3/32): ceph-mds-16.2.4-0.el8.x86_64.rpm 14 MB/s | 2.4 MB 00:00 2024-02-19T21:53:00.686 INFO:teuthology.orchestra.run.smithi053.stdout:(4/32): ceph-mgr-modules-core-16.2.4-0.el8.noar 3.7 MB/s | 220 kB 00:00 2024-02-19T21:53:00.803 INFO:teuthology.orchestra.run.smithi053.stdout:(5/32): ceph-mon-16.2.4-0.el8.x86_64.rpm 18 MB/s | 5.0 MB 00:00 2024-02-19T21:53:00.844 INFO:teuthology.orchestra.run.smithi053.stdout:(6/32): python3-mako-1.0.6-14.el8.noarch.rpm 990 kB/s | 157 kB 00:00 2024-02-19T21:53:00.887 INFO:teuthology.orchestra.run.smithi053.stdout:(7/32): python3-markupsafe-0.23-19.el8.x86_64.r 465 kB/s | 39 kB 00:00 2024-02-19T21:53:00.912 INFO:teuthology.orchestra.run.smithi053.stdout:(8/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 1.5 MB/s | 103 kB 00:00 2024-02-19T21:53:00.945 INFO:teuthology.orchestra.run.smithi053.stdout:(9/32): python3-pytz-2017.2-11.el8.noarch.rpm 976 kB/s | 57 kB 00:00 2024-02-19T21:53:01.161 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-02-19T21:53:01.162 INFO:teuthology.orchestra.run.smithi053.stdout:(10/32): ceph-osd-16.2.4-0.el8.x86_64.rpm 31 MB/s | 18 MB 00:00 2024-02-19T21:53:01.188 INFO:teuthology.orchestra.run.smithi053.stdout:(11/32): python3-werkzeug-0.12.2-4.el8.noarch.r 1.6 MB/s | 457 kB 00:00 2024-02-19T21:53:01.213 INFO:teuthology.orchestra.run.smithi053.stdout:(12/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 931 kB/s | 249 kB 00:00 2024-02-19T21:53:01.338 INFO:teuthology.orchestra.run.smithi053.stdout:(13/32): python3-pycparser-2.14-14.el8.noarch.r 727 kB/s | 109 kB 00:00 2024-02-19T21:53:01.357 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2024-02-19T21:53:01.422 INFO:teuthology.orchestra.run.smithi053.stdout:(14/32): python3-cryptography-3.2.1-7.el8.x86_6 2.4 MB/s | 641 kB 00:00 2024-02-19T21:53:01.546 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2024-02-19T21:53:01.697 INFO:teuthology.orchestra.run.smithi053.stdout:(15/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 90 kB/s | 44 kB 00:00 2024-02-19T21:53:01.801 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2024-02-19T21:53:01.991 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2024-02-19T21:53:02.172 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2024-02-19T21:53:02.322 INFO:teuthology.orchestra.run.smithi053.stdout:(16/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 188 kB/s | 185 kB 00:00 2024-02-19T21:53:02.361 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2024-02-19T21:53:02.406 INFO:teuthology.orchestra.run.smithi053.stdout:(17/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 175 kB/s | 173 kB 00:00 2024-02-19T21:53:02.448 INFO:teuthology.orchestra.run.smithi053.stdout:(18/32): python3-jaraco-6.2-6.el8.noarch.rpm 89 kB/s | 11 kB 00:00 2024-02-19T21:53:02.523 INFO:teuthology.orchestra.run.smithi053.stdout:(19/32): python3-jaraco-functools-2.0-4.el8.noa 150 kB/s | 18 kB 00:00 2024-02-19T21:53:02.555 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2024-02-19T21:53:02.665 INFO:teuthology.orchestra.run.smithi053.stdout:(20/32): python3-logutils-0.3.5-11.el8.noarch.r 225 kB/s | 49 kB 00:00 2024-02-19T21:53:02.732 INFO:teuthology.orchestra.run.smithi053.stdout:(21/32): python3-more-itertools-7.2.0-3.el8.noa 281 kB/s | 59 kB 00:00 2024-02-19T21:53:02.732 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2024-02-19T21:53:02.866 INFO:teuthology.orchestra.run.smithi053.stdout:(22/32): python3-portend-2.6-1.el8.noarch.rpm 116 kB/s | 16 kB 00:00 2024-02-19T21:53:02.933 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2024-02-19T21:53:02.999 INFO:teuthology.orchestra.run.smithi053.stdout:(23/32): python3-simplegeneric-0.8.1-17.el8.noa 141 kB/s | 19 kB 00:00 2024-02-19T21:53:03.133 INFO:teuthology.orchestra.run.smithi053.stdout:(24/32): python3-singledispatch-3.4.0.3-18.el8. 180 kB/s | 24 kB 00:00 2024-02-19T21:53:03.146 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2024-02-19T21:53:03.200 INFO:teuthology.orchestra.run.smithi053.stdout:(25/32): python3-cherrypy-18.4.0-1.el8.noarch.r 256 kB/s | 384 kB 00:01 2024-02-19T21:53:03.267 INFO:teuthology.orchestra.run.smithi053.stdout:(26/32): python3-tempora-1.14.1-5.el8.noarch.rp 218 kB/s | 29 kB 00:00 2024-02-19T21:53:03.334 INFO:teuthology.orchestra.run.smithi053.stdout:(27/32): python3-trustme-0.6.0-4.el8.noarch.rpm 205 kB/s | 27 kB 00:00 2024-02-19T21:53:03.358 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2024-02-19T21:53:03.591 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2024-02-19T21:53:03.659 INFO:teuthology.orchestra.run.smithi053.stdout:(28/32): python3-pecan-1.3.2-9.el8.noarch.rpm 285 kB/s | 283 kB 00:00 2024-02-19T21:53:03.934 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2024-02-19T21:53:03.968 INFO:teuthology.orchestra.run.smithi053.stdout:(29/32): python3-webtest-2.0.33-1.el8.noarch.rp 277 kB/s | 85 kB 00:00 2024-02-19T21:53:04.084 INFO:teuthology.orchestra.run.smithi053.stdout:(30/32): python3-zc-lockfile-2.0-2.el8.noarch.r 197 kB/s | 23 kB 00:00 2024-02-19T21:53:04.134 INFO:teuthology.orchestra.run.smithi053.stdout:(31/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 314 kB/s | 251 kB 00:00 2024-02-19T21:53:04.177 INFO:teuthology.orchestra.run.smithi053.stdout:(32/32): python3-waitress-1.4.3-1.el8.noarch.rp 281 kB/s | 255 kB 00:00 2024-02-19T21:53:04.178 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2024-02-19T21:53:04.178 INFO:teuthology.orchestra.run.smithi053.stdout:Total 7.0 MB/s | 31 MB 00:04 2024-02-19T21:53:04.187 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-cryptography-3.2.1-7.el8.x86_64 14/32 2024-02-19T21:53:04.261 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2024-02-19T21:53:04.303 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2024-02-19T21:53:04.303 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2024-02-19T21:53:04.351 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2024-02-19T21:53:04.618 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2024-02-19T21:53:04.621 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2024-02-19T21:53:04.621 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2024-02-19T21:53:04.786 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-02-19T21:53:05.089 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2024-02-19T21:53:05.324 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2024-02-19T21:53:05.496 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2024-02-19T21:53:05.690 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2024-02-19T21:53:05.753 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2024-02-19T21:53:05.946 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2024-02-19T21:53:05.947 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2024-02-19T21:53:06.147 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2024-02-19T21:53:06.153 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2024-02-19T21:53:06.369 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2024-02-19T21:53:06.384 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2024-02-19T21:53:06.560 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2024-02-19T21:53:06.704 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2024-02-19T21:53:06.741 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2024-02-19T21:53:06.932 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2024-02-19T21:53:06.999 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2024-02-19T21:53:07.090 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2024-02-19T21:53:07.293 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2024-02-19T21:53:07.365 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 27/32 2024-02-19T21:53:07.505 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2024-02-19T21:53:07.586 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2024-02-19T21:53:07.717 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2024-02-19T21:53:07.921 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2024-02-19T21:53:08.171 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2024-02-19T21:53:08.570 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2024-02-19T21:53:08.840 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-cryptography-3.2.1-7.el8.x86_64 14/32 2024-02-19T21:53:08.994 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2024-02-19T21:53:09.121 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2024-02-19T21:53:09.225 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2024-02-19T21:53:09.327 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2024-02-19T21:53:09.438 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-02-19T21:53:09.731 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2024-02-19T21:53:09.759 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2024-02-19T21:53:09.926 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2024-02-19T21:53:09.969 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2024-02-19T21:53:10.115 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2024-02-19T21:53:10.190 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2024-02-19T21:53:10.300 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2024-02-19T21:53:10.360 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2024-02-19T21:53:10.376 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2024-02-19T21:53:10.509 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-2:16.2.4-0.el8.x86_64 32/32 2024-02-19T21:53:10.570 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2024-02-19T21:53:10.758 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2024-02-19T21:53:10.909 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-2:16.2.4-0.el8.x86_64 32/32 2024-02-19T21:53:10.909 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-2:16.2.4-0.el8.x86_64 1/32 2024-02-19T21:53:10.909 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mds-2:16.2.4-0.el8.x86_64 2/32 2024-02-19T21:53:10.909 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mgr-2:16.2.4-0.el8.x86_64 3/32 2024-02-19T21:53:10.909 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mon-2:16.2.4-0.el8.x86_64 4/32 2024-02-19T21:53:10.909 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-osd-2:16.2.4-0.el8.x86_64 5/32 2024-02-19T21:53:10.909 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 6/32 2024-02-19T21:53:10.909 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2024-02-19T21:53:10.910 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2024-02-19T21:53:10.910 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2024-02-19T21:53:10.910 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2024-02-19T21:53:10.910 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2024-02-19T21:53:10.910 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2024-02-19T21:53:10.910 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-cryptography-3.2.1-7.el8.x86_64 13/32 2024-02-19T21:53:10.910 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2024-02-19T21:53:10.910 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2024-02-19T21:53:10.910 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2024-02-19T21:53:10.910 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-02-19T21:53:10.911 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2024-02-19T21:53:10.911 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2024-02-19T21:53:10.911 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2024-02-19T21:53:10.911 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2024-02-19T21:53:10.911 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2024-02-19T21:53:10.911 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2024-02-19T21:53:10.911 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2024-02-19T21:53:10.912 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2024-02-19T21:53:10.912 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2024-02-19T21:53:10.912 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2024-02-19T21:53:10.912 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2024-02-19T21:53:10.912 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2024-02-19T21:53:10.912 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2024-02-19T21:53:10.912 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2024-02-19T21:53:10.986 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2024-02-19T21:53:11.355 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2024-02-19T21:53:11.657 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2024-02-19T21:53:11.780 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2024-02-19T21:53:11.780 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T21:53:11.781 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-02-19T21:53:11.781 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-2:16.2.4-0.el8.x86_64 2024-02-19T21:53:11.781 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mds-2:16.2.4-0.el8.x86_64 2024-02-19T21:53:11.781 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-2:16.2.4-0.el8.x86_64 2024-02-19T21:53:11.781 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 2024-02-19T21:53:11.781 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mon-2:16.2.4-0.el8.x86_64 2024-02-19T21:53:11.781 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-osd-2:16.2.4-0.el8.x86_64 2024-02-19T21:53:11.781 INFO:teuthology.orchestra.run.smithi086.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-02-19T21:53:11.781 INFO:teuthology.orchestra.run.smithi086.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-02-19T21:53:11.781 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-02-19T21:53:11.781 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-02-19T21:53:11.781 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-02-19T21:53:11.781 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cryptography-3.2.1-7.el8.x86_64 2024-02-19T21:53:11.781 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco-6.2-6.el8.noarch 2024-02-19T21:53:11.782 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-02-19T21:53:11.782 INFO:teuthology.orchestra.run.smithi086.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-02-19T21:53:11.782 INFO:teuthology.orchestra.run.smithi086.stdout: python3-mako-1.0.6-14.el8.noarch 2024-02-19T21:53:11.782 INFO:teuthology.orchestra.run.smithi086.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-02-19T21:53:11.782 INFO:teuthology.orchestra.run.smithi086.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-02-19T21:53:11.782 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-02-19T21:53:11.782 INFO:teuthology.orchestra.run.smithi086.stdout: python3-portend-2.6-1.el8.noarch 2024-02-19T21:53:11.782 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-02-19T21:53:11.782 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pycparser-2.14-14.el8.noarch 2024-02-19T21:53:11.782 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pytz-2017.2-11.el8.noarch 2024-02-19T21:53:11.782 INFO:teuthology.orchestra.run.smithi086.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-02-19T21:53:11.782 INFO:teuthology.orchestra.run.smithi086.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-02-19T21:53:11.782 INFO:teuthology.orchestra.run.smithi086.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-02-19T21:53:11.782 INFO:teuthology.orchestra.run.smithi086.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-02-19T21:53:11.782 INFO:teuthology.orchestra.run.smithi086.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-02-19T21:53:11.782 INFO:teuthology.orchestra.run.smithi086.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-02-19T21:53:11.782 INFO:teuthology.orchestra.run.smithi086.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-02-19T21:53:11.783 INFO:teuthology.orchestra.run.smithi086.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-02-19T21:53:11.783 INFO:teuthology.orchestra.run.smithi086.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-02-19T21:53:11.783 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T21:53:11.783 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-02-19T21:53:12.054 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 27/32 2024-02-19T21:53:12.063 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-base 2024-02-19T21:53:12.266 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2024-02-19T21:53:12.819 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:09 ago on Mon 19 Feb 2024 09:52:03 PM UTC. 2024-02-19T21:53:13.896 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2024-02-19T21:53:14.099 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2024-02-19T21:53:14.324 INFO:teuthology.orchestra.run.smithi086.stdout:Package ceph-base-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-19T21:53:14.371 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-02-19T21:53:14.372 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2024-02-19T21:53:14.372 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-02-19T21:53:14.475 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install cephadm 2024-02-19T21:53:14.558 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2024-02-19T21:53:14.749 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2024-02-19T21:53:14.984 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2024-02-19T21:53:15.135 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2024-02-19T21:53:15.153 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2024-02-19T21:53:15.226 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:12 ago on Mon 19 Feb 2024 09:52:03 PM UTC. 2024-02-19T21:53:15.277 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-2:16.2.4-0.el8.x86_64 32/32 2024-02-19T21:53:15.687 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-2:16.2.4-0.el8.x86_64 32/32 2024-02-19T21:53:15.687 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-2:16.2.4-0.el8.x86_64 1/32 2024-02-19T21:53:15.687 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-mds-2:16.2.4-0.el8.x86_64 2/32 2024-02-19T21:53:15.687 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-mgr-2:16.2.4-0.el8.x86_64 3/32 2024-02-19T21:53:15.687 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-mon-2:16.2.4-0.el8.x86_64 4/32 2024-02-19T21:53:15.687 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-osd-2:16.2.4-0.el8.x86_64 5/32 2024-02-19T21:53:15.687 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 6/32 2024-02-19T21:53:15.687 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2024-02-19T21:53:15.687 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2024-02-19T21:53:15.687 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2024-02-19T21:53:15.687 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2024-02-19T21:53:15.687 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2024-02-19T21:53:15.688 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2024-02-19T21:53:15.688 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-cryptography-3.2.1-7.el8.x86_64 13/32 2024-02-19T21:53:15.688 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2024-02-19T21:53:15.688 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2024-02-19T21:53:15.688 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2024-02-19T21:53:15.688 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-02-19T21:53:15.688 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2024-02-19T21:53:15.688 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2024-02-19T21:53:15.688 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2024-02-19T21:53:15.688 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2024-02-19T21:53:15.688 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2024-02-19T21:53:15.688 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2024-02-19T21:53:15.688 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2024-02-19T21:53:15.688 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2024-02-19T21:53:15.688 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2024-02-19T21:53:15.689 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2024-02-19T21:53:15.689 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2024-02-19T21:53:15.689 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2024-02-19T21:53:15.689 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2024-02-19T21:53:15.689 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2024-02-19T21:53:16.456 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2024-02-19T21:53:16.456 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-02-19T21:53:16.456 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2024-02-19T21:53:16.456 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-2:16.2.4-0.el8.x86_64 2024-02-19T21:53:16.456 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mds-2:16.2.4-0.el8.x86_64 2024-02-19T21:53:16.456 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-2:16.2.4-0.el8.x86_64 2024-02-19T21:53:16.456 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 2024-02-19T21:53:16.456 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mon-2:16.2.4-0.el8.x86_64 2024-02-19T21:53:16.456 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-osd-2:16.2.4-0.el8.x86_64 2024-02-19T21:53:16.457 INFO:teuthology.orchestra.run.smithi053.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-02-19T21:53:16.457 INFO:teuthology.orchestra.run.smithi053.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-02-19T21:53:16.457 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-02-19T21:53:16.457 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-02-19T21:53:16.457 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-02-19T21:53:16.457 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cryptography-3.2.1-7.el8.x86_64 2024-02-19T21:53:16.457 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jaraco-6.2-6.el8.noarch 2024-02-19T21:53:16.457 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-02-19T21:53:16.457 INFO:teuthology.orchestra.run.smithi053.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-02-19T21:53:16.457 INFO:teuthology.orchestra.run.smithi053.stdout: python3-mako-1.0.6-14.el8.noarch 2024-02-19T21:53:16.457 INFO:teuthology.orchestra.run.smithi053.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-02-19T21:53:16.457 INFO:teuthology.orchestra.run.smithi053.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-02-19T21:53:16.457 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-02-19T21:53:16.457 INFO:teuthology.orchestra.run.smithi053.stdout: python3-portend-2.6-1.el8.noarch 2024-02-19T21:53:16.458 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-02-19T21:53:16.458 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pycparser-2.14-14.el8.noarch 2024-02-19T21:53:16.458 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pytz-2017.2-11.el8.noarch 2024-02-19T21:53:16.458 INFO:teuthology.orchestra.run.smithi053.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-02-19T21:53:16.458 INFO:teuthology.orchestra.run.smithi053.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-02-19T21:53:16.458 INFO:teuthology.orchestra.run.smithi053.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-02-19T21:53:16.458 INFO:teuthology.orchestra.run.smithi053.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-02-19T21:53:16.458 INFO:teuthology.orchestra.run.smithi053.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-02-19T21:53:16.458 INFO:teuthology.orchestra.run.smithi053.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-02-19T21:53:16.458 INFO:teuthology.orchestra.run.smithi053.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-02-19T21:53:16.458 INFO:teuthology.orchestra.run.smithi053.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-02-19T21:53:16.458 INFO:teuthology.orchestra.run.smithi053.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-02-19T21:53:16.458 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-02-19T21:53:16.458 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-02-19T21:53:16.774 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-02-19T21:53:16.775 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-02-19T21:53:16.775 INFO:teuthology.orchestra.run.smithi086.stdout: Package Architecture Version Repository Size 2024-02-19T21:53:16.775 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-02-19T21:53:16.775 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-02-19T21:53:16.775 INFO:teuthology.orchestra.run.smithi086.stdout: cephadm noarch 2:16.2.4-0.el8 ceph-noarch 71 k 2024-02-19T21:53:16.775 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T21:53:16.775 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-02-19T21:53:16.775 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-02-19T21:53:16.775 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2024-02-19T21:53:16.776 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T21:53:16.776 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 71 k 2024-02-19T21:53:16.776 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 290 k 2024-02-19T21:53:16.776 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-02-19T21:53:16.820 INFO:teuthology.orchestra.run.smithi086.stdout:cephadm-16.2.4-0.el8.noarch.rpm 1.6 MB/s | 71 kB 00:00 2024-02-19T21:53:16.821 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-02-19T21:53:16.821 INFO:teuthology.orchestra.run.smithi086.stdout:Total 1.5 MB/s | 71 kB 00:00 2024-02-19T21:53:16.821 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-02-19T21:53:16.828 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-02-19T21:53:16.828 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-02-19T21:53:16.830 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-02-19T21:53:16.833 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-02-19T21:53:16.876 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-02-19T21:53:17.312 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install ceph-base 2024-02-19T21:53:17.419 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2024-02-19T21:53:17.589 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : cephadm-2:16.2.4-0.el8.noarch 1/1 2024-02-19T21:53:17.816 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2024-02-19T21:53:18.103 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:01:14 ago on Mon 19 Feb 2024 09:52:04 PM UTC. 2024-02-19T21:53:18.206 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : cephadm-2:16.2.4-0.el8.noarch 1/1 2024-02-19T21:53:18.206 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T21:53:18.206 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-02-19T21:53:18.206 INFO:teuthology.orchestra.run.smithi086.stdout: cephadm-2:16.2.4-0.el8.noarch 2024-02-19T21:53:18.206 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T21:53:18.206 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-02-19T21:53:18.425 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-immutable-object-cache 2024-02-19T21:53:19.183 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:16 ago on Mon 19 Feb 2024 09:52:03 PM UTC. 2024-02-19T21:53:19.709 INFO:teuthology.orchestra.run.smithi053.stdout:Package ceph-base-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-19T21:53:19.761 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-02-19T21:53:19.762 INFO:teuthology.orchestra.run.smithi053.stdout:Nothing to do. 2024-02-19T21:53:19.762 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-02-19T21:53:19.867 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install cephadm 2024-02-19T21:53:20.651 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:01:16 ago on Mon 19 Feb 2024 09:52:04 PM UTC. 2024-02-19T21:53:20.738 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-02-19T21:53:20.738 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-02-19T21:53:20.738 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repo Size 2024-02-19T21:53:20.739 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-02-19T21:53:20.739 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-02-19T21:53:20.739 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-immutable-object-cache x86_64 2:16.2.4-0.el8 ceph 163 k 2024-02-19T21:53:20.739 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T21:53:20.739 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-02-19T21:53:20.739 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-02-19T21:53:20.739 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2024-02-19T21:53:20.739 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T21:53:20.740 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 163 k 2024-02-19T21:53:20.740 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 587 k 2024-02-19T21:53:20.740 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-02-19T21:53:21.067 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-immutable-object-cache-16.2.4-0.el8.x86_64 498 kB/s | 163 kB 00:00 2024-02-19T21:53:21.068 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-02-19T21:53:21.068 INFO:teuthology.orchestra.run.smithi086.stdout:Total 496 kB/s | 163 kB 00:00 2024-02-19T21:53:21.068 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-02-19T21:53:21.079 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-02-19T21:53:21.079 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-02-19T21:53:21.129 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-02-19T21:53:21.132 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-02-19T21:53:21.250 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-02-19T21:53:21.466 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T21:53:21.877 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T21:53:22.244 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T21:53:22.244 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T21:53:22.244 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-02-19T21:53:22.244 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 2024-02-19T21:53:22.244 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T21:53:22.244 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-02-19T21:53:22.325 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-02-19T21:53:22.326 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-02-19T21:53:22.326 INFO:teuthology.orchestra.run.smithi053.stdout: Package Architecture Version Repository Size 2024-02-19T21:53:22.326 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-02-19T21:53:22.326 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2024-02-19T21:53:22.326 INFO:teuthology.orchestra.run.smithi053.stdout: cephadm noarch 2:16.2.4-0.el8 ceph-noarch 71 k 2024-02-19T21:53:22.326 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-02-19T21:53:22.327 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2024-02-19T21:53:22.327 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-02-19T21:53:22.327 INFO:teuthology.orchestra.run.smithi053.stdout:Install 1 Package 2024-02-19T21:53:22.327 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-02-19T21:53:22.327 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 71 k 2024-02-19T21:53:22.327 INFO:teuthology.orchestra.run.smithi053.stdout:Installed size: 290 k 2024-02-19T21:53:22.327 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2024-02-19T21:53:22.451 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-mgr 2024-02-19T21:53:22.753 INFO:teuthology.orchestra.run.smithi053.stdout:cephadm-16.2.4-0.el8.noarch.rpm 167 kB/s | 71 kB 00:00 2024-02-19T21:53:22.754 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2024-02-19T21:53:22.754 INFO:teuthology.orchestra.run.smithi053.stdout:Total 166 kB/s | 71 kB 00:00 2024-02-19T21:53:22.755 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2024-02-19T21:53:22.763 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2024-02-19T21:53:22.763 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2024-02-19T21:53:22.766 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2024-02-19T21:53:22.769 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2024-02-19T21:53:22.912 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2024-02-19T21:53:23.215 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:20 ago on Mon 19 Feb 2024 09:52:03 PM UTC. 2024-02-19T21:53:23.426 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2024-02-19T21:53:23.569 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : cephadm-2:16.2.4-0.el8.noarch 1/1 2024-02-19T21:53:23.813 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2024-02-19T21:53:24.199 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : cephadm-2:16.2.4-0.el8.noarch 1/1 2024-02-19T21:53:24.199 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-02-19T21:53:24.200 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2024-02-19T21:53:24.200 INFO:teuthology.orchestra.run.smithi053.stdout: cephadm-2:16.2.4-0.el8.noarch 2024-02-19T21:53:24.200 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-02-19T21:53:24.200 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-02-19T21:53:24.431 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install ceph-immutable-object-cache 2024-02-19T21:53:24.724 INFO:teuthology.orchestra.run.smithi086.stdout:Package ceph-mgr-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-19T21:53:24.771 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-02-19T21:53:24.772 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2024-02-19T21:53:24.772 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-02-19T21:53:24.885 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-mgr-dashboard 2024-02-19T21:53:25.216 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:01:21 ago on Mon 19 Feb 2024 09:52:04 PM UTC. 2024-02-19T21:53:25.639 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:22 ago on Mon 19 Feb 2024 09:52:03 PM UTC. 2024-02-19T21:53:26.870 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-02-19T21:53:26.870 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-02-19T21:53:26.871 INFO:teuthology.orchestra.run.smithi053.stdout: Package Arch Version Repo Size 2024-02-19T21:53:26.871 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-02-19T21:53:26.871 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2024-02-19T21:53:26.871 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-immutable-object-cache x86_64 2:16.2.4-0.el8 ceph 163 k 2024-02-19T21:53:26.871 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-02-19T21:53:26.871 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2024-02-19T21:53:26.871 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-02-19T21:53:26.871 INFO:teuthology.orchestra.run.smithi053.stdout:Install 1 Package 2024-02-19T21:53:26.871 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-02-19T21:53:26.872 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 163 k 2024-02-19T21:53:26.872 INFO:teuthology.orchestra.run.smithi053.stdout:Installed size: 587 k 2024-02-19T21:53:26.872 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2024-02-19T21:53:26.903 INFO:teuthology.orchestra.run.smithi053.stdout:ceph-immutable-object-cache-16.2.4-0.el8.x86_64 5.2 MB/s | 163 kB 00:00 2024-02-19T21:53:26.903 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2024-02-19T21:53:26.903 INFO:teuthology.orchestra.run.smithi053.stdout:Total 5.0 MB/s | 163 kB 00:00 2024-02-19T21:53:26.904 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2024-02-19T21:53:26.915 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2024-02-19T21:53:26.915 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2024-02-19T21:53:26.965 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2024-02-19T21:53:26.969 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2024-02-19T21:53:27.078 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2024-02-19T21:53:27.199 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-02-19T21:53:27.200 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-02-19T21:53:27.200 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2024-02-19T21:53:27.200 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-02-19T21:53:27.200 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-02-19T21:53:27.201 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-dashboard noarch 2:16.2.4-0.el8 ceph-noarch 2.7 M 2024-02-19T21:53:27.201 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2024-02-19T21:53:27.201 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-grafana-dashboards noarch 2:16.2.4-0.el8 ceph-noarch 24 k 2024-02-19T21:53:27.201 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-prometheus-alerts noarch 2:16.2.4-0.el8 ceph-noarch 8.9 k 2024-02-19T21:53:27.201 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2024-02-19T21:53:27.201 INFO:teuthology.orchestra.run.smithi086.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-02-19T21:53:27.201 INFO:teuthology.orchestra.run.smithi086.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-02-19T21:53:27.201 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T21:53:27.201 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-02-19T21:53:27.201 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-02-19T21:53:27.201 INFO:teuthology.orchestra.run.smithi086.stdout:Install 6 Packages 2024-02-19T21:53:27.201 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T21:53:27.202 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 3.0 M 2024-02-19T21:53:27.202 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 15 M 2024-02-19T21:53:27.202 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-02-19T21:53:27.250 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T21:53:27.462 INFO:teuthology.orchestra.run.smithi086.stdout:(1/6): ceph-grafana-dashboards-16.2.4-0.el8.noa 470 kB/s | 24 kB 00:00 2024-02-19T21:53:27.487 INFO:teuthology.orchestra.run.smithi086.stdout:(2/6): ceph-prometheus-alerts-16.2.4-0.el8.noar 117 kB/s | 8.9 kB 00:00 2024-02-19T21:53:27.546 INFO:teuthology.orchestra.run.smithi086.stdout:(3/6): ceph-mgr-dashboard-16.2.4-0.el8.noarch.r 20 MB/s | 2.7 MB 00:00 2024-02-19T21:53:27.571 INFO:teuthology.orchestra.run.smithi086.stdout:(4/6): python3-jwt-1.6.1-2.el8.noarch.rpm 391 kB/s | 43 kB 00:00 2024-02-19T21:53:27.666 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T21:53:27.921 INFO:teuthology.orchestra.run.smithi086.stdout:(5/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 77 kB/s | 34 kB 00:00 2024-02-19T21:53:28.069 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T21:53:28.069 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-02-19T21:53:28.069 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2024-02-19T21:53:28.069 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 2024-02-19T21:53:28.069 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-02-19T21:53:28.069 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-02-19T21:53:28.222 INFO:teuthology.orchestra.run.smithi086.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 291 kB/s | 196 kB 00:00 2024-02-19T21:53:28.224 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-02-19T21:53:28.225 INFO:teuthology.orchestra.run.smithi086.stdout:Total 2.9 MB/s | 3.0 MB 00:01 2024-02-19T21:53:28.235 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-02-19T21:53:28.245 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-02-19T21:53:28.245 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-02-19T21:53:28.324 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install ceph-mgr 2024-02-19T21:53:28.407 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-02-19T21:53:28.410 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-02-19T21:53:28.701 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-02-19T21:53:28.904 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2024-02-19T21:53:29.115 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:01:25 ago on Mon 19 Feb 2024 09:52:04 PM UTC. 2024-02-19T21:53:29.141 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2024-02-19T21:53:29.350 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2024-02-19T21:53:29.479 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 4/6 2024-02-19T21:53:30.469 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 5/6 2024-02-19T21:53:30.729 INFO:teuthology.orchestra.run.smithi053.stdout:Package ceph-mgr-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-19T21:53:30.732 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2024-02-19T21:53:30.779 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-02-19T21:53:30.780 INFO:teuthology.orchestra.run.smithi053.stdout:Nothing to do. 2024-02-19T21:53:30.780 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-02-19T21:53:30.888 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install ceph-mgr-dashboard 2024-02-19T21:53:31.018 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2024-02-19T21:53:31.019 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 1/6 2024-02-19T21:53:31.019 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2/6 2024-02-19T21:53:31.019 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 3/6 2024-02-19T21:53:31.019 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2024-02-19T21:53:31.019 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2024-02-19T21:53:31.407 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2024-02-19T21:53:31.408 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T21:53:31.408 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-02-19T21:53:31.408 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 2024-02-19T21:53:31.408 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2024-02-19T21:53:31.408 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 2024-02-19T21:53:31.408 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-02-19T21:53:31.408 INFO:teuthology.orchestra.run.smithi086.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-02-19T21:53:31.408 INFO:teuthology.orchestra.run.smithi086.stdout: python3-routes-2.4.1-12.el8.noarch 2024-02-19T21:53:31.408 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T21:53:31.408 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-02-19T21:53:31.527 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-mgr-diskprediction-local 2024-02-19T21:53:31.673 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:01:27 ago on Mon 19 Feb 2024 09:52:04 PM UTC. 2024-02-19T21:53:32.297 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:29 ago on Mon 19 Feb 2024 09:52:03 PM UTC. 2024-02-19T21:53:33.331 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-02-19T21:53:33.332 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-02-19T21:53:33.332 INFO:teuthology.orchestra.run.smithi053.stdout: Package Arch Version Repository Size 2024-02-19T21:53:33.332 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-02-19T21:53:33.332 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2024-02-19T21:53:33.333 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-dashboard noarch 2:16.2.4-0.el8 ceph-noarch 2.7 M 2024-02-19T21:53:33.333 INFO:teuthology.orchestra.run.smithi053.stdout:Installing dependencies: 2024-02-19T21:53:33.333 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-grafana-dashboards noarch 2:16.2.4-0.el8 ceph-noarch 24 k 2024-02-19T21:53:33.333 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-prometheus-alerts noarch 2:16.2.4-0.el8 ceph-noarch 8.9 k 2024-02-19T21:53:33.333 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2024-02-19T21:53:33.333 INFO:teuthology.orchestra.run.smithi053.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-02-19T21:53:33.333 INFO:teuthology.orchestra.run.smithi053.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-02-19T21:53:33.333 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-02-19T21:53:33.333 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2024-02-19T21:53:33.333 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-02-19T21:53:33.333 INFO:teuthology.orchestra.run.smithi053.stdout:Install 6 Packages 2024-02-19T21:53:33.333 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-02-19T21:53:33.334 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 3.0 M 2024-02-19T21:53:33.334 INFO:teuthology.orchestra.run.smithi053.stdout:Installed size: 15 M 2024-02-19T21:53:33.334 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2024-02-19T21:53:33.749 INFO:teuthology.orchestra.run.smithi053.stdout:(1/6): ceph-grafana-dashboards-16.2.4-0.el8.noa 394 kB/s | 24 kB 00:00 2024-02-19T21:53:33.774 INFO:teuthology.orchestra.run.smithi053.stdout:(2/6): ceph-prometheus-alerts-16.2.4-0.el8.noar 104 kB/s | 8.9 kB 00:00 2024-02-19T21:53:33.832 INFO:teuthology.orchestra.run.smithi053.stdout:(3/6): ceph-mgr-dashboard-16.2.4-0.el8.noarch.r 18 MB/s | 2.7 MB 00:00 2024-02-19T21:53:33.849 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-02-19T21:53:33.850 INFO:teuthology.orchestra.run.smithi086.stdout:============================================================================================ 2024-02-19T21:53:33.851 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2024-02-19T21:53:33.851 INFO:teuthology.orchestra.run.smithi086.stdout:============================================================================================ 2024-02-19T21:53:33.851 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-02-19T21:53:33.851 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-diskprediction-local noarch 2:16.2.4-0.el8 ceph-noarch 8.6 M 2024-02-19T21:53:33.851 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2024-02-19T21:53:33.851 INFO:teuthology.orchestra.run.smithi086.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2024-02-19T21:53:33.851 INFO:teuthology.orchestra.run.smithi086.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2024-02-19T21:53:33.851 INFO:teuthology.orchestra.run.smithi086.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2024-02-19T21:53:33.851 INFO:teuthology.orchestra.run.smithi086.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2024-02-19T21:53:33.851 INFO:teuthology.orchestra.run.smithi086.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2024-02-19T21:53:33.851 INFO:teuthology.orchestra.run.smithi086.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2024-02-19T21:53:33.851 INFO:teuthology.orchestra.run.smithi086.stdout: python3-scipy x86_64 1.0.0-21.module_el8+762+77bd8591 appstream 14 M 2024-02-19T21:53:33.851 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T21:53:33.851 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-02-19T21:53:33.852 INFO:teuthology.orchestra.run.smithi086.stdout:============================================================================================ 2024-02-19T21:53:33.852 INFO:teuthology.orchestra.run.smithi086.stdout:Install 8 Packages 2024-02-19T21:53:33.852 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T21:53:33.852 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 37 M 2024-02-19T21:53:33.852 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 210 M 2024-02-19T21:53:33.852 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-02-19T21:53:33.857 INFO:teuthology.orchestra.run.smithi053.stdout:(4/6): python3-jwt-1.6.1-2.el8.noarch.rpm 391 kB/s | 43 kB 00:00 2024-02-19T21:53:34.041 INFO:teuthology.orchestra.run.smithi053.stdout:(5/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 126 kB/s | 34 kB 00:00 2024-02-19T21:53:34.208 INFO:teuthology.orchestra.run.smithi053.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 530 kB/s | 196 kB 00:00 2024-02-19T21:53:34.211 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2024-02-19T21:53:34.212 INFO:teuthology.orchestra.run.smithi053.stdout:Total 3.4 MB/s | 3.0 MB 00:00 2024-02-19T21:53:34.223 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2024-02-19T21:53:34.226 INFO:teuthology.orchestra.run.smithi086.stdout:(1/8): ceph-mgr-diskprediction-local-16.2.4-0.e 61 MB/s | 8.6 MB 00:00 2024-02-19T21:53:34.235 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2024-02-19T21:53:34.235 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2024-02-19T21:53:34.421 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2024-02-19T21:53:34.425 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2024-02-19T21:53:34.695 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2024-02-19T21:53:34.924 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2024-02-19T21:53:34.985 INFO:teuthology.orchestra.run.smithi086.stdout:(2/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 4.9 MB/s | 3.7 MB 00:00 2024-02-19T21:53:35.035 INFO:teuthology.orchestra.run.smithi086.stdout:(3/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 4.4 MB/s | 225 kB 00:00 2024-02-19T21:53:35.169 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2024-02-19T21:53:35.360 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2024-02-19T21:53:35.491 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 4/6 2024-02-19T21:53:36.524 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 5/6 2024-02-19T21:53:36.779 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2024-02-19T21:53:37.081 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2024-02-19T21:53:37.081 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 1/6 2024-02-19T21:53:37.081 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2/6 2024-02-19T21:53:37.081 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 3/6 2024-02-19T21:53:37.081 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2024-02-19T21:53:37.081 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2024-02-19T21:53:37.119 INFO:teuthology.orchestra.run.smithi086.stdout:(4/8): openblas-0.3.15-4.el8.x86_64.rpm 1.6 MB/s | 4.7 MB 00:03 2024-02-19T21:53:37.203 INFO:teuthology.orchestra.run.smithi086.stdout:(5/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 1.6 MB/s | 4.9 MB 00:03 2024-02-19T21:53:37.278 INFO:teuthology.orchestra.run.smithi086.stdout:(6/8): libquadmath-8.5.0-21.el8.x86_64.rpm 2.3 MB/s | 172 kB 00:00 2024-02-19T21:53:37.303 INFO:teuthology.orchestra.run.smithi086.stdout:(7/8): libgfortran-8.5.0-21.el8.x86_64.rpm 3.4 MB/s | 645 kB 00:00 2024-02-19T21:53:37.496 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2024-02-19T21:53:37.496 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-02-19T21:53:37.496 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2024-02-19T21:53:37.496 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 2024-02-19T21:53:37.496 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2024-02-19T21:53:37.496 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 2024-02-19T21:53:37.496 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-02-19T21:53:37.496 INFO:teuthology.orchestra.run.smithi053.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-02-19T21:53:37.497 INFO:teuthology.orchestra.run.smithi053.stdout: python3-routes-2.4.1-12.el8.noarch 2024-02-19T21:53:37.497 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-02-19T21:53:37.497 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-02-19T21:53:37.631 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install ceph-mgr-diskprediction-local 2024-02-19T21:53:38.419 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:01:34 ago on Mon 19 Feb 2024 09:52:04 PM UTC. 2024-02-19T21:53:38.622 INFO:teuthology.orchestra.run.smithi086.stdout:(8/8): python3-scipy-1.0.0-21.module_el8+762+77 4.0 MB/s | 14 MB 00:03 2024-02-19T21:53:38.623 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-02-19T21:53:38.623 INFO:teuthology.orchestra.run.smithi086.stdout:Total 7.8 MB/s | 37 MB 00:04 2024-02-19T21:53:38.856 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-02-19T21:53:38.874 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-02-19T21:53:38.874 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-02-19T21:53:39.203 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-02-19T21:53:39.206 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-02-19T21:53:39.819 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-02-19T21:53:39.947 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2024-02-19T21:53:40.068 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2024-02-19T21:53:40.083 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-02-19T21:53:40.084 INFO:teuthology.orchestra.run.smithi053.stdout:============================================================================================ 2024-02-19T21:53:40.084 INFO:teuthology.orchestra.run.smithi053.stdout: Package Arch Version Repository Size 2024-02-19T21:53:40.084 INFO:teuthology.orchestra.run.smithi053.stdout:============================================================================================ 2024-02-19T21:53:40.085 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2024-02-19T21:53:40.085 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-diskprediction-local noarch 2:16.2.4-0.el8 ceph-noarch 8.6 M 2024-02-19T21:53:40.085 INFO:teuthology.orchestra.run.smithi053.stdout:Installing dependencies: 2024-02-19T21:53:40.085 INFO:teuthology.orchestra.run.smithi053.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2024-02-19T21:53:40.085 INFO:teuthology.orchestra.run.smithi053.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2024-02-19T21:53:40.085 INFO:teuthology.orchestra.run.smithi053.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2024-02-19T21:53:40.085 INFO:teuthology.orchestra.run.smithi053.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2024-02-19T21:53:40.085 INFO:teuthology.orchestra.run.smithi053.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2024-02-19T21:53:40.085 INFO:teuthology.orchestra.run.smithi053.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2024-02-19T21:53:40.085 INFO:teuthology.orchestra.run.smithi053.stdout: python3-scipy x86_64 1.0.0-21.module_el8+762+77bd8591 appstream 14 M 2024-02-19T21:53:40.085 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-02-19T21:53:40.085 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2024-02-19T21:53:40.086 INFO:teuthology.orchestra.run.smithi053.stdout:============================================================================================ 2024-02-19T21:53:40.086 INFO:teuthology.orchestra.run.smithi053.stdout:Install 8 Packages 2024-02-19T21:53:40.086 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-02-19T21:53:40.087 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 37 M 2024-02-19T21:53:40.087 INFO:teuthology.orchestra.run.smithi053.stdout:Installed size: 210 M 2024-02-19T21:53:40.087 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2024-02-19T21:53:40.240 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2024-02-19T21:53:40.480 INFO:teuthology.orchestra.run.smithi053.stdout:(1/8): ceph-mgr-diskprediction-local-16.2.4-0.e 54 MB/s | 8.6 MB 00:00 2024-02-19T21:53:40.820 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2024-02-19T21:53:40.988 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2024-02-19T21:53:41.603 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2024-02-19T21:53:41.774 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-02-19T21:53:42.492 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-02-19T21:53:42.680 INFO:teuthology.orchestra.run.smithi053.stdout:(2/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 1.7 MB/s | 3.7 MB 00:02 2024-02-19T21:53:42.832 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2024-02-19T21:53:42.864 INFO:teuthology.orchestra.run.smithi053.stdout:(3/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 1.2 MB/s | 225 kB 00:00 2024-02-19T21:53:43.066 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-02-19T21:53:43.706 INFO:teuthology.orchestra.run.smithi053.stdout:(4/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 1.4 MB/s | 4.9 MB 00:03 2024-02-19T21:53:43.798 INFO:teuthology.orchestra.run.smithi053.stdout:(5/8): openblas-0.3.15-4.el8.x86_64.rpm 1.4 MB/s | 4.7 MB 00:03 2024-02-19T21:53:43.865 INFO:teuthology.orchestra.run.smithi053.stdout:(6/8): libgfortran-8.5.0-21.el8.x86_64.rpm 4.0 MB/s | 645 kB 00:00 2024-02-19T21:53:43.923 INFO:teuthology.orchestra.run.smithi053.stdout:(7/8): libquadmath-8.5.0-21.el8.x86_64.rpm 1.3 MB/s | 172 kB 00:00 2024-02-19T21:53:44.844 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-02-19T21:53:46.137 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 7/8 2024-02-19T21:53:46.359 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2024-02-19T21:53:46.672 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2024-02-19T21:53:46.672 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 1/8 2024-02-19T21:53:46.673 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2024-02-19T21:53:46.673 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2024-02-19T21:53:46.673 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2024-02-19T21:53:46.673 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2024-02-19T21:53:46.673 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 6/8 2024-02-19T21:53:46.673 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2024-02-19T21:53:47.080 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2024-02-19T21:53:47.080 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T21:53:47.080 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-02-19T21:53:47.080 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 2024-02-19T21:53:47.080 INFO:teuthology.orchestra.run.smithi086.stdout: libgfortran-8.5.0-21.el8.x86_64 2024-02-19T21:53:47.080 INFO:teuthology.orchestra.run.smithi086.stdout: libquadmath-8.5.0-21.el8.x86_64 2024-02-19T21:53:47.080 INFO:teuthology.orchestra.run.smithi086.stdout: openblas-0.3.15-4.el8.x86_64 2024-02-19T21:53:47.081 INFO:teuthology.orchestra.run.smithi086.stdout: openblas-threads-0.3.15-4.el8.x86_64 2024-02-19T21:53:47.081 INFO:teuthology.orchestra.run.smithi086.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-02-19T21:53:47.081 INFO:teuthology.orchestra.run.smithi086.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-02-19T21:53:47.081 INFO:teuthology.orchestra.run.smithi086.stdout: python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_64 2024-02-19T21:53:47.081 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T21:53:47.081 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-02-19T21:53:47.195 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-mgr-rook 2024-02-19T21:53:47.955 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:44 ago on Mon 19 Feb 2024 09:52:03 PM UTC. 2024-02-19T21:53:48.999 INFO:teuthology.orchestra.run.smithi053.stdout:(8/8): python3-scipy-1.0.0-21.module_el8+762+77 2.3 MB/s | 14 MB 00:06 2024-02-19T21:53:48.999 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2024-02-19T21:53:49.000 INFO:teuthology.orchestra.run.smithi053.stdout:Total 4.2 MB/s | 37 MB 00:08 2024-02-19T21:53:49.239 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2024-02-19T21:53:49.258 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2024-02-19T21:53:49.258 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2024-02-19T21:53:49.502 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-02-19T21:53:49.504 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-02-19T21:53:49.504 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2024-02-19T21:53:49.504 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-02-19T21:53:49.504 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-02-19T21:53:49.504 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-rook noarch 2:16.2.4-0.el8 ceph-noarch 27 k 2024-02-19T21:53:49.504 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2024-02-19T21:53:49.505 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-02-19T21:53:49.505 INFO:teuthology.orchestra.run.smithi086.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-02-19T21:53:49.505 INFO:teuthology.orchestra.run.smithi086.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-02-19T21:53:49.505 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2024-02-19T21:53:49.505 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2024-02-19T21:53:49.505 INFO:teuthology.orchestra.run.smithi086.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-02-19T21:53:49.505 INFO:teuthology.orchestra.run.smithi086.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2024-02-19T21:53:49.505 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2024-02-19T21:53:49.505 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2024-02-19T21:53:49.505 INFO:teuthology.orchestra.run.smithi086.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2024-02-19T21:53:49.505 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-02-19T21:53:49.505 INFO:teuthology.orchestra.run.smithi086.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-02-19T21:53:49.506 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T21:53:49.506 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-02-19T21:53:49.506 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-02-19T21:53:49.506 INFO:teuthology.orchestra.run.smithi086.stdout:Install 13 Packages 2024-02-19T21:53:49.506 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T21:53:49.506 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 1.9 M 2024-02-19T21:53:49.506 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 24 M 2024-02-19T21:53:49.507 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-02-19T21:53:49.610 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2024-02-19T21:53:49.613 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2024-02-19T21:53:50.069 INFO:teuthology.orchestra.run.smithi086.stdout:(1/13): ceph-mgr-rook-16.2.4-0.el8.noarch.rpm 667 kB/s | 27 kB 00:00 2024-02-19T21:53:50.111 INFO:teuthology.orchestra.run.smithi086.stdout:(2/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 325 kB/s | 27 kB 00:00 2024-02-19T21:53:50.136 INFO:teuthology.orchestra.run.smithi086.stdout:(3/13): python3-jsonpointer-1.10-11.el8.noarch. 187 kB/s | 20 kB 00:00 2024-02-19T21:53:50.170 INFO:teuthology.orchestra.run.smithi086.stdout:(4/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 1.2 MB/s | 126 kB 00:00 2024-02-19T21:53:50.195 INFO:teuthology.orchestra.run.smithi086.stdout:(5/13): python3-pyasn1-modules-0.3.7-6.el8.noar 1.8 MB/s | 110 kB 00:00 2024-02-19T21:53:50.220 INFO:teuthology.orchestra.run.smithi086.stdout:(6/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 1.8 MB/s | 155 kB 00:00 2024-02-19T21:53:50.236 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2024-02-19T21:53:50.246 INFO:teuthology.orchestra.run.smithi086.stdout:(7/13): python3-requests-oauthlib-1.0.0-1.el8.n 565 kB/s | 43 kB 00:00 2024-02-19T21:53:50.372 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2024-02-19T21:53:50.499 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2024-02-19T21:53:50.638 INFO:teuthology.orchestra.run.smithi086.stdout:(8/13): python3-certifi-2018.10.15-7.el8.noarch 39 kB/s | 16 kB 00:00 2024-02-19T21:53:50.651 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2024-02-19T21:53:50.663 INFO:teuthology.orchestra.run.smithi086.stdout:(9/13): python3-cachetools-3.1.1-4.el8.noarch.r 71 kB/s | 33 kB 00:00 2024-02-19T21:53:50.688 INFO:teuthology.orchestra.run.smithi086.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 184 kB/s | 81 kB 00:00 2024-02-19T21:53:50.797 INFO:teuthology.orchestra.run.smithi086.stdout:(11/13): python3-rsa-4.9-2.el8.noarch.rpm 481 kB/s | 64 kB 00:00 2024-02-19T21:53:50.822 INFO:teuthology.orchestra.run.smithi086.stdout:(12/13): python3-websocket-client-0.56.0-5.el8. 458 kB/s | 61 kB 00:00 2024-02-19T21:53:50.856 INFO:teuthology.orchestra.run.smithi086.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 5.4 MB/s | 1.2 MB 00:00 2024-02-19T21:53:50.858 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-02-19T21:53:50.858 INFO:teuthology.orchestra.run.smithi086.stdout:Total 1.4 MB/s | 1.9 MB 00:01 2024-02-19T21:53:50.917 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-02-19T21:53:50.933 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-02-19T21:53:50.933 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-02-19T21:53:51.020 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-02-19T21:53:51.023 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-02-19T21:53:51.286 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2024-02-19T21:53:51.522 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2024-02-19T21:53:51.544 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-02-19T21:53:51.806 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2024-02-19T21:53:52.015 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2024-02-19T21:53:52.182 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2024-02-19T21:53:52.238 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2024-02-19T21:53:52.347 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-02-19T21:53:52.485 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2024-02-19T21:53:52.692 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2024-02-19T21:53:52.917 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2024-02-19T21:53:53.171 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-02-19T21:53:53.246 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2024-02-19T21:53:53.493 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2024-02-19T21:53:53.526 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2024-02-19T21:53:53.755 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-02-19T21:53:54.521 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2024-02-19T21:53:54.835 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2024-02-19T21:53:55.030 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2024-02-19T21:53:55.211 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2024-02-19T21:53:55.420 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2024-02-19T21:53:55.711 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2024-02-19T21:53:55.711 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mgr-rook-2:16.2.4-0.el8.noarch 1/13 2024-02-19T21:53:55.711 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2024-02-19T21:53:55.711 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2024-02-19T21:53:55.711 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2024-02-19T21:53:55.712 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2024-02-19T21:53:55.712 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2024-02-19T21:53:55.712 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2024-02-19T21:53:55.712 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2024-02-19T21:53:55.712 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2024-02-19T21:53:55.712 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2024-02-19T21:53:55.712 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2024-02-19T21:53:55.712 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2024-02-19T21:53:55.823 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-02-19T21:53:56.163 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2024-02-19T21:53:56.163 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T21:53:56.163 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-02-19T21:53:56.163 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-rook-2:16.2.4-0.el8.noarch 2024-02-19T21:53:56.163 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-02-19T21:53:56.163 INFO:teuthology.orchestra.run.smithi086.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-02-19T21:53:56.164 INFO:teuthology.orchestra.run.smithi086.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-02-19T21:53:56.164 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-02-19T21:53:56.164 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-02-19T21:53:56.164 INFO:teuthology.orchestra.run.smithi086.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-02-19T21:53:56.164 INFO:teuthology.orchestra.run.smithi086.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-02-19T21:53:56.164 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-02-19T21:53:56.164 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-02-19T21:53:56.164 INFO:teuthology.orchestra.run.smithi086.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-02-19T21:53:56.164 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rsa-4.9-2.el8.noarch 2024-02-19T21:53:56.164 INFO:teuthology.orchestra.run.smithi086.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-02-19T21:53:56.164 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T21:53:56.164 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-02-19T21:53:56.282 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-mgr-cephadm 2024-02-19T21:53:57.030 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:54 ago on Mon 19 Feb 2024 09:52:03 PM UTC. 2024-02-19T21:53:57.180 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 7/8 2024-02-19T21:53:57.411 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2024-02-19T21:53:57.751 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2024-02-19T21:53:57.751 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 1/8 2024-02-19T21:53:57.751 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2024-02-19T21:53:57.751 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2024-02-19T21:53:57.751 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2024-02-19T21:53:57.751 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2024-02-19T21:53:57.751 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 6/8 2024-02-19T21:53:57.751 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2024-02-19T21:53:58.167 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2024-02-19T21:53:58.167 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-02-19T21:53:58.167 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2024-02-19T21:53:58.168 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 2024-02-19T21:53:58.168 INFO:teuthology.orchestra.run.smithi053.stdout: libgfortran-8.5.0-21.el8.x86_64 2024-02-19T21:53:58.168 INFO:teuthology.orchestra.run.smithi053.stdout: libquadmath-8.5.0-21.el8.x86_64 2024-02-19T21:53:58.168 INFO:teuthology.orchestra.run.smithi053.stdout: openblas-0.3.15-4.el8.x86_64 2024-02-19T21:53:58.168 INFO:teuthology.orchestra.run.smithi053.stdout: openblas-threads-0.3.15-4.el8.x86_64 2024-02-19T21:53:58.168 INFO:teuthology.orchestra.run.smithi053.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-02-19T21:53:58.168 INFO:teuthology.orchestra.run.smithi053.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-02-19T21:53:58.168 INFO:teuthology.orchestra.run.smithi053.stdout: python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_64 2024-02-19T21:53:58.168 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-02-19T21:53:58.168 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-02-19T21:53:58.280 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install ceph-mgr-rook 2024-02-19T21:53:58.569 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-02-19T21:53:58.570 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-02-19T21:53:58.570 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2024-02-19T21:53:58.570 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-02-19T21:53:58.570 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-02-19T21:53:58.570 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-cephadm noarch 2:16.2.4-0.el8 ceph-noarch 89 k 2024-02-19T21:53:58.570 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2024-02-19T21:53:58.570 INFO:teuthology.orchestra.run.smithi086.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2024-02-19T21:53:58.571 INFO:teuthology.orchestra.run.smithi086.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2024-02-19T21:53:58.571 INFO:teuthology.orchestra.run.smithi086.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2024-02-19T21:53:58.571 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jinja2 noarch 2.10.1-4.el8 appstream 578 k 2024-02-19T21:53:58.571 INFO:teuthology.orchestra.run.smithi086.stdout: python3-remoto noarch 1.2.1-1.el8 epel 49 k 2024-02-19T21:53:58.571 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T21:53:58.571 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-02-19T21:53:58.571 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-02-19T21:53:58.571 INFO:teuthology.orchestra.run.smithi086.stdout:Install 6 Packages 2024-02-19T21:53:58.571 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T21:53:58.572 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 5.9 M 2024-02-19T21:53:58.572 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 24 M 2024-02-19T21:53:58.572 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-02-19T21:53:58.841 INFO:teuthology.orchestra.run.smithi086.stdout:(1/6): ceph-mgr-cephadm-16.2.4-0.el8.noarch.rpm 2.0 MB/s | 89 kB 00:00 2024-02-19T21:53:59.008 INFO:teuthology.orchestra.run.smithi086.stdout:(2/6): python3-jinja2-2.10.1-4.el8.noarch.rpm 2.7 MB/s | 578 kB 00:00 2024-02-19T21:53:59.071 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:01:55 ago on Mon 19 Feb 2024 09:52:04 PM UTC. 2024-02-19T21:53:59.116 INFO:teuthology.orchestra.run.smithi086.stdout:(3/6): python3-apipkg-1.5-6.el8.noarch.rpm 77 kB/s | 21 kB 00:00 2024-02-19T21:53:59.292 INFO:teuthology.orchestra.run.smithi086.stdout:(4/6): python3-remoto-1.2.1-1.el8.noarch.rpm 279 kB/s | 49 kB 00:00 2024-02-19T21:53:59.609 INFO:teuthology.orchestra.run.smithi086.stdout:(5/6): python3-execnet-1.7.1-1.el8.noarch.rpm 591 kB/s | 355 kB 00:00 2024-02-19T21:54:00.459 INFO:teuthology.orchestra.run.smithi086.stdout:(6/6): python3-babel-2.5.1-7.el8.noarch.rpm 2.9 MB/s | 4.8 MB 00:01 2024-02-19T21:54:00.460 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-02-19T21:54:00.460 INFO:teuthology.orchestra.run.smithi086.stdout:Total 3.1 MB/s | 5.9 MB 00:01 2024-02-19T21:54:00.514 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-02-19T21:54:00.524 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-02-19T21:54:00.524 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-02-19T21:54:00.629 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-02-19T21:54:00.632 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-02-19T21:54:00.745 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-02-19T21:54:00.746 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-02-19T21:54:00.747 INFO:teuthology.orchestra.run.smithi053.stdout: Package Arch Version Repository Size 2024-02-19T21:54:00.747 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-02-19T21:54:00.747 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2024-02-19T21:54:00.747 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-rook noarch 2:16.2.4-0.el8 ceph-noarch 27 k 2024-02-19T21:54:00.747 INFO:teuthology.orchestra.run.smithi053.stdout:Installing dependencies: 2024-02-19T21:54:00.747 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-02-19T21:54:00.747 INFO:teuthology.orchestra.run.smithi053.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-02-19T21:54:00.747 INFO:teuthology.orchestra.run.smithi053.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-02-19T21:54:00.747 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2024-02-19T21:54:00.747 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2024-02-19T21:54:00.748 INFO:teuthology.orchestra.run.smithi053.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-02-19T21:54:00.748 INFO:teuthology.orchestra.run.smithi053.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2024-02-19T21:54:00.748 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2024-02-19T21:54:00.748 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2024-02-19T21:54:00.748 INFO:teuthology.orchestra.run.smithi053.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2024-02-19T21:54:00.748 INFO:teuthology.orchestra.run.smithi053.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-02-19T21:54:00.748 INFO:teuthology.orchestra.run.smithi053.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-02-19T21:54:00.748 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-02-19T21:54:00.748 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2024-02-19T21:54:00.748 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-02-19T21:54:00.748 INFO:teuthology.orchestra.run.smithi053.stdout:Install 13 Packages 2024-02-19T21:54:00.748 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-02-19T21:54:00.749 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 1.9 M 2024-02-19T21:54:00.749 INFO:teuthology.orchestra.run.smithi053.stdout:Installed size: 24 M 2024-02-19T21:54:00.749 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2024-02-19T21:54:00.899 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-02-19T21:54:01.152 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2024-02-19T21:54:01.196 INFO:teuthology.orchestra.run.smithi053.stdout:(1/13): ceph-mgr-rook-16.2.4-0.el8.noarch.rpm 658 kB/s | 27 kB 00:00 2024-02-19T21:54:01.271 INFO:teuthology.orchestra.run.smithi053.stdout:(2/13): python3-jsonpointer-1.10-11.el8.noarch. 172 kB/s | 20 kB 00:00 2024-02-19T21:54:01.297 INFO:teuthology.orchestra.run.smithi053.stdout:(3/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 188 kB/s | 27 kB 00:00 2024-02-19T21:54:01.372 INFO:teuthology.orchestra.run.smithi053.stdout:(4/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 2.0 MB/s | 155 kB 00:00 2024-02-19T21:54:01.397 INFO:teuthology.orchestra.run.smithi053.stdout:(5/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 630 kB/s | 126 kB 00:00 2024-02-19T21:54:01.422 INFO:teuthology.orchestra.run.smithi053.stdout:(6/13): python3-pyasn1-modules-0.3.7-6.el8.noar 729 kB/s | 110 kB 00:00 2024-02-19T21:54:01.447 INFO:teuthology.orchestra.run.smithi053.stdout:(7/13): python3-requests-oauthlib-1.0.0-1.el8.n 564 kB/s | 43 kB 00:00 2024-02-19T21:54:01.455 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2024-02-19T21:54:01.514 INFO:teuthology.orchestra.run.smithi053.stdout:(8/13): python3-certifi-2018.10.15-7.el8.noarch 177 kB/s | 16 kB 00:00 2024-02-19T21:54:01.548 INFO:teuthology.orchestra.run.smithi053.stdout:(9/13): python3-cachetools-3.1.1-4.el8.noarch.r 218 kB/s | 33 kB 00:00 2024-02-19T21:54:01.573 INFO:teuthology.orchestra.run.smithi053.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 647 kB/s | 81 kB 00:00 2024-02-19T21:54:01.599 INFO:teuthology.orchestra.run.smithi053.stdout:(11/13): python3-rsa-4.9-2.el8.noarch.rpm 1.3 MB/s | 64 kB 00:00 2024-02-19T21:54:01.624 INFO:teuthology.orchestra.run.smithi053.stdout:(12/13): python3-websocket-client-0.56.0-5.el8. 1.2 MB/s | 61 kB 00:00 2024-02-19T21:54:01.691 INFO:teuthology.orchestra.run.smithi053.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 6.6 MB/s | 1.2 MB 00:00 2024-02-19T21:54:01.693 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2024-02-19T21:54:01.694 INFO:teuthology.orchestra.run.smithi053.stdout:Total 2.0 MB/s | 1.9 MB 00:00 2024-02-19T21:54:01.752 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2024-02-19T21:54:01.768 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2024-02-19T21:54:01.769 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2024-02-19T21:54:01.863 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2024-02-19T21:54:01.866 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2024-02-19T21:54:02.402 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-remoto-1.2.1-1.el8.noarch 3/6 2024-02-19T21:54:02.404 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2024-02-19T21:54:02.609 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2024-02-19T21:54:02.733 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 4/6 2024-02-19T21:54:02.906 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2024-02-19T21:54:02.931 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jinja2-2.10.1-4.el8.noarch 5/6 2024-02-19T21:54:03.141 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2024-02-19T21:54:03.168 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2024-02-19T21:54:03.336 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2024-02-19T21:54:03.453 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2024-02-19T21:54:03.453 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 1/6 2024-02-19T21:54:03.454 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/6 2024-02-19T21:54:03.454 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jinja2-2.10.1-4.el8.noarch 3/6 2024-02-19T21:54:03.454 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2024-02-19T21:54:03.454 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2024-02-19T21:54:03.527 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2024-02-19T21:54:03.770 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2024-02-19T21:54:03.834 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-remoto-1.2.1-1.el8.noarch 6/6 2024-02-19T21:54:03.834 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T21:54:03.834 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-02-19T21:54:03.834 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch python3-apipkg-1.5-6.el8.noarch 2024-02-19T21:54:03.834 INFO:teuthology.orchestra.run.smithi086.stdout: python3-babel-2.5.1-7.el8.noarch python3-execnet-1.7.1-1.el8.noarch 2024-02-19T21:54:03.835 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jinja2-2.10.1-4.el8.noarch python3-remoto-1.2.1-1.el8.noarch 2024-02-19T21:54:03.835 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T21:54:03.835 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-02-19T21:54:03.967 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-fuse 2024-02-19T21:54:04.037 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2024-02-19T21:54:04.285 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2024-02-19T21:54:04.729 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:02:01 ago on Mon 19 Feb 2024 09:52:03 PM UTC. 2024-02-19T21:54:05.384 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2024-02-19T21:54:05.635 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2024-02-19T21:54:05.819 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2024-02-19T21:54:05.994 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2024-02-19T21:54:06.161 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2024-02-19T21:54:06.282 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-02-19T21:54:06.283 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-02-19T21:54:06.283 INFO:teuthology.orchestra.run.smithi086.stdout: Package Architecture Version Repository Size 2024-02-19T21:54:06.283 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-02-19T21:54:06.283 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-02-19T21:54:06.283 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-fuse x86_64 2:16.2.4-0.el8 ceph 885 k 2024-02-19T21:54:06.283 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2024-02-19T21:54:06.283 INFO:teuthology.orchestra.run.smithi086.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2024-02-19T21:54:06.283 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T21:54:06.284 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-02-19T21:54:06.284 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-02-19T21:54:06.284 INFO:teuthology.orchestra.run.smithi086.stdout:Install 2 Packages 2024-02-19T21:54:06.284 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T21:54:06.284 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 968 k 2024-02-19T21:54:06.285 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 3.2 M 2024-02-19T21:54:06.285 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-02-19T21:54:06.379 INFO:teuthology.orchestra.run.smithi086.stdout:(1/2): ceph-fuse-16.2.4-0.el8.x86_64.rpm 16 MB/s | 885 kB 00:00 2024-02-19T21:54:06.404 INFO:teuthology.orchestra.run.smithi086.stdout:(2/2): fuse-2.9.7-17.el8.x86_64.rpm 1.0 MB/s | 83 kB 00:00 2024-02-19T21:54:06.404 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-02-19T21:54:06.405 INFO:teuthology.orchestra.run.smithi086.stdout:Total 7.8 MB/s | 968 kB 00:00 2024-02-19T21:54:06.414 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-02-19T21:54:06.435 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-02-19T21:54:06.435 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-02-19T21:54:06.474 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2024-02-19T21:54:06.474 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-mgr-rook-2:16.2.4-0.el8.noarch 1/13 2024-02-19T21:54:06.474 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2024-02-19T21:54:06.475 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2024-02-19T21:54:06.475 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2024-02-19T21:54:06.475 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2024-02-19T21:54:06.475 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2024-02-19T21:54:06.475 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2024-02-19T21:54:06.475 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2024-02-19T21:54:06.475 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2024-02-19T21:54:06.475 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2024-02-19T21:54:06.475 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2024-02-19T21:54:06.475 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2024-02-19T21:54:06.522 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-02-19T21:54:06.526 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-02-19T21:54:06.707 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-02-19T21:54:06.958 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2024-02-19T21:54:06.958 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-02-19T21:54:06.958 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2024-02-19T21:54:06.958 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-rook-2:16.2.4-0.el8.noarch 2024-02-19T21:54:06.958 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-02-19T21:54:06.958 INFO:teuthology.orchestra.run.smithi053.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-02-19T21:54:06.958 INFO:teuthology.orchestra.run.smithi053.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-02-19T21:54:06.958 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-02-19T21:54:06.958 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-02-19T21:54:06.959 INFO:teuthology.orchestra.run.smithi053.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-02-19T21:54:06.959 INFO:teuthology.orchestra.run.smithi053.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-02-19T21:54:06.959 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-02-19T21:54:06.959 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-02-19T21:54:06.959 INFO:teuthology.orchestra.run.smithi053.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-02-19T21:54:06.959 INFO:teuthology.orchestra.run.smithi053.stdout: python3-rsa-4.9-2.el8.noarch 2024-02-19T21:54:06.959 INFO:teuthology.orchestra.run.smithi053.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-02-19T21:54:06.959 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-02-19T21:54:06.959 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-02-19T21:54:06.959 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2024-02-19T21:54:07.152 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2024-02-19T21:54:07.182 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install ceph-mgr-cephadm 2024-02-19T21:54:07.573 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2024-02-19T21:54:07.573 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-fuse-2:16.2.4-0.el8.x86_64 1/2 2024-02-19T21:54:07.972 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:02:03 ago on Mon 19 Feb 2024 09:52:04 PM UTC. 2024-02-19T21:54:07.992 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2024-02-19T21:54:07.992 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T21:54:07.992 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-02-19T21:54:07.992 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-fuse-2:16.2.4-0.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2024-02-19T21:54:07.992 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T21:54:07.993 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-02-19T21:54:08.182 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install librados-devel 2024-02-19T21:54:08.946 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:02:05 ago on Mon 19 Feb 2024 09:52:03 PM UTC. 2024-02-19T21:54:09.626 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-02-19T21:54:09.627 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-02-19T21:54:09.627 INFO:teuthology.orchestra.run.smithi053.stdout: Package Arch Version Repository Size 2024-02-19T21:54:09.628 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-02-19T21:54:09.628 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2024-02-19T21:54:09.628 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-cephadm noarch 2:16.2.4-0.el8 ceph-noarch 89 k 2024-02-19T21:54:09.628 INFO:teuthology.orchestra.run.smithi053.stdout:Installing dependencies: 2024-02-19T21:54:09.628 INFO:teuthology.orchestra.run.smithi053.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2024-02-19T21:54:09.628 INFO:teuthology.orchestra.run.smithi053.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2024-02-19T21:54:09.628 INFO:teuthology.orchestra.run.smithi053.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2024-02-19T21:54:09.628 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jinja2 noarch 2.10.1-4.el8 appstream 578 k 2024-02-19T21:54:09.628 INFO:teuthology.orchestra.run.smithi053.stdout: python3-remoto noarch 1.2.1-1.el8 epel 49 k 2024-02-19T21:54:09.628 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-02-19T21:54:09.629 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2024-02-19T21:54:09.629 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-02-19T21:54:09.629 INFO:teuthology.orchestra.run.smithi053.stdout:Install 6 Packages 2024-02-19T21:54:09.629 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-02-19T21:54:09.629 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 5.9 M 2024-02-19T21:54:09.629 INFO:teuthology.orchestra.run.smithi053.stdout:Installed size: 24 M 2024-02-19T21:54:09.630 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2024-02-19T21:54:10.033 INFO:teuthology.orchestra.run.smithi053.stdout:(1/6): ceph-mgr-cephadm-16.2.4-0.el8.noarch.rpm 2.1 MB/s | 89 kB 00:00 2024-02-19T21:54:10.234 INFO:teuthology.orchestra.run.smithi053.stdout:(2/6): python3-apipkg-1.5-6.el8.noarch.rpm 106 kB/s | 21 kB 00:00 2024-02-19T21:54:10.506 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-02-19T21:54:10.506 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-02-19T21:54:10.506 INFO:teuthology.orchestra.run.smithi086.stdout: Package Architecture Version Repository Size 2024-02-19T21:54:10.506 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-02-19T21:54:10.506 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-02-19T21:54:10.506 INFO:teuthology.orchestra.run.smithi086.stdout: librados-devel x86_64 2:16.2.4-0.el8 ceph 131 k 2024-02-19T21:54:10.506 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T21:54:10.507 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-02-19T21:54:10.507 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-02-19T21:54:10.507 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2024-02-19T21:54:10.507 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T21:54:10.507 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 131 k 2024-02-19T21:54:10.507 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 439 k 2024-02-19T21:54:10.507 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-02-19T21:54:10.550 INFO:teuthology.orchestra.run.smithi086.stdout:librados-devel-16.2.4-0.el8.x86_64.rpm 3.0 MB/s | 131 kB 00:00 2024-02-19T21:54:10.551 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-02-19T21:54:10.551 INFO:teuthology.orchestra.run.smithi086.stdout:Total 2.9 MB/s | 131 kB 00:00 2024-02-19T21:54:10.551 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-02-19T21:54:10.556 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-02-19T21:54:10.556 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-02-19T21:54:10.605 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-02-19T21:54:10.608 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-02-19T21:54:10.722 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-02-19T21:54:10.959 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T21:54:11.250 INFO:teuthology.orchestra.run.smithi053.stdout:(3/6): python3-jinja2-2.10.1-4.el8.noarch.rpm 460 kB/s | 578 kB 00:01 2024-02-19T21:54:11.251 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T21:54:11.275 INFO:teuthology.orchestra.run.smithi053.stdout:(4/6): python3-execnet-1.7.1-1.el8.noarch.rpm 341 kB/s | 355 kB 00:01 2024-02-19T21:54:11.459 INFO:teuthology.orchestra.run.smithi053.stdout:(5/6): python3-remoto-1.2.1-1.el8.noarch.rpm 235 kB/s | 49 kB 00:00 2024-02-19T21:54:11.622 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T21:54:11.622 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T21:54:11.622 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-02-19T21:54:11.622 INFO:teuthology.orchestra.run.smithi086.stdout: librados-devel-2:16.2.4-0.el8.x86_64 2024-02-19T21:54:11.622 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T21:54:11.622 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-02-19T21:54:11.855 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install libcephfs2 2024-02-19T21:54:11.892 INFO:teuthology.orchestra.run.smithi053.stdout:(6/6): python3-babel-2.5.1-7.el8.noarch.rpm 2.5 MB/s | 4.8 MB 00:01 2024-02-19T21:54:11.895 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2024-02-19T21:54:11.895 INFO:teuthology.orchestra.run.smithi053.stdout:Total 2.6 MB/s | 5.9 MB 00:02 2024-02-19T21:54:11.950 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2024-02-19T21:54:11.961 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2024-02-19T21:54:11.961 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2024-02-19T21:54:12.076 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2024-02-19T21:54:12.079 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2024-02-19T21:54:12.355 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2024-02-19T21:54:12.607 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2024-02-19T21:54:12.620 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:02:09 ago on Mon 19 Feb 2024 09:52:03 PM UTC. 2024-02-19T21:54:12.900 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2024-02-19T21:54:13.861 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-remoto-1.2.1-1.el8.noarch 3/6 2024-02-19T21:54:14.100 INFO:teuthology.orchestra.run.smithi086.stdout:Package libcephfs2-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-19T21:54:14.146 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-02-19T21:54:14.147 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2024-02-19T21:54:14.147 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-02-19T21:54:14.224 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 4/6 2024-02-19T21:54:14.250 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install libcephfs-devel 2024-02-19T21:54:14.441 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-jinja2-2.10.1-4.el8.noarch 5/6 2024-02-19T21:54:14.660 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2024-02-19T21:54:14.957 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2024-02-19T21:54:14.957 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 1/6 2024-02-19T21:54:14.957 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/6 2024-02-19T21:54:14.958 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-jinja2-2.10.1-4.el8.noarch 3/6 2024-02-19T21:54:14.958 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2024-02-19T21:54:14.958 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2024-02-19T21:54:15.003 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:02:12 ago on Mon 19 Feb 2024 09:52:03 PM UTC. 2024-02-19T21:54:15.396 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-remoto-1.2.1-1.el8.noarch 6/6 2024-02-19T21:54:15.397 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-02-19T21:54:15.397 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2024-02-19T21:54:15.397 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch python3-apipkg-1.5-6.el8.noarch 2024-02-19T21:54:15.397 INFO:teuthology.orchestra.run.smithi053.stdout: python3-babel-2.5.1-7.el8.noarch python3-execnet-1.7.1-1.el8.noarch 2024-02-19T21:54:15.397 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jinja2-2.10.1-4.el8.noarch python3-remoto-1.2.1-1.el8.noarch 2024-02-19T21:54:15.397 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-02-19T21:54:15.397 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-02-19T21:54:15.531 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install ceph-fuse 2024-02-19T21:54:16.321 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:02:12 ago on Mon 19 Feb 2024 09:52:04 PM UTC. 2024-02-19T21:54:16.562 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-02-19T21:54:16.563 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-02-19T21:54:16.563 INFO:teuthology.orchestra.run.smithi086.stdout: Package Architecture Version Repository Size 2024-02-19T21:54:16.564 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-02-19T21:54:16.564 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-02-19T21:54:16.564 INFO:teuthology.orchestra.run.smithi086.stdout: libcephfs-devel x86_64 2:16.2.4-0.el8 ceph 24 k 2024-02-19T21:54:16.564 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T21:54:16.564 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-02-19T21:54:16.565 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-02-19T21:54:16.565 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2024-02-19T21:54:16.565 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T21:54:16.565 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 24 k 2024-02-19T21:54:16.565 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 94 k 2024-02-19T21:54:16.565 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-02-19T21:54:17.229 INFO:teuthology.orchestra.run.smithi086.stdout:libcephfs-devel-16.2.4-0.el8.x86_64.rpm 36 kB/s | 24 kB 00:00 2024-02-19T21:54:17.230 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-02-19T21:54:17.230 INFO:teuthology.orchestra.run.smithi086.stdout:Total 36 kB/s | 24 kB 00:00 2024-02-19T21:54:17.230 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-02-19T21:54:17.234 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-02-19T21:54:17.234 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-02-19T21:54:17.237 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-02-19T21:54:17.240 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-02-19T21:54:17.299 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-02-19T21:54:17.525 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T21:54:17.785 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T21:54:17.976 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-02-19T21:54:17.977 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-02-19T21:54:17.977 INFO:teuthology.orchestra.run.smithi053.stdout: Package Architecture Version Repository Size 2024-02-19T21:54:17.977 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-02-19T21:54:17.977 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2024-02-19T21:54:17.977 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-fuse x86_64 2:16.2.4-0.el8 ceph 885 k 2024-02-19T21:54:17.977 INFO:teuthology.orchestra.run.smithi053.stdout:Installing dependencies: 2024-02-19T21:54:17.977 INFO:teuthology.orchestra.run.smithi053.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2024-02-19T21:54:17.977 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-02-19T21:54:17.977 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2024-02-19T21:54:17.977 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-02-19T21:54:17.977 INFO:teuthology.orchestra.run.smithi053.stdout:Install 2 Packages 2024-02-19T21:54:17.977 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-02-19T21:54:17.978 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 968 k 2024-02-19T21:54:17.978 INFO:teuthology.orchestra.run.smithi053.stdout:Installed size: 3.2 M 2024-02-19T21:54:17.978 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2024-02-19T21:54:18.076 INFO:teuthology.orchestra.run.smithi053.stdout:(1/2): ceph-fuse-16.2.4-0.el8.x86_64.rpm 15 MB/s | 885 kB 00:00 2024-02-19T21:54:18.093 INFO:teuthology.orchestra.run.smithi053.stdout:(2/2): fuse-2.9.7-17.el8.x86_64.rpm 1.1 MB/s | 83 kB 00:00 2024-02-19T21:54:18.093 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2024-02-19T21:54:18.094 INFO:teuthology.orchestra.run.smithi053.stdout:Total 8.2 MB/s | 968 kB 00:00 2024-02-19T21:54:18.104 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2024-02-19T21:54:18.126 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2024-02-19T21:54:18.126 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2024-02-19T21:54:18.133 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T21:54:18.133 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T21:54:18.133 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-02-19T21:54:18.133 INFO:teuthology.orchestra.run.smithi086.stdout: libcephfs-devel-2:16.2.4-0.el8.x86_64 2024-02-19T21:54:18.134 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T21:54:18.134 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-02-19T21:54:18.218 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2024-02-19T21:54:18.221 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2024-02-19T21:54:18.254 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install librados2 2024-02-19T21:54:18.426 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2024-02-19T21:54:18.682 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2024-02-19T21:54:18.868 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2024-02-19T21:54:19.006 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:02:16 ago on Mon 19 Feb 2024 09:52:03 PM UTC. 2024-02-19T21:54:19.280 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2024-02-19T21:54:19.280 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-fuse-2:16.2.4-0.el8.x86_64 1/2 2024-02-19T21:54:19.711 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2024-02-19T21:54:19.711 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-02-19T21:54:19.711 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2024-02-19T21:54:19.711 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-fuse-2:16.2.4-0.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2024-02-19T21:54:19.711 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-02-19T21:54:19.712 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-02-19T21:54:19.918 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install librados-devel 2024-02-19T21:54:20.524 INFO:teuthology.orchestra.run.smithi086.stdout:Package librados2-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-19T21:54:20.571 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-02-19T21:54:20.572 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2024-02-19T21:54:20.572 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-02-19T21:54:20.669 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install librbd1 2024-02-19T21:54:20.716 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:02:16 ago on Mon 19 Feb 2024 09:52:04 PM UTC. 2024-02-19T21:54:21.426 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:02:18 ago on Mon 19 Feb 2024 09:52:03 PM UTC. 2024-02-19T21:54:22.375 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-02-19T21:54:22.375 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-02-19T21:54:22.376 INFO:teuthology.orchestra.run.smithi053.stdout: Package Architecture Version Repository Size 2024-02-19T21:54:22.376 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-02-19T21:54:22.376 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2024-02-19T21:54:22.376 INFO:teuthology.orchestra.run.smithi053.stdout: librados-devel x86_64 2:16.2.4-0.el8 ceph 131 k 2024-02-19T21:54:22.376 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-02-19T21:54:22.376 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2024-02-19T21:54:22.376 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-02-19T21:54:22.376 INFO:teuthology.orchestra.run.smithi053.stdout:Install 1 Package 2024-02-19T21:54:22.376 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-02-19T21:54:22.377 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 131 k 2024-02-19T21:54:22.377 INFO:teuthology.orchestra.run.smithi053.stdout:Installed size: 439 k 2024-02-19T21:54:22.377 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2024-02-19T21:54:22.876 INFO:teuthology.orchestra.run.smithi053.stdout:librados-devel-16.2.4-0.el8.x86_64.rpm 263 kB/s | 131 kB 00:00 2024-02-19T21:54:22.881 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2024-02-19T21:54:22.881 INFO:teuthology.orchestra.run.smithi053.stdout:Total 262 kB/s | 131 kB 00:00 2024-02-19T21:54:22.881 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2024-02-19T21:54:22.884 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2024-02-19T21:54:22.884 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2024-02-19T21:54:22.928 INFO:teuthology.orchestra.run.smithi086.stdout:Package librbd1-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-19T21:54:22.944 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2024-02-19T21:54:22.947 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2024-02-19T21:54:22.975 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-02-19T21:54:22.976 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2024-02-19T21:54:22.976 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-02-19T21:54:23.083 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install python3-rados 2024-02-19T21:54:23.147 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2024-02-19T21:54:23.359 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T21:54:23.653 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T21:54:23.838 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:02:20 ago on Mon 19 Feb 2024 09:52:03 PM UTC. 2024-02-19T21:54:24.071 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T21:54:24.071 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-02-19T21:54:24.071 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2024-02-19T21:54:24.071 INFO:teuthology.orchestra.run.smithi053.stdout: librados-devel-2:16.2.4-0.el8.x86_64 2024-02-19T21:54:24.072 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-02-19T21:54:24.072 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-02-19T21:54:24.316 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install libcephfs2 2024-02-19T21:54:25.108 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:02:21 ago on Mon 19 Feb 2024 09:52:04 PM UTC. 2024-02-19T21:54:25.350 INFO:teuthology.orchestra.run.smithi086.stdout:Package python3-rados-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-19T21:54:25.396 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-02-19T21:54:25.397 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2024-02-19T21:54:25.398 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-02-19T21:54:25.498 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install python3-rgw 2024-02-19T21:54:26.253 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:02:23 ago on Mon 19 Feb 2024 09:52:03 PM UTC. 2024-02-19T21:54:26.712 INFO:teuthology.orchestra.run.smithi053.stdout:Package libcephfs2-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-19T21:54:26.763 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-02-19T21:54:26.764 INFO:teuthology.orchestra.run.smithi053.stdout:Nothing to do. 2024-02-19T21:54:26.764 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-02-19T21:54:26.867 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install libcephfs-devel 2024-02-19T21:54:27.657 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:02:23 ago on Mon 19 Feb 2024 09:52:04 PM UTC. 2024-02-19T21:54:27.765 INFO:teuthology.orchestra.run.smithi086.stdout:Package python3-rgw-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-19T21:54:27.811 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-02-19T21:54:27.813 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2024-02-19T21:54:27.813 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-02-19T21:54:27.911 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install python3-cephfs 2024-02-19T21:54:28.658 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:02:25 ago on Mon 19 Feb 2024 09:52:03 PM UTC. 2024-02-19T21:54:29.340 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-02-19T21:54:29.341 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-02-19T21:54:29.341 INFO:teuthology.orchestra.run.smithi053.stdout: Package Architecture Version Repository Size 2024-02-19T21:54:29.341 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-02-19T21:54:29.341 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2024-02-19T21:54:29.341 INFO:teuthology.orchestra.run.smithi053.stdout: libcephfs-devel x86_64 2:16.2.4-0.el8 ceph 24 k 2024-02-19T21:54:29.341 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-02-19T21:54:29.341 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2024-02-19T21:54:29.341 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-02-19T21:54:29.341 INFO:teuthology.orchestra.run.smithi053.stdout:Install 1 Package 2024-02-19T21:54:29.341 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-02-19T21:54:29.342 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 24 k 2024-02-19T21:54:29.342 INFO:teuthology.orchestra.run.smithi053.stdout:Installed size: 94 k 2024-02-19T21:54:29.342 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2024-02-19T21:54:29.411 INFO:teuthology.orchestra.run.smithi053.stdout:libcephfs-devel-16.2.4-0.el8.x86_64.rpm 351 kB/s | 24 kB 00:00 2024-02-19T21:54:29.412 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2024-02-19T21:54:29.412 INFO:teuthology.orchestra.run.smithi053.stdout:Total 345 kB/s | 24 kB 00:00 2024-02-19T21:54:29.412 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2024-02-19T21:54:29.416 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2024-02-19T21:54:29.416 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2024-02-19T21:54:29.418 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2024-02-19T21:54:29.422 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2024-02-19T21:54:29.472 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2024-02-19T21:54:29.687 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T21:54:29.960 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T21:54:30.159 INFO:teuthology.orchestra.run.smithi086.stdout:Package python3-cephfs-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-19T21:54:30.206 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-02-19T21:54:30.207 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2024-02-19T21:54:30.207 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-02-19T21:54:30.309 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install python3-rbd 2024-02-19T21:54:30.333 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T21:54:30.334 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-02-19T21:54:30.334 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2024-02-19T21:54:30.334 INFO:teuthology.orchestra.run.smithi053.stdout: libcephfs-devel-2:16.2.4-0.el8.x86_64 2024-02-19T21:54:30.334 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-02-19T21:54:30.334 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-02-19T21:54:30.461 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install librados2 2024-02-19T21:54:31.063 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:02:28 ago on Mon 19 Feb 2024 09:52:03 PM UTC. 2024-02-19T21:54:31.249 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:02:27 ago on Mon 19 Feb 2024 09:52:04 PM UTC. 2024-02-19T21:54:32.570 INFO:teuthology.orchestra.run.smithi086.stdout:Package python3-rbd-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-19T21:54:32.616 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-02-19T21:54:32.617 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2024-02-19T21:54:32.618 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-02-19T21:54:32.736 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install rbd-fuse 2024-02-19T21:54:32.871 INFO:teuthology.orchestra.run.smithi053.stdout:Package librados2-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-19T21:54:32.921 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-02-19T21:54:32.922 INFO:teuthology.orchestra.run.smithi053.stdout:Nothing to do. 2024-02-19T21:54:32.922 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-02-19T21:54:33.029 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install librbd1 2024-02-19T21:54:33.488 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:02:30 ago on Mon 19 Feb 2024 09:52:03 PM UTC. 2024-02-19T21:54:33.821 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:02:29 ago on Mon 19 Feb 2024 09:52:04 PM UTC. 2024-02-19T21:54:35.060 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-02-19T21:54:35.061 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-02-19T21:54:35.061 INFO:teuthology.orchestra.run.smithi086.stdout: Package Architecture Version Repository Size 2024-02-19T21:54:35.061 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-02-19T21:54:35.061 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-02-19T21:54:35.061 INFO:teuthology.orchestra.run.smithi086.stdout: rbd-fuse x86_64 2:16.2.4-0.el8 ceph 89 k 2024-02-19T21:54:35.061 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T21:54:35.061 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-02-19T21:54:35.061 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-02-19T21:54:35.061 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2024-02-19T21:54:35.061 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T21:54:35.062 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 89 k 2024-02-19T21:54:35.062 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 298 k 2024-02-19T21:54:35.062 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-02-19T21:54:35.106 INFO:teuthology.orchestra.run.smithi086.stdout:rbd-fuse-16.2.4-0.el8.x86_64.rpm 2.0 MB/s | 89 kB 00:00 2024-02-19T21:54:35.106 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-02-19T21:54:35.107 INFO:teuthology.orchestra.run.smithi086.stdout:Total 1.9 MB/s | 89 kB 00:00 2024-02-19T21:54:35.107 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-02-19T21:54:35.117 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-02-19T21:54:35.117 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-02-19T21:54:35.166 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-02-19T21:54:35.169 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-02-19T21:54:35.272 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-02-19T21:54:35.431 INFO:teuthology.orchestra.run.smithi053.stdout:Package librbd1-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-19T21:54:35.480 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T21:54:35.481 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-02-19T21:54:35.482 INFO:teuthology.orchestra.run.smithi053.stdout:Nothing to do. 2024-02-19T21:54:35.482 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-02-19T21:54:35.583 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install python3-rados 2024-02-19T21:54:35.780 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T21:54:36.164 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T21:54:36.164 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T21:54:36.165 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-02-19T21:54:36.165 INFO:teuthology.orchestra.run.smithi086.stdout: rbd-fuse-2:16.2.4-0.el8.x86_64 2024-02-19T21:54:36.165 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T21:54:36.165 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-02-19T21:54:36.377 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:02:32 ago on Mon 19 Feb 2024 09:52:04 PM UTC. 2024-02-19T21:54:36.408 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install rbd-mirror 2024-02-19T21:54:37.170 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:02:34 ago on Mon 19 Feb 2024 09:52:03 PM UTC. 2024-02-19T21:54:37.989 INFO:teuthology.orchestra.run.smithi053.stdout:Package python3-rados-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-19T21:54:38.039 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-02-19T21:54:38.040 INFO:teuthology.orchestra.run.smithi053.stdout:Nothing to do. 2024-02-19T21:54:38.040 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-02-19T21:54:38.140 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install python3-rgw 2024-02-19T21:54:38.710 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-02-19T21:54:38.711 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-02-19T21:54:38.711 INFO:teuthology.orchestra.run.smithi086.stdout: Package Architecture Version Repository Size 2024-02-19T21:54:38.711 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-02-19T21:54:38.711 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-02-19T21:54:38.711 INFO:teuthology.orchestra.run.smithi086.stdout: rbd-mirror x86_64 2:16.2.4-0.el8 ceph 3.7 M 2024-02-19T21:54:38.711 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T21:54:38.711 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-02-19T21:54:38.712 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-02-19T21:54:38.712 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2024-02-19T21:54:38.712 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T21:54:38.712 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 3.7 M 2024-02-19T21:54:38.712 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 18 M 2024-02-19T21:54:38.712 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-02-19T21:54:38.793 INFO:teuthology.orchestra.run.smithi086.stdout:rbd-mirror-16.2.4-0.el8.x86_64.rpm 46 MB/s | 3.7 MB 00:00 2024-02-19T21:54:38.793 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-02-19T21:54:38.794 INFO:teuthology.orchestra.run.smithi086.stdout:Total 45 MB/s | 3.7 MB 00:00 2024-02-19T21:54:38.794 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-02-19T21:54:38.806 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-02-19T21:54:38.806 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-02-19T21:54:38.868 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-02-19T21:54:38.871 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-02-19T21:54:38.929 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:02:34 ago on Mon 19 Feb 2024 09:52:04 PM UTC. 2024-02-19T21:54:39.331 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-02-19T21:54:39.523 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T21:54:39.943 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T21:54:40.338 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T21:54:40.338 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T21:54:40.338 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-02-19T21:54:40.338 INFO:teuthology.orchestra.run.smithi086.stdout: rbd-mirror-2:16.2.4-0.el8.x86_64 2024-02-19T21:54:40.338 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T21:54:40.338 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-02-19T21:54:40.546 INFO:teuthology.orchestra.run.smithi053.stdout:Package python3-rgw-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-19T21:54:40.578 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install rbd-nbd 2024-02-19T21:54:40.597 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-02-19T21:54:40.598 INFO:teuthology.orchestra.run.smithi053.stdout:Nothing to do. 2024-02-19T21:54:40.598 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-02-19T21:54:40.704 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install python3-cephfs 2024-02-19T21:54:41.343 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:02:38 ago on Mon 19 Feb 2024 09:52:03 PM UTC. 2024-02-19T21:54:41.487 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:02:37 ago on Mon 19 Feb 2024 09:52:04 PM UTC. 2024-02-19T21:54:42.896 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-02-19T21:54:42.897 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-02-19T21:54:42.897 INFO:teuthology.orchestra.run.smithi086.stdout: Package Architecture Version Repository Size 2024-02-19T21:54:42.897 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-02-19T21:54:42.897 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-02-19T21:54:42.897 INFO:teuthology.orchestra.run.smithi086.stdout: rbd-nbd x86_64 2:16.2.4-0.el8 ceph 173 k 2024-02-19T21:54:42.897 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T21:54:42.897 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-02-19T21:54:42.898 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-02-19T21:54:42.898 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2024-02-19T21:54:42.898 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T21:54:42.898 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 173 k 2024-02-19T21:54:42.898 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 649 k 2024-02-19T21:54:42.898 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-02-19T21:54:42.946 INFO:teuthology.orchestra.run.smithi086.stdout:rbd-nbd-16.2.4-0.el8.x86_64.rpm 3.5 MB/s | 173 kB 00:00 2024-02-19T21:54:42.947 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-02-19T21:54:42.947 INFO:teuthology.orchestra.run.smithi086.stdout:Total 3.4 MB/s | 173 kB 00:00 2024-02-19T21:54:42.947 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-02-19T21:54:42.958 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-02-19T21:54:42.958 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-02-19T21:54:43.008 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-02-19T21:54:43.011 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-02-19T21:54:43.114 INFO:teuthology.orchestra.run.smithi053.stdout:Package python3-cephfs-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-19T21:54:43.122 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-02-19T21:54:43.165 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-02-19T21:54:43.167 INFO:teuthology.orchestra.run.smithi053.stdout:Nothing to do. 2024-02-19T21:54:43.167 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-02-19T21:54:43.265 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install python3-rbd 2024-02-19T21:54:43.355 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T21:54:43.645 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T21:54:44.037 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T21:54:44.038 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T21:54:44.038 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-02-19T21:54:44.038 INFO:teuthology.orchestra.run.smithi086.stdout: rbd-nbd-2:16.2.4-0.el8.x86_64 2024-02-19T21:54:44.038 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T21:54:44.038 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-02-19T21:54:44.056 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:02:40 ago on Mon 19 Feb 2024 09:52:04 PM UTC. 2024-02-19T21:54:44.271 DEBUG:teuthology.parallel:result is None 2024-02-19T21:54:45.676 INFO:teuthology.orchestra.run.smithi053.stdout:Package python3-rbd-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-19T21:54:45.727 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-02-19T21:54:45.728 INFO:teuthology.orchestra.run.smithi053.stdout:Nothing to do. 2024-02-19T21:54:45.728 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-02-19T21:54:45.835 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install rbd-fuse 2024-02-19T21:54:46.624 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:02:42 ago on Mon 19 Feb 2024 09:52:04 PM UTC. 2024-02-19T21:54:48.304 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-02-19T21:54:48.304 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-02-19T21:54:48.305 INFO:teuthology.orchestra.run.smithi053.stdout: Package Architecture Version Repository Size 2024-02-19T21:54:48.305 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-02-19T21:54:48.305 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2024-02-19T21:54:48.305 INFO:teuthology.orchestra.run.smithi053.stdout: rbd-fuse x86_64 2:16.2.4-0.el8 ceph 89 k 2024-02-19T21:54:48.305 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-02-19T21:54:48.306 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2024-02-19T21:54:48.306 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-02-19T21:54:48.306 INFO:teuthology.orchestra.run.smithi053.stdout:Install 1 Package 2024-02-19T21:54:48.306 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-02-19T21:54:48.306 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 89 k 2024-02-19T21:54:48.306 INFO:teuthology.orchestra.run.smithi053.stdout:Installed size: 298 k 2024-02-19T21:54:48.306 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2024-02-19T21:54:48.344 INFO:teuthology.orchestra.run.smithi053.stdout:rbd-fuse-16.2.4-0.el8.x86_64.rpm 2.1 MB/s | 89 kB 00:00 2024-02-19T21:54:48.345 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2024-02-19T21:54:48.345 INFO:teuthology.orchestra.run.smithi053.stdout:Total 2.0 MB/s | 89 kB 00:00 2024-02-19T21:54:48.346 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2024-02-19T21:54:48.356 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2024-02-19T21:54:48.356 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2024-02-19T21:54:48.409 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2024-02-19T21:54:48.412 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2024-02-19T21:54:48.516 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2024-02-19T21:54:48.705 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T21:54:48.995 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T21:54:49.394 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T21:54:49.394 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-02-19T21:54:49.394 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2024-02-19T21:54:49.394 INFO:teuthology.orchestra.run.smithi053.stdout: rbd-fuse-2:16.2.4-0.el8.x86_64 2024-02-19T21:54:49.394 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-02-19T21:54:49.394 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-02-19T21:54:49.610 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install rbd-mirror 2024-02-19T21:54:50.403 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:02:46 ago on Mon 19 Feb 2024 09:52:04 PM UTC. 2024-02-19T21:54:52.069 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-02-19T21:54:52.070 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-02-19T21:54:52.070 INFO:teuthology.orchestra.run.smithi053.stdout: Package Architecture Version Repository Size 2024-02-19T21:54:52.070 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-02-19T21:54:52.070 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2024-02-19T21:54:52.070 INFO:teuthology.orchestra.run.smithi053.stdout: rbd-mirror x86_64 2:16.2.4-0.el8 ceph 3.7 M 2024-02-19T21:54:52.070 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-02-19T21:54:52.070 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2024-02-19T21:54:52.071 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-02-19T21:54:52.071 INFO:teuthology.orchestra.run.smithi053.stdout:Install 1 Package 2024-02-19T21:54:52.071 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-02-19T21:54:52.071 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 3.7 M 2024-02-19T21:54:52.071 INFO:teuthology.orchestra.run.smithi053.stdout:Installed size: 18 M 2024-02-19T21:54:52.071 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2024-02-19T21:54:52.172 INFO:teuthology.orchestra.run.smithi053.stdout:rbd-mirror-16.2.4-0.el8.x86_64.rpm 37 MB/s | 3.7 MB 00:00 2024-02-19T21:54:52.173 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2024-02-19T21:54:52.174 INFO:teuthology.orchestra.run.smithi053.stdout:Total 36 MB/s | 3.7 MB 00:00 2024-02-19T21:54:52.174 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2024-02-19T21:54:52.186 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2024-02-19T21:54:52.187 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2024-02-19T21:54:52.255 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2024-02-19T21:54:52.258 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2024-02-19T21:54:52.749 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2024-02-19T21:54:52.939 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T21:54:53.355 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T21:54:53.753 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T21:54:53.754 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-02-19T21:54:53.754 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2024-02-19T21:54:53.754 INFO:teuthology.orchestra.run.smithi053.stdout: rbd-mirror-2:16.2.4-0.el8.x86_64 2024-02-19T21:54:53.754 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-02-19T21:54:53.754 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-02-19T21:54:53.987 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install rbd-nbd 2024-02-19T21:54:54.787 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:02:50 ago on Mon 19 Feb 2024 09:52:04 PM UTC. 2024-02-19T21:54:56.459 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-02-19T21:54:56.460 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-02-19T21:54:56.460 INFO:teuthology.orchestra.run.smithi053.stdout: Package Architecture Version Repository Size 2024-02-19T21:54:56.460 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-02-19T21:54:56.460 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2024-02-19T21:54:56.460 INFO:teuthology.orchestra.run.smithi053.stdout: rbd-nbd x86_64 2:16.2.4-0.el8 ceph 173 k 2024-02-19T21:54:56.461 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-02-19T21:54:56.461 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2024-02-19T21:54:56.461 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-02-19T21:54:56.461 INFO:teuthology.orchestra.run.smithi053.stdout:Install 1 Package 2024-02-19T21:54:56.461 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-02-19T21:54:56.461 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 173 k 2024-02-19T21:54:56.462 INFO:teuthology.orchestra.run.smithi053.stdout:Installed size: 649 k 2024-02-19T21:54:56.462 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2024-02-19T21:54:56.500 INFO:teuthology.orchestra.run.smithi053.stdout:rbd-nbd-16.2.4-0.el8.x86_64.rpm 4.4 MB/s | 173 kB 00:00 2024-02-19T21:54:56.501 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2024-02-19T21:54:56.501 INFO:teuthology.orchestra.run.smithi053.stdout:Total 4.3 MB/s | 173 kB 00:00 2024-02-19T21:54:56.501 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2024-02-19T21:54:56.512 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2024-02-19T21:54:56.512 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2024-02-19T21:54:56.566 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2024-02-19T21:54:56.568 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2024-02-19T21:54:56.700 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2024-02-19T21:54:56.884 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T21:54:57.169 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T21:54:57.582 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T21:54:57.582 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-02-19T21:54:57.582 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2024-02-19T21:54:57.582 INFO:teuthology.orchestra.run.smithi053.stdout: rbd-nbd-2:16.2.4-0.el8.x86_64 2024-02-19T21:54:57.582 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-02-19T21:54:57.582 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-02-19T21:54:57.790 DEBUG:teuthology.parallel:result is None 2024-02-19T21:54:57.791 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-02-19T21:54:57.791 INFO:teuthology.packaging:ref: None 2024-02-19T21:54:57.791 INFO:teuthology.packaging:tag: v16.2.4 2024-02-19T21:54:57.791 INFO:teuthology.packaging:branch: None 2024-02-19T21:54:57.791 INFO:teuthology.packaging:sha1: 81bd20d634209c7cb82c18be12b4b5a05643ebf1 2024-02-19T21:54:57.792 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2024-02-19T21:54:57.939 DEBUG:teuthology.orchestra.run.smithi053:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-02-19T21:54:57.962 INFO:teuthology.orchestra.run.smithi053.stdout:16.2.4-0.el8 2024-02-19T21:54:57.962 INFO:teuthology.packaging:The installed version of ceph is 16.2.4-0.el8 2024-02-19T21:54:57.963 INFO:teuthology.task.install:The correct ceph version 16.2.4-0 is installed. 2024-02-19T21:54:57.964 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-02-19T21:54:57.964 INFO:teuthology.packaging:ref: None 2024-02-19T21:54:57.965 INFO:teuthology.packaging:tag: v16.2.4 2024-02-19T21:54:57.965 INFO:teuthology.packaging:branch: None 2024-02-19T21:54:57.965 INFO:teuthology.packaging:sha1: 81bd20d634209c7cb82c18be12b4b5a05643ebf1 2024-02-19T21:54:57.965 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2024-02-19T21:54:58.189 DEBUG:teuthology.orchestra.run.smithi086:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-02-19T21:54:58.216 INFO:teuthology.orchestra.run.smithi086.stdout:16.2.4-0.el8 2024-02-19T21:54:58.216 INFO:teuthology.packaging:The installed version of ceph is 16.2.4-0.el8 2024-02-19T21:54:58.216 INFO:teuthology.task.install:The correct ceph version 16.2.4-0 is installed. 2024-02-19T21:54:58.218 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-02-19T21:54:58.218 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-02-19T21:54:58.218 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-02-19T21:54:58.246 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-02-19T21:54:58.246 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-02-19T21:54:58.288 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-02-19T21:54:58.289 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-02-19T21:54:58.289 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd of=/usr/bin/daemon-helper 2024-02-19T21:54:58.316 DEBUG:teuthology.orchestra.run.smithi053:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-02-19T21:54:58.383 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-02-19T21:54:58.383 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/usr/bin/daemon-helper 2024-02-19T21:54:58.414 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-02-19T21:54:58.484 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-02-19T21:54:58.484 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-02-19T21:54:58.484 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd of=/usr/bin/adjust-ulimits 2024-02-19T21:54:58.512 DEBUG:teuthology.orchestra.run.smithi053:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-02-19T21:54:58.579 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-02-19T21:54:58.579 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/usr/bin/adjust-ulimits 2024-02-19T21:54:58.608 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-02-19T21:54:58.677 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-02-19T21:54:58.677 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-02-19T21:54:58.677 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd of=/usr/bin/stdin-killer 2024-02-19T21:54:58.704 DEBUG:teuthology.orchestra.run.smithi053:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-02-19T21:54:58.772 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-02-19T21:54:58.772 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/usr/bin/stdin-killer 2024-02-19T21:54:58.800 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-02-19T21:54:58.868 INFO:teuthology.run_tasks:Running task print... 2024-02-19T21:54:58.878 INFO:teuthology.task.print:**** done install task... 2024-02-19T21:54:58.878 INFO:teuthology.run_tasks:Running task cephadm... 2024-02-19T21:54:58.951 INFO:tasks.cephadm:Config: {'avoid_pacific_features': True, 'cephadm_branch': 'v16.2.4', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'image': 'docker.io/ceph/ceph:v16.2.4', 'roleless': True, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'mon pg warn min per osd': 0}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(PG_DEGRADED\\)', 'Degraded data redundancy', '\\(PG_', 'MDS_INSUFFICIENT_STANDBY', 'deprecated feature inline_data', 'compat changed unexpectedly', 'overall HEALTH_', 'OSD_DOWN', 'OSD_', 'but it is still running', 'is not responding'], 'sha1': '81bd20d634209c7cb82c18be12b4b5a05643ebf1'} 2024-02-19T21:54:58.951 INFO:tasks.cephadm:Cluster image is docker.io/ceph/ceph:v16.2.4 2024-02-19T21:54:58.955 INFO:tasks.cephadm:Cluster fsid is 867b6b52-cf71-11ee-95bb-87774f69a715 2024-02-19T21:54:58.955 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-02-19T21:54:58.955 INFO:tasks.cephadm:No mon roles; fabricating mons 2024-02-19T21:54:58.955 INFO:tasks.cephadm:Monitor IPs: {'mon.smithi053': '172.21.15.53', 'mon.smithi086': '172.21.15.86'} 2024-02-19T21:54:58.956 INFO:tasks.cephadm:Normalizing hostnames... 2024-02-19T21:54:58.956 DEBUG:teuthology.orchestra.run.smithi053:> sudo hostname $(hostname -s) 2024-02-19T21:54:58.984 DEBUG:teuthology.orchestra.run.smithi086:> sudo hostname $(hostname -s) 2024-02-19T21:54:59.013 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref v16.2.4)... 2024-02-19T21:54:59.014 DEBUG:teuthology.orchestra.run.smithi053:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v16.2.4/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-02-19T21:54:59.161 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-rw-r--. 1 ubuntu ubuntu 292005 Feb 19 21:54 /home/ubuntu/cephtest/cephadm 2024-02-19T21:54:59.161 DEBUG:teuthology.orchestra.run.smithi086:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v16.2.4/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-02-19T21:54:59.305 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-rw-r--. 1 ubuntu ubuntu 292005 Feb 19 21:54 /home/ubuntu/cephtest/cephadm 2024-02-19T21:54:59.306 DEBUG:teuthology.orchestra.run.smithi053:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-02-19T21:54:59.326 DEBUG:teuthology.orchestra.run.smithi086:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-02-19T21:54:59.357 DEBUG:teuthology.orchestra.run.smithi053:> sudo mkdir -p /etc/ceph 2024-02-19T21:54:59.396 DEBUG:teuthology.orchestra.run.smithi086:> sudo mkdir -p /etc/ceph 2024-02-19T21:54:59.425 DEBUG:teuthology.orchestra.run.smithi053:> sudo chmod 777 /etc/ceph 2024-02-19T21:54:59.464 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod 777 /etc/ceph 2024-02-19T21:54:59.493 INFO:tasks.cephadm:Writing seed config... 2024-02-19T21:54:59.494 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-02-19T21:54:59.494 INFO:tasks.cephadm: override: [client] debug client = 20 2024-02-19T21:54:59.494 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-02-19T21:54:59.494 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-02-19T21:54:59.494 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-02-19T21:54:59.494 INFO:tasks.cephadm: override: [global] mon pg warn min per osd = 0 2024-02-19T21:54:59.494 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-02-19T21:54:59.494 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-02-19T21:54:59.494 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-02-19T21:54:59.494 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-02-19T21:54:59.494 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-02-19T21:54:59.495 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-02-19T21:54:59.495 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-02-19T21:54:59.495 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-02-19T21:54:59.495 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-02-19T21:54:59.495 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-02-19T21:54:59.495 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-02-19T21:54:59.495 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-02-19T21:54:59.495 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-02-19T21:54:59.495 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-02-19T21:54:59.495 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-02-19T21:54:59.495 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-02-19T21:54:59.495 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2024-02-19T21:54:59.495 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2024-02-19T21:54:59.495 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2024-02-19T21:54:59.495 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-02-19T21:54:59.496 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-02-19T21:54:59.496 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2024-02-19T21:54:59.496 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2024-02-19T21:54:59.496 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-02-19T21:54:59.496 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-02-19T21:54:59.496 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2024-02-19T21:54:59.496 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-02-19T21:54:59.496 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-02-19T21:54:59.496 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-02-19T21:54:59.496 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-02-19T21:54:59.496 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-02-19T21:54:59.496 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-02-19T21:54:59.497 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-02-19T21:54:59.497 DEBUG:teuthology.orchestra.run.smithi053:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-02-19T21:54:59.522 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 867b6b52-cf71-11ee-95bb-87774f69a715 mon pg warn min per osd = 0 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-02-19T21:54:59.522 DEBUG:teuthology.orchestra.run.smithi053:mon.smithi053> sudo journalctl -f -n 0 -u ceph-867b6b52-cf71-11ee-95bb-87774f69a715@mon.smithi053.service 2024-02-19T21:54:59.565 INFO:tasks.cephadm:Bootstrapping... 2024-02-19T21:54:59.565 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 -v bootstrap --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-ip 172.21.15.53 && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-02-19T21:54:59.593 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:-- Logs begin at Mon 2024-02-19 21:44:37 UTC. -- 2024-02-19T21:54:59.827 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2024-02-19T21:55:00.784 INFO:teuthology.orchestra.run.smithi053.stderr:/bin/podman: 4.9.3-dev 2024-02-19T21:55:00.794 INFO:teuthology.orchestra.run.smithi053.stderr:Verifying podman|docker is present... 2024-02-19T21:55:00.794 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2024-02-19T21:55:00.818 INFO:teuthology.orchestra.run.smithi053.stderr:/bin/podman: 4.9.3-dev 2024-02-19T21:55:00.830 INFO:teuthology.orchestra.run.smithi053.stderr:Verifying lvm2 is present... 2024-02-19T21:55:00.831 INFO:teuthology.orchestra.run.smithi053.stderr:Verifying time synchronization is in place... 2024-02-19T21:55:00.831 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: systemctl is-enabled chrony.service 2024-02-19T21:55:00.838 INFO:teuthology.orchestra.run.smithi053.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-02-19T21:55:00.839 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: systemctl is-active chrony.service 2024-02-19T21:55:00.845 INFO:teuthology.orchestra.run.smithi053.stderr:systemctl: inactive 2024-02-19T21:55:00.846 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: systemctl is-enabled chronyd.service 2024-02-19T21:55:00.852 INFO:teuthology.orchestra.run.smithi053.stderr:systemctl: enabled 2024-02-19T21:55:00.853 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: systemctl is-active chronyd.service 2024-02-19T21:55:00.858 INFO:teuthology.orchestra.run.smithi053.stderr:systemctl: active 2024-02-19T21:55:00.859 INFO:teuthology.orchestra.run.smithi053.stderr:Unit chronyd.service is enabled and running 2024-02-19T21:55:00.860 INFO:teuthology.orchestra.run.smithi053.stderr:Repeating the final host check... 2024-02-19T21:55:00.860 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2024-02-19T21:55:00.882 INFO:teuthology.orchestra.run.smithi053.stderr:/bin/podman: 4.9.3-dev 2024-02-19T21:55:00.891 INFO:teuthology.orchestra.run.smithi053.stderr:podman|docker (/bin/podman) is present 2024-02-19T21:55:00.891 INFO:teuthology.orchestra.run.smithi053.stderr:systemctl is present 2024-02-19T21:55:00.891 INFO:teuthology.orchestra.run.smithi053.stderr:lvcreate is present 2024-02-19T21:55:00.891 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: systemctl is-enabled chrony.service 2024-02-19T21:55:00.897 INFO:teuthology.orchestra.run.smithi053.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-02-19T21:55:00.897 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: systemctl is-active chrony.service 2024-02-19T21:55:00.903 INFO:teuthology.orchestra.run.smithi053.stderr:systemctl: inactive 2024-02-19T21:55:00.903 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: systemctl is-enabled chronyd.service 2024-02-19T21:55:00.909 INFO:teuthology.orchestra.run.smithi053.stderr:systemctl: enabled 2024-02-19T21:55:00.910 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: systemctl is-active chronyd.service 2024-02-19T21:55:00.915 INFO:teuthology.orchestra.run.smithi053.stderr:systemctl: active 2024-02-19T21:55:00.916 INFO:teuthology.orchestra.run.smithi053.stderr:Unit chronyd.service is enabled and running 2024-02-19T21:55:00.916 INFO:teuthology.orchestra.run.smithi053.stderr:Host looks OK 2024-02-19T21:55:00.916 INFO:teuthology.orchestra.run.smithi053.stderr:Cluster fsid: 867b6b52-cf71-11ee-95bb-87774f69a715 2024-02-19T21:55:00.916 INFO:teuthology.orchestra.run.smithi053.stderr:Acquiring lock 140182440573416 on /run/cephadm/867b6b52-cf71-11ee-95bb-87774f69a715.lock 2024-02-19T21:55:00.917 INFO:teuthology.orchestra.run.smithi053.stderr:Lock 140182440573416 acquired on /run/cephadm/867b6b52-cf71-11ee-95bb-87774f69a715.lock 2024-02-19T21:55:00.917 INFO:teuthology.orchestra.run.smithi053.stderr:Verifying IP 172.21.15.53 port 3300 ... 2024-02-19T21:55:00.917 INFO:teuthology.orchestra.run.smithi053.stderr:Verifying IP 172.21.15.53 port 6789 ... 2024-02-19T21:55:00.917 INFO:teuthology.orchestra.run.smithi053.stderr:Base mon IP is 172.21.15.53, final addrv is [v2:172.21.15.53:3300,v1:172.21.15.53:6789] 2024-02-19T21:55:00.918 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: /sbin/ip route ls 2024-02-19T21:55:00.920 INFO:teuthology.orchestra.run.smithi053.stderr:/sbin/ip: default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.53 metric 100 2024-02-19T21:55:00.920 INFO:teuthology.orchestra.run.smithi053.stderr:/sbin/ip: 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.53 metric 100 2024-02-19T21:55:00.921 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: /sbin/ip -6 route ls 2024-02-19T21:55:00.923 INFO:teuthology.orchestra.run.smithi053.stderr:/sbin/ip: ::1 dev lo proto kernel metric 256 pref medium 2024-02-19T21:55:00.923 INFO:teuthology.orchestra.run.smithi053.stderr:/sbin/ip: fe80::/64 dev ens1f0 proto kernel metric 256 pref medium 2024-02-19T21:55:00.923 INFO:teuthology.orchestra.run.smithi053.stderr:/sbin/ip: default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 1024 expires 1788sec hoplimit 64 pref medium 2024-02-19T21:55:00.924 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: /sbin/ip -6 addr ls 2024-02-19T21:55:00.926 INFO:teuthology.orchestra.run.smithi053.stderr:/sbin/ip: 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-02-19T21:55:00.926 INFO:teuthology.orchestra.run.smithi053.stderr:/sbin/ip: inet6 ::1/128 scope host 2024-02-19T21:55:00.926 INFO:teuthology.orchestra.run.smithi053.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-02-19T21:55:00.926 INFO:teuthology.orchestra.run.smithi053.stderr:/sbin/ip: 3: ens1f0: mtu 1500 state UP qlen 1000 2024-02-19T21:55:00.926 INFO:teuthology.orchestra.run.smithi053.stderr:/sbin/ip: inet6 fe80::ae1f:6bff:fef8:3718/64 scope link 2024-02-19T21:55:00.926 INFO:teuthology.orchestra.run.smithi053.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-02-19T21:55:00.927 INFO:teuthology.orchestra.run.smithi053.stderr:Mon IP 172.21.15.53 is in CIDR network 172.21.0.0/20 2024-02-19T21:55:00.927 INFO:teuthology.orchestra.run.smithi053.stderr:- internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-02-19T21:55:00.929 INFO:teuthology.orchestra.run.smithi053.stderr:Pulling container image docker.io/ceph/ceph:v16.2.4... 2024-02-19T21:55:00.929 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: /bin/podman pull docker.io/ceph/ceph:v16.2.4 2024-02-19T21:55:01.026 INFO:teuthology.orchestra.run.smithi053.stderr:/bin/podman: Trying to pull docker.io/ceph/ceph:v16.2.4... 2024-02-19T21:55:07.595 INFO:teuthology.orchestra.run.smithi053.stderr:/bin/podman: Getting image source signatures 2024-02-19T21:55:07.603 INFO:teuthology.orchestra.run.smithi053.stderr:/bin/podman: Copying blob sha256:551101690791055467b1840be133b2e6b029d263771bc6c46a96bb7acdacac39 2024-02-19T21:55:07.603 INFO:teuthology.orchestra.run.smithi053.stderr:/bin/podman: Copying blob sha256:7a0437f04f83f084b7ed68ad9c4a4947e12fc4e1b006b38129bac89114ec3621 2024-02-19T21:55:19.828 INFO:teuthology.orchestra.run.smithi053.stderr:/bin/podman: Copying config sha256:8d91d370c2b86c07de46146aba8d36718eaefa69b1880c77fa312fda6efd7d29 2024-02-19T21:55:19.961 INFO:teuthology.orchestra.run.smithi053.stderr:/bin/podman: Writing manifest to image destination 2024-02-19T21:55:20.266 INFO:teuthology.orchestra.run.smithi053.stderr:/bin/podman: 8d91d370c2b86c07de46146aba8d36718eaefa69b1880c77fa312fda6efd7d29 2024-02-19T21:55:20.277 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi053 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --version 2024-02-19T21:55:20.911 INFO:teuthology.orchestra.run.smithi053.stderr:ceph: ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable) 2024-02-19T21:55:21.454 INFO:teuthology.orchestra.run.smithi053.stderr:Ceph version: ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable) 2024-02-19T21:55:21.454 INFO:teuthology.orchestra.run.smithi053.stderr:Extracting ceph user uid/gid from container image... 2024-02-19T21:55:21.454 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi053 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 -c %u %g /var/lib/ceph 2024-02-19T21:55:22.032 INFO:teuthology.orchestra.run.smithi053.stderr:stat: 167 167 2024-02-19T21:55:22.546 INFO:teuthology.orchestra.run.smithi053.stderr:Creating initial keys... 2024-02-19T21:55:22.546 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi053 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2024-02-19T21:55:23.057 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-authtool: AQBKztNlYwIyOhAAy6cePVD++/s1I8qnt5EFVQ== 2024-02-19T21:55:23.630 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi053 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2024-02-19T21:55:24.210 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-authtool: AQBMztNlpsPkBhAAwijO7sTGo77Y9gxPeMUp/Q== 2024-02-19T21:55:24.732 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi053 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2024-02-19T21:55:25.324 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-authtool: AQBNztNlV/aqDRAAyLFrc0wZxUx15PtzXrUfQw== 2024-02-19T21:55:25.855 INFO:teuthology.orchestra.run.smithi053.stderr:Creating initial monmap... 2024-02-19T21:55:25.855 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/monmaptool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi053 -e CEPH_USE_RANDOM_NONCE=1 -v /tmp/ceph-tmp8zlozya1:/tmp/monmap:z docker.io/ceph/ceph:v16.2.4 --create --clobber --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 --addv smithi053 [v2:172.21.15.53:3300,v1:172.21.15.53:6789] /tmp/monmap 2024-02-19T21:55:26.415 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: monmap file /tmp/monmap 2024-02-19T21:55:26.416 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: set fsid to 867b6b52-cf71-11ee-95bb-87774f69a715 2024-02-19T21:55:26.416 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-02-19T21:55:27.266 INFO:teuthology.orchestra.run.smithi053.stderr:monmaptool for smithi053 [v2:172.21.15.53:3300,v1:172.21.15.53:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-02-19T21:55:27.266 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/monmaptool: set fsid to 867b6b52-cf71-11ee-95bb-87774f69a715 2024-02-19T21:55:27.266 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-02-19T21:55:27.266 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-02-19T21:55:27.267 INFO:teuthology.orchestra.run.smithi053.stderr:Creating mon... 2024-02-19T21:55:27.267 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-mon --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi053 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/867b6b52-cf71-11ee-95bb-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/867b6b52-cf71-11ee-95bb-87774f69a715/mon.smithi053:/var/lib/ceph/mon/ceph-smithi053:z -v /tmp/ceph-tmp2m42v88v:/tmp/keyring:z -v /tmp/ceph-tmp8zlozya1:/tmp/monmap:z docker.io/ceph/ceph:v16.2.4 --mkfs -i smithi053 --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -c /dev/null --monmap /tmp/monmap --keyring /tmp/keyring --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true 2024-02-19T21:55:27.794 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.712+0000 7fb08620c700 0 set uid:gid to 167:167 (ceph:ceph) 2024-02-19T21:55:27.794 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.712+0000 7fb08620c700 1 imported monmap: 2024-02-19T21:55:27.794 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: epoch 0 2024-02-19T21:55:27.794 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: fsid 867b6b52-cf71-11ee-95bb-87774f69a715 2024-02-19T21:55:27.794 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: last_changed 2024-02-19T21:55:26.357744+0000 2024-02-19T21:55:27.794 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: created 2024-02-19T21:55:26.357744+0000 2024-02-19T21:55:27.794 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: min_mon_release 0 (unknown) 2024-02-19T21:55:27.794 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: election_strategy: 1 2024-02-19T21:55:27.795 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: 0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.smithi053 2024-02-19T21:55:27.795 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: 2024-02-19T21:55:27.795 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.712+0000 7fb08620c700 0 /usr/bin/ceph-mon: set fsid to 867b6b52-cf71-11ee-95bb-87774f69a715 2024-02-19T21:55:27.795 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.713+0000 7fb08620c700 4 rocksdb: RocksDB version: 6.8.1 2024-02-19T21:55:27.795 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: 2024-02-19T21:55:27.795 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.713+0000 7fb08620c700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-02-19T21:55:27.795 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.713+0000 7fb08620c700 4 rocksdb: Compile date May 13 2021 2024-02-19T21:55:27.795 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.713+0000 7fb08620c700 4 rocksdb: DB SUMMARY 2024-02-19T21:55:27.795 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: 2024-02-19T21:55:27.795 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.713+0000 7fb08620c700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi053/store.db dir, Total Num: 0, files: 2024-02-19T21:55:27.795 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: 2024-02-19T21:55:27.795 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.713+0000 7fb08620c700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi053/store.db: 2024-02-19T21:55:27.795 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: 2024-02-19T21:55:27.795 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.713+0000 7fb08620c700 4 rocksdb: Options.error_if_exists: 0 2024-02-19T21:55:27.795 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.713+0000 7fb08620c700 4 rocksdb: Options.create_if_missing: 1 2024-02-19T21:55:27.796 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.713+0000 7fb08620c700 4 rocksdb: Options.paranoid_checks: 1 2024-02-19T21:55:27.796 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.713+0000 7fb08620c700 4 rocksdb: Options.env: 0x561d710c41c0 2024-02-19T21:55:27.796 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.713+0000 7fb08620c700 4 rocksdb: Options.fs: Posix File System 2024-02-19T21:55:27.796 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.713+0000 7fb08620c700 4 rocksdb: Options.info_log: 0x561d7318cec0 2024-02-19T21:55:27.796 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.713+0000 7fb08620c700 4 rocksdb: Options.max_file_opening_threads: 16 2024-02-19T21:55:27.796 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.713+0000 7fb08620c700 4 rocksdb: Options.statistics: (nil) 2024-02-19T21:55:27.796 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.713+0000 7fb08620c700 4 rocksdb: Options.use_fsync: 0 2024-02-19T21:55:27.796 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.713+0000 7fb08620c700 4 rocksdb: Options.max_log_file_size: 0 2024-02-19T21:55:27.796 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.713+0000 7fb08620c700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-19T21:55:27.796 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.713+0000 7fb08620c700 4 rocksdb: Options.log_file_time_to_roll: 0 2024-02-19T21:55:27.796 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.713+0000 7fb08620c700 4 rocksdb: Options.keep_log_file_num: 1000 2024-02-19T21:55:27.796 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.713+0000 7fb08620c700 4 rocksdb: Options.recycle_log_file_num: 0 2024-02-19T21:55:27.796 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.713+0000 7fb08620c700 4 rocksdb: Options.allow_fallocate: 1 2024-02-19T21:55:27.796 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.713+0000 7fb08620c700 4 rocksdb: Options.allow_mmap_reads: 0 2024-02-19T21:55:27.797 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.713+0000 7fb08620c700 4 rocksdb: Options.allow_mmap_writes: 0 2024-02-19T21:55:27.797 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.713+0000 7fb08620c700 4 rocksdb: Options.use_direct_reads: 0 2024-02-19T21:55:27.797 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.713+0000 7fb08620c700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-19T21:55:27.797 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.713+0000 7fb08620c700 4 rocksdb: Options.create_missing_column_families: 0 2024-02-19T21:55:27.797 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.713+0000 7fb08620c700 4 rocksdb: Options.db_log_dir: 2024-02-19T21:55:27.797 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.713+0000 7fb08620c700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi053/store.db 2024-02-19T21:55:27.797 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.713+0000 7fb08620c700 4 rocksdb: Options.table_cache_numshardbits: 6 2024-02-19T21:55:27.797 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.713+0000 7fb08620c700 4 rocksdb: Options.max_subcompactions: 1 2024-02-19T21:55:27.797 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.713+0000 7fb08620c700 4 rocksdb: Options.max_background_flushes: -1 2024-02-19T21:55:27.797 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.713+0000 7fb08620c700 4 rocksdb: Options.WAL_ttl_seconds: 0 2024-02-19T21:55:27.797 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.713+0000 7fb08620c700 4 rocksdb: Options.WAL_size_limit_MB: 0 2024-02-19T21:55:27.797 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.713+0000 7fb08620c700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-02-19T21:55:27.797 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.713+0000 7fb08620c700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-19T21:55:27.797 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.713+0000 7fb08620c700 4 rocksdb: Options.is_fd_close_on_exec: 1 2024-02-19T21:55:27.797 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.713+0000 7fb08620c700 4 rocksdb: Options.advise_random_on_open: 1 2024-02-19T21:55:27.798 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.713+0000 7fb08620c700 4 rocksdb: Options.db_write_buffer_size: 0 2024-02-19T21:55:27.798 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.713+0000 7fb08620c700 4 rocksdb: Options.write_buffer_manager: 0x561d73193530 2024-02-19T21:55:27.798 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.713+0000 7fb08620c700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-19T21:55:27.798 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.713+0000 7fb08620c700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-02-19T21:55:27.798 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.713+0000 7fb08620c700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-19T21:55:27.798 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.713+0000 7fb08620c700 4 rocksdb: Options.use_adaptive_mutex: 0 2024-02-19T21:55:27.798 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.713+0000 7fb08620c700 4 rocksdb: Options.rate_limiter: (nil) 2024-02-19T21:55:27.798 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.713+0000 7fb08620c700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-19T21:55:27.798 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.713+0000 7fb08620c700 4 rocksdb: Options.wal_recovery_mode: 2 2024-02-19T21:55:27.798 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.713+0000 7fb08620c700 4 rocksdb: Options.enable_thread_tracking: 0 2024-02-19T21:55:27.798 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.713+0000 7fb08620c700 4 rocksdb: Options.enable_pipelined_write: 0 2024-02-19T21:55:27.798 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.713+0000 7fb08620c700 4 rocksdb: Options.unordered_write: 0 2024-02-19T21:55:27.798 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.713+0000 7fb08620c700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-19T21:55:27.798 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.713+0000 7fb08620c700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-19T21:55:27.798 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.713+0000 7fb08620c700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-19T21:55:27.799 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.713+0000 7fb08620c700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-19T21:55:27.799 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.713+0000 7fb08620c700 4 rocksdb: Options.row_cache: None 2024-02-19T21:55:27.799 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.713+0000 7fb08620c700 4 rocksdb: Options.wal_filter: None 2024-02-19T21:55:27.799 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.713+0000 7fb08620c700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-19T21:55:27.799 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.713+0000 7fb08620c700 4 rocksdb: Options.allow_ingest_behind: 0 2024-02-19T21:55:27.799 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.713+0000 7fb08620c700 4 rocksdb: Options.preserve_deletes: 0 2024-02-19T21:55:27.799 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.713+0000 7fb08620c700 4 rocksdb: Options.two_write_queues: 0 2024-02-19T21:55:27.799 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.713+0000 7fb08620c700 4 rocksdb: Options.manual_wal_flush: 0 2024-02-19T21:55:27.799 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.713+0000 7fb08620c700 4 rocksdb: Options.atomic_flush: 0 2024-02-19T21:55:27.799 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.713+0000 7fb08620c700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-19T21:55:27.799 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.713+0000 7fb08620c700 4 rocksdb: Options.persist_stats_to_disk: 0 2024-02-19T21:55:27.799 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.713+0000 7fb08620c700 4 rocksdb: Options.write_dbid_to_manifest: 0 2024-02-19T21:55:27.799 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.713+0000 7fb08620c700 4 rocksdb: Options.log_readahead_size: 0 2024-02-19T21:55:27.799 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.713+0000 7fb08620c700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2024-02-19T21:55:27.800 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.713+0000 7fb08620c700 4 rocksdb: Options.max_background_jobs: 2 2024-02-19T21:55:27.800 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.713+0000 7fb08620c700 4 rocksdb: Options.max_background_compactions: -1 2024-02-19T21:55:27.800 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.713+0000 7fb08620c700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-19T21:55:27.800 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.713+0000 7fb08620c700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-19T21:55:27.800 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.713+0000 7fb08620c700 4 rocksdb: Options.delayed_write_rate : 16777216 2024-02-19T21:55:27.800 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.713+0000 7fb08620c700 4 rocksdb: Options.max_total_wal_size: 0 2024-02-19T21:55:27.800 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.713+0000 7fb08620c700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-19T21:55:27.800 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.713+0000 7fb08620c700 4 rocksdb: Options.stats_dump_period_sec: 600 2024-02-19T21:55:27.800 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.713+0000 7fb08620c700 4 rocksdb: Options.stats_persist_period_sec: 600 2024-02-19T21:55:27.800 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.713+0000 7fb08620c700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-19T21:55:27.800 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.713+0000 7fb08620c700 4 rocksdb: Options.max_open_files: -1 2024-02-19T21:55:27.800 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.713+0000 7fb08620c700 4 rocksdb: Options.bytes_per_sync: 0 2024-02-19T21:55:27.800 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.713+0000 7fb08620c700 4 rocksdb: Options.wal_bytes_per_sync: 0 2024-02-19T21:55:27.800 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.713+0000 7fb08620c700 4 rocksdb: Options.strict_bytes_per_sync: 0 2024-02-19T21:55:27.801 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.713+0000 7fb08620c700 4 rocksdb: Options.compaction_readahead_size: 0 2024-02-19T21:55:27.801 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.713+0000 7fb08620c700 4 rocksdb: Compression algorithms supported: 2024-02-19T21:55:27.801 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.713+0000 7fb08620c700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-19T21:55:27.801 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.713+0000 7fb08620c700 4 rocksdb: kZSTD supported: 0 2024-02-19T21:55:27.801 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.713+0000 7fb08620c700 4 rocksdb: kXpressCompression supported: 0 2024-02-19T21:55:27.801 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.713+0000 7fb08620c700 4 rocksdb: kLZ4HCCompression supported: 1 2024-02-19T21:55:27.801 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.713+0000 7fb08620c700 4 rocksdb: kLZ4Compression supported: 1 2024-02-19T21:55:27.801 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.713+0000 7fb08620c700 4 rocksdb: kBZip2Compression supported: 0 2024-02-19T21:55:27.801 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.713+0000 7fb08620c700 4 rocksdb: kZlibCompression supported: 1 2024-02-19T21:55:27.801 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.713+0000 7fb08620c700 4 rocksdb: kSnappyCompression supported: 1 2024-02-19T21:55:27.801 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.713+0000 7fb08620c700 4 rocksdb: Fast CRC32 supported: Supported on x86 2024-02-19T21:55:27.801 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.713+0000 7fb08620c700 4 rocksdb: [db_impl/db_impl_open.cc:273] Creating manifest 1 2024-02-19T21:55:27.801 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: 2024-02-19T21:55:27.801 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.714+0000 7fb08620c700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi053/store.db/MANIFEST-000001 2024-02-19T21:55:27.801 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: 2024-02-19T21:55:27.802 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.714+0000 7fb08620c700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2024-02-19T21:55:27.802 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: 2024-02-19T21:55:27.802 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.714+0000 7fb08620c700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-19T21:55:27.802 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.714+0000 7fb08620c700 4 rocksdb: Options.merge_operator: 2024-02-19T21:55:27.802 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.714+0000 7fb08620c700 4 rocksdb: Options.compaction_filter: None 2024-02-19T21:55:27.802 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.714+0000 7fb08620c700 4 rocksdb: Options.compaction_filter_factory: None 2024-02-19T21:55:27.802 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.714+0000 7fb08620c700 4 rocksdb: Options.memtable_factory: SkipListFactory 2024-02-19T21:55:27.802 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.714+0000 7fb08620c700 4 rocksdb: Options.table_factory: BlockBasedTable 2024-02-19T21:55:27.802 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.714+0000 7fb08620c700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x561d730a5cb8) 2024-02-19T21:55:27.802 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks: 1 2024-02-19T21:55:27.802 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks_with_high_priority: 0 2024-02-19T21:55:27.802 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-19T21:55:27.802 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: pin_top_level_index_and_filter: 1 2024-02-19T21:55:27.802 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: index_type: 0 2024-02-19T21:55:27.803 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: data_block_index_type: 0 2024-02-19T21:55:27.803 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: index_shortening: 1 2024-02-19T21:55:27.803 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: data_block_hash_table_util_ratio: 0.750000 2024-02-19T21:55:27.803 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: hash_index_allow_collision: 1 2024-02-19T21:55:27.803 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: checksum: 1 2024-02-19T21:55:27.803 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: no_block_cache: 0 2024-02-19T21:55:27.803 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: block_cache: 0x561d730dad10 2024-02-19T21:55:27.803 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: block_cache_name: BinnedLRUCache 2024-02-19T21:55:27.803 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: block_cache_options: 2024-02-19T21:55:27.803 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: capacity : 536870912 2024-02-19T21:55:27.803 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: num_shard_bits : 4 2024-02-19T21:55:27.803 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: strict_capacity_limit : 0 2024-02-19T21:55:27.803 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: high_pri_pool_ratio: 0.000 2024-02-19T21:55:27.803 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: block_cache_compressed: (nil) 2024-02-19T21:55:27.803 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: persistent_cache: (nil) 2024-02-19T21:55:27.803 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: block_size: 4096 2024-02-19T21:55:27.804 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: block_size_deviation: 10 2024-02-19T21:55:27.804 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: block_restart_interval: 16 2024-02-19T21:55:27.804 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: index_block_restart_interval: 1 2024-02-19T21:55:27.804 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: metadata_block_size: 4096 2024-02-19T21:55:27.804 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: partition_filters: 0 2024-02-19T21:55:27.804 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: use_delta_encoding: 1 2024-02-19T21:55:27.804 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: filter_policy: rocksdb.BuiltinBloomFilter 2024-02-19T21:55:27.804 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: whole_key_filtering: 1 2024-02-19T21:55:27.804 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: verify_compression: 0 2024-02-19T21:55:27.804 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: read_amp_bytes_per_bit: 0 2024-02-19T21:55:27.804 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: format_version: 2 2024-02-19T21:55:27.804 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: enable_index_compression: 1 2024-02-19T21:55:27.804 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: block_align: 0 2024-02-19T21:55:27.804 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: 2024-02-19T21:55:27.805 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.714+0000 7fb08620c700 4 rocksdb: Options.write_buffer_size: 33554432 2024-02-19T21:55:27.805 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.714+0000 7fb08620c700 4 rocksdb: Options.max_write_buffer_number: 2 2024-02-19T21:55:27.805 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.714+0000 7fb08620c700 4 rocksdb: Options.compression: NoCompression 2024-02-19T21:55:27.805 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.714+0000 7fb08620c700 4 rocksdb: Options.bottommost_compression: Disabled 2024-02-19T21:55:27.805 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.714+0000 7fb08620c700 4 rocksdb: Options.prefix_extractor: nullptr 2024-02-19T21:55:27.805 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.714+0000 7fb08620c700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-19T21:55:27.805 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.714+0000 7fb08620c700 4 rocksdb: Options.num_levels: 7 2024-02-19T21:55:27.805 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.714+0000 7fb08620c700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-19T21:55:27.805 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.714+0000 7fb08620c700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-19T21:55:27.805 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.714+0000 7fb08620c700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-02-19T21:55:27.805 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.714+0000 7fb08620c700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-19T21:55:27.805 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.714+0000 7fb08620c700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-19T21:55:27.805 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.714+0000 7fb08620c700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-19T21:55:27.805 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.714+0000 7fb08620c700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-19T21:55:27.806 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.714+0000 7fb08620c700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-19T21:55:27.806 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.714+0000 7fb08620c700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-19T21:55:27.806 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.714+0000 7fb08620c700 4 rocksdb: Options.compression_opts.window_bits: -14 2024-02-19T21:55:27.806 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.714+0000 7fb08620c700 4 rocksdb: Options.compression_opts.level: 32767 2024-02-19T21:55:27.806 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.714+0000 7fb08620c700 4 rocksdb: Options.compression_opts.strategy: 0 2024-02-19T21:55:27.806 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.714+0000 7fb08620c700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-19T21:55:27.806 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.714+0000 7fb08620c700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-19T21:55:27.806 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.714+0000 7fb08620c700 4 rocksdb: Options.compression_opts.enabled: false 2024-02-19T21:55:27.806 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.714+0000 7fb08620c700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-19T21:55:27.806 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.714+0000 7fb08620c700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-19T21:55:27.806 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.714+0000 7fb08620c700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-19T21:55:27.806 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.714+0000 7fb08620c700 4 rocksdb: Options.target_file_size_base: 67108864 2024-02-19T21:55:27.806 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.714+0000 7fb08620c700 4 rocksdb: Options.target_file_size_multiplier: 1 2024-02-19T21:55:27.806 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.714+0000 7fb08620c700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-19T21:55:27.806 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.714+0000 7fb08620c700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-19T21:55:27.807 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.714+0000 7fb08620c700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-19T21:55:27.807 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.714+0000 7fb08620c700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-19T21:55:27.807 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.714+0000 7fb08620c700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-19T21:55:27.807 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.714+0000 7fb08620c700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-19T21:55:27.807 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.714+0000 7fb08620c700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-19T21:55:27.807 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.714+0000 7fb08620c700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-19T21:55:27.807 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.714+0000 7fb08620c700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-19T21:55:27.807 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.714+0000 7fb08620c700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-19T21:55:27.807 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.714+0000 7fb08620c700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-19T21:55:27.807 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.714+0000 7fb08620c700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-19T21:55:27.807 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.714+0000 7fb08620c700 4 rocksdb: Options.arena_block_size: 4194304 2024-02-19T21:55:27.807 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.714+0000 7fb08620c700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-19T21:55:27.807 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.714+0000 7fb08620c700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-19T21:55:27.807 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.714+0000 7fb08620c700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-02-19T21:55:27.808 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.714+0000 7fb08620c700 4 rocksdb: Options.disable_auto_compactions: 0 2024-02-19T21:55:27.808 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.714+0000 7fb08620c700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-19T21:55:27.808 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.714+0000 7fb08620c700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-19T21:55:27.808 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.714+0000 7fb08620c700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-19T21:55:27.808 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.714+0000 7fb08620c700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-19T21:55:27.808 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.714+0000 7fb08620c700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-19T21:55:27.808 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.714+0000 7fb08620c700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-19T21:55:27.808 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.714+0000 7fb08620c700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-19T21:55:27.808 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.714+0000 7fb08620c700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-19T21:55:27.808 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.714+0000 7fb08620c700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-19T21:55:27.808 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.714+0000 7fb08620c700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-19T21:55:27.808 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.714+0000 7fb08620c700 4 rocksdb: Options.table_properties_collectors: 2024-02-19T21:55:27.808 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.714+0000 7fb08620c700 4 rocksdb: Options.inplace_update_support: 0 2024-02-19T21:55:27.808 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.714+0000 7fb08620c700 4 rocksdb: Options.inplace_update_num_locks: 10000 2024-02-19T21:55:27.809 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.714+0000 7fb08620c700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-19T21:55:27.809 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.714+0000 7fb08620c700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-19T21:55:27.809 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.714+0000 7fb08620c700 4 rocksdb: Options.memtable_huge_page_size: 0 2024-02-19T21:55:27.809 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.714+0000 7fb08620c700 4 rocksdb: Options.bloom_locality: 0 2024-02-19T21:55:27.809 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.714+0000 7fb08620c700 4 rocksdb: Options.max_successive_merges: 0 2024-02-19T21:55:27.809 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.714+0000 7fb08620c700 4 rocksdb: Options.optimize_filters_for_hits: 0 2024-02-19T21:55:27.809 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.714+0000 7fb08620c700 4 rocksdb: Options.paranoid_file_checks: 0 2024-02-19T21:55:27.809 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.714+0000 7fb08620c700 4 rocksdb: Options.force_consistency_checks: 0 2024-02-19T21:55:27.809 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.714+0000 7fb08620c700 4 rocksdb: Options.report_bg_io_stats: 0 2024-02-19T21:55:27.809 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.714+0000 7fb08620c700 4 rocksdb: Options.ttl: 2592000 2024-02-19T21:55:27.809 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.714+0000 7fb08620c700 4 rocksdb: Options.periodic_compaction_seconds: 0 2024-02-19T21:55:27.809 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.714+0000 7fb08620c700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi053/store.db/MANIFEST-000001 succeeded,manifest_file_number is 1, next_file_number is 3, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-02-19T21:55:27.809 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: 2024-02-19T21:55:27.809 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.714+0000 7fb08620c700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 0 2024-02-19T21:55:27.810 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: 2024-02-19T21:55:27.810 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.715+0000 7fb08620c700 4 rocksdb: DB pointer 0x561d7319f800 2024-02-19T21:55:27.810 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.715+0000 7fb06f0a5700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2024-02-19T21:55:27.810 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.715+0000 7fb06f0a5700 4 rocksdb: [db_impl/db_impl.cc:851] 2024-02-19T21:55:27.810 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: ** DB Stats ** 2024-02-19T21:55:27.810 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2024-02-19T21:55:27.810 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-19T21:55:27.810 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-19T21:55:27.810 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-19T21:55:27.810 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-19T21:55:27.810 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-02-19T21:55:27.810 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-19T21:55:27.810 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: 2024-02-19T21:55:27.810 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-02-19T21:55:27.811 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-19T21:55:27.811 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-19T21:55:27.811 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-19T21:55:27.811 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-19T21:55:27.811 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: 2024-02-19T21:55:27.811 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-02-19T21:55:27.811 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-19T21:55:27.811 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-19T21:55:27.811 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2024-02-19T21:55:27.811 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2024-02-19T21:55:27.811 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-19T21:55:27.811 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2024-02-19T21:55:27.811 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2024-02-19T21:55:27.811 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2024-02-19T21:55:27.811 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-19T21:55:27.812 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-19T21:55:27.812 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-19T21:55:27.812 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: 2024-02-19T21:55:27.812 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2024-02-19T21:55:27.812 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: 2024-02-19T21:55:27.812 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-02-19T21:55:27.812 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-19T21:55:27.812 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-19T21:55:27.812 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-19T21:55:27.812 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-19T21:55:27.812 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: 2024-02-19T21:55:27.812 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-02-19T21:55:27.812 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-19T21:55:27.812 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-19T21:55:27.813 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2024-02-19T21:55:27.813 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2024-02-19T21:55:27.813 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-19T21:55:27.813 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2024-02-19T21:55:27.813 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2024-02-19T21:55:27.813 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2024-02-19T21:55:27.813 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-19T21:55:27.813 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-19T21:55:27.813 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-19T21:55:27.813 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: 2024-02-19T21:55:27.813 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2024-02-19T21:55:27.813 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: 2024-02-19T21:55:27.813 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.717+0000 7fb08620c700 4 rocksdb: [db_impl/db_impl.cc:397] Shutdown: canceling all background work 2024-02-19T21:55:27.813 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.717+0000 7fb08620c700 4 rocksdb: [db_impl/db_impl.cc:573] Shutdown complete 2024-02-19T21:55:27.813 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-mon: debug 2024-02-19T21:55:27.717+0000 7fb08620c700 0 /usr/bin/ceph-mon: created monfs at /var/lib/ceph/mon/ceph-smithi053 for mon.smithi053 2024-02-19T21:55:28.384 INFO:teuthology.orchestra.run.smithi053.stderr:create mon.smithi053 on 2024-02-19T21:55:28.384 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: install -d -m0770 -o 167 -g 167 /var/run/ceph/867b6b52-cf71-11ee-95bb-87774f69a715 2024-02-19T21:55:28.391 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: systemctl disable ceph.target 2024-02-19T21:55:28.400 INFO:teuthology.orchestra.run.smithi053.stderr:systemctl: Removed /etc/systemd/system/multi-user.target.wants/ceph.target. 2024-02-19T21:55:28.494 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: systemctl enable ceph.target 2024-02-19T21:55:28.502 INFO:teuthology.orchestra.run.smithi053.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-02-19T21:55:28.589 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: systemctl start ceph.target 2024-02-19T21:55:28.596 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: systemctl enable ceph-867b6b52-cf71-11ee-95bb-87774f69a715.target 2024-02-19T21:55:28.602 INFO:teuthology.orchestra.run.smithi053.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-867b6b52-cf71-11ee-95bb-87774f69a715.target → /etc/systemd/system/ceph-867b6b52-cf71-11ee-95bb-87774f69a715.target. 2024-02-19T21:55:28.602 INFO:teuthology.orchestra.run.smithi053.stderr:systemctl: Created symlink /etc/systemd/system/ceph.target.wants/ceph-867b6b52-cf71-11ee-95bb-87774f69a715.target → /etc/systemd/system/ceph-867b6b52-cf71-11ee-95bb-87774f69a715.target. 2024-02-19T21:55:28.686 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: systemctl start ceph-867b6b52-cf71-11ee-95bb-87774f69a715.target 2024-02-19T21:55:28.693 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: systemctl daemon-reload 2024-02-19T21:55:28.782 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: systemctl stop ceph-867b6b52-cf71-11ee-95bb-87774f69a715@mon.smithi053 2024-02-19T21:55:28.809 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: systemctl reset-failed ceph-867b6b52-cf71-11ee-95bb-87774f69a715@mon.smithi053 2024-02-19T21:55:28.814 INFO:teuthology.orchestra.run.smithi053.stderr:systemctl: Failed to reset failed state of unit ceph-867b6b52-cf71-11ee-95bb-87774f69a715@mon.smithi053.service: Unit ceph-867b6b52-cf71-11ee-95bb-87774f69a715@mon.smithi053.service not loaded. 2024-02-19T21:55:28.815 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: systemctl enable ceph-867b6b52-cf71-11ee-95bb-87774f69a715@mon.smithi053 2024-02-19T21:55:28.821 INFO:teuthology.orchestra.run.smithi053.stderr:systemctl: Created symlink /etc/systemd/system/ceph-867b6b52-cf71-11ee-95bb-87774f69a715.target.wants/ceph-867b6b52-cf71-11ee-95bb-87774f69a715@mon.smithi053.service → /etc/systemd/system/ceph-867b6b52-cf71-11ee-95bb-87774f69a715@.service. 2024-02-19T21:55:28.909 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: systemctl start ceph-867b6b52-cf71-11ee-95bb-87774f69a715@mon.smithi053 2024-02-19T21:55:28.952 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:28 smithi053 systemd[1]: Starting Ceph mon.smithi053 for 867b6b52-cf71-11ee-95bb-87774f69a715... 2024-02-19T21:55:29.413 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: systemctl is-enabled firewalld.service 2024-02-19T21:55:29.421 INFO:teuthology.orchestra.run.smithi053.stderr:systemctl: disabled 2024-02-19T21:55:29.422 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: systemctl is-active firewalld.service 2024-02-19T21:55:29.429 INFO:teuthology.orchestra.run.smithi053.stderr:systemctl: inactive 2024-02-19T21:55:29.430 INFO:teuthology.orchestra.run.smithi053.stderr:firewalld.service is not enabled 2024-02-19T21:55:29.431 INFO:teuthology.orchestra.run.smithi053.stderr:Not possible to enable service . firewalld.service is not available 2024-02-19T21:55:29.431 INFO:teuthology.orchestra.run.smithi053.stderr:Not possible to open ports <[]>. firewalld.service is not available 2024-02-19T21:55:29.431 INFO:teuthology.orchestra.run.smithi053.stderr:Waiting for mon to start... 2024-02-19T21:55:29.431 INFO:teuthology.orchestra.run.smithi053.stderr:Waiting for mon... 2024-02-19T21:55:29.431 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi053 -e CEPH_USE_RANDOM_NONCE=1 -v /var/lib/ceph/867b6b52-cf71-11ee-95bb-87774f69a715/mon.smithi053:/var/lib/ceph/mon/ceph-smithi053:z -v /tmp/ceph-tmpoyd_8g9j:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp0zy4ixsn:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status 2024-02-19T21:55:29.674 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:29 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[37738]: cluster 2024-02-19T21:55:29.401698+0000 mon.smithi053 (mon.0) 0 : [INF] mkfs 867b6b52-cf71-11ee-95bb-87774f69a715 2024-02-19T21:55:29.674 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:29 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[37738]: cluster 2024-02-19T21:55:29.398704+0000 mon.smithi053 (mon.0) 1 : cluster [INF] mon.smithi053 is new leader, mons smithi053 in quorum (ranks 0) 2024-02-19T21:55:29.675 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:29 smithi053 bash[37723]: 20b21fe0215a09ba48aa48509d6e48459ef133c315ef592b7ba39a2740d52af9 2024-02-19T21:55:29.675 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:29 smithi053 systemd[1]: Started Ceph mon.smithi053 for 867b6b52-cf71-11ee-95bb-87774f69a715. 2024-02-19T21:55:29.920 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:29.920+0000 7f8b36228700 1 Processor -- start 2024-02-19T21:55:29.923 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:29.920+0000 7f8b36228700 1 -- start start 2024-02-19T21:55:29.923 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:29.920+0000 7f8b36228700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8b300703b0 0x7f8b30100d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T21:55:29.923 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:29.920+0000 7f8b36228700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b300fc700 con 0x7f8b300703b0 2024-02-19T21:55:29.923 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:29.921+0000 7f8b35226700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8b300703b0 0x7f8b30100d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:55:29.923 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:29.921+0000 7f8b35226700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8b300703b0 0x7f8b30100d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:48838/0 (socket says 172.21.15.53:48838) 2024-02-19T21:55:29.923 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:29.921+0000 7f8b35226700 1 -- 172.21.15.53:0/4158814880 learned_addr learned my addr 172.21.15.53:0/4158814880 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-02-19T21:55:29.923 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:29.921+0000 7f8b35226700 1 -- 172.21.15.53:0/4158814880 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b30101200 con 0x7f8b300703b0 2024-02-19T21:55:29.923 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:29.921+0000 7f8b35226700 1 --2- 172.21.15.53:0/4158814880 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8b300703b0 0x7f8b30100d10 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f8b2c00b5f0 tx=0x7f8b2c006ee0).ready entity=mon.0 client_cookie=4cf2367bafe647b9 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:55:29.923 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:29.921+0000 7f8b1ffff700 1 -- 172.21.15.53:0/4158814880 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8b2c00d600 con 0x7f8b300703b0 2024-02-19T21:55:29.923 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:29.921+0000 7f8b1ffff700 1 -- 172.21.15.53:0/4158814880 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b2c00d760 con 0x7f8b300703b0 2024-02-19T21:55:29.924 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:29.921+0000 7f8b1ffff700 1 -- 172.21.15.53:0/4158814880 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8b2c00da30 con 0x7f8b300703b0 2024-02-19T21:55:29.924 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:29.922+0000 7f8b36228700 1 -- 172.21.15.53:0/4158814880 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8b300703b0 msgr2=0x7f8b30100d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:55:29.924 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:29.922+0000 7f8b36228700 1 --2- 172.21.15.53:0/4158814880 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8b300703b0 0x7f8b30100d10 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f8b2c00b5f0 tx=0x7f8b2c006ee0).stop 2024-02-19T21:55:29.924 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:29.922+0000 7f8b36228700 1 -- 172.21.15.53:0/4158814880 shutdown_connections 2024-02-19T21:55:29.924 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:29.922+0000 7f8b36228700 1 --2- 172.21.15.53:0/4158814880 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8b300703b0 0x7f8b30100d10 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:55:29.924 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:29.922+0000 7f8b36228700 1 -- 172.21.15.53:0/4158814880 >> 172.21.15.53:0/4158814880 conn(0x7f8b300f6390 msgr2=0x7f8b300f87b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:55:29.924 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:29.922+0000 7f8b36228700 1 -- 172.21.15.53:0/4158814880 shutdown_connections 2024-02-19T21:55:29.924 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:29.922+0000 7f8b36228700 1 -- 172.21.15.53:0/4158814880 wait complete. 2024-02-19T21:55:29.924 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:29.922+0000 7f8b36228700 1 Processor -- start 2024-02-19T21:55:29.924 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:29.922+0000 7f8b36228700 1 -- start start 2024-02-19T21:55:29.924 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:29.923+0000 7f8b36228700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8b300703b0 0x7f8b3010c380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:55:29.924 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:29.923+0000 7f8b36228700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b3010c870 con 0x7f8b300703b0 2024-02-19T21:55:29.924 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:29.923+0000 7f8b35226700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8b300703b0 0x7f8b3010c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:55:29.924 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:29.923+0000 7f8b35226700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8b300703b0 0x7f8b3010c380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:48850/0 (socket says 172.21.15.53:48850) 2024-02-19T21:55:29.925 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:29.923+0000 7f8b35226700 1 -- 172.21.15.53:0/134563886 learned_addr learned my addr 172.21.15.53:0/134563886 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-02-19T21:55:29.926 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:29.923+0000 7f8b35226700 1 -- 172.21.15.53:0/134563886 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b2c00b200 con 0x7f8b300703b0 2024-02-19T21:55:29.926 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:29.923+0000 7f8b35226700 1 --2- 172.21.15.53:0/134563886 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8b300703b0 0x7f8b3010c380 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f8b2c006c70 tx=0x7f8b2c016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:55:29.926 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:29.923+0000 7f8b1dffb700 1 -- 172.21.15.53:0/134563886 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8b2c011b40 con 0x7f8b300703b0 2024-02-19T21:55:29.926 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:29.923+0000 7f8b1dffb700 1 -- 172.21.15.53:0/134563886 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b2c011ca0 con 0x7f8b300703b0 2024-02-19T21:55:29.926 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:29.923+0000 7f8b36228700 1 -- 172.21.15.53:0/134563886 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b3010ca70 con 0x7f8b300703b0 2024-02-19T21:55:29.926 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:29.923+0000 7f8b1dffb700 1 -- 172.21.15.53:0/134563886 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8b2c01d430 con 0x7f8b300703b0 2024-02-19T21:55:29.926 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:29.924+0000 7f8b36228700 1 -- 172.21.15.53:0/134563886 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b3010cf00 con 0x7f8b300703b0 2024-02-19T21:55:29.926 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:29.924+0000 7f8b1dffb700 1 -- 172.21.15.53:0/134563886 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f8b2c00f070 con 0x7f8b300703b0 2024-02-19T21:55:29.926 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:29.924+0000 7f8b1dffb700 1 -- 172.21.15.53:0/134563886 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f8b2c027460 con 0x7f8b300703b0 2024-02-19T21:55:29.926 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:29.925+0000 7f8b36228700 1 -- 172.21.15.53:0/134563886 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b30045ad0 con 0x7f8b300703b0 2024-02-19T21:55:29.927 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:29.927+0000 7f8b1dffb700 1 -- 172.21.15.53:0/134563886 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f8b2c016070 con 0x7f8b300703b0 2024-02-19T21:55:29.999 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:29.997+0000 7f8b36228700 1 -- 172.21.15.53:0/134563886 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7f8b30105fd0 con 0x7f8b300703b0 2024-02-19T21:55:30.000 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:29.997+0000 7f8b1dffb700 1 -- 172.21.15.53:0/134563886 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+325 (secure 0 0 0) 0x7f8b2c036070 con 0x7f8b300703b0 2024-02-19T21:55:30.000 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: cluster: 2024-02-19T21:55:30.000 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: id: 867b6b52-cf71-11ee-95bb-87774f69a715 2024-02-19T21:55:30.000 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: health: HEALTH_OK 2024-02-19T21:55:30.000 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:30.000 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: services: 2024-02-19T21:55:30.000 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: mon: 1 daemons, quorum smithi053 (age 0.597345s) 2024-02-19T21:55:30.000 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: mgr: no daemons active 2024-02-19T21:55:30.000 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: osd: 0 osds: 0 up, 0 in 2024-02-19T21:55:30.000 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:30.000 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: data: 2024-02-19T21:55:30.000 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: pools: 0 pools, 0 pgs 2024-02-19T21:55:30.001 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: objects: 0 objects, 0 B 2024-02-19T21:55:30.001 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: usage: 0 B used, 0 B / 0 B avail 2024-02-19T21:55:30.001 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: pgs: 2024-02-19T21:55:30.001 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:30.001 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:29.998+0000 7f8b36228700 1 -- 172.21.15.53:0/134563886 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8b300703b0 msgr2=0x7f8b3010c380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:55:30.001 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:29.998+0000 7f8b36228700 1 --2- 172.21.15.53:0/134563886 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8b300703b0 0x7f8b3010c380 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f8b2c006c70 tx=0x7f8b2c016000).stop 2024-02-19T21:55:30.001 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:29.998+0000 7f8b36228700 1 -- 172.21.15.53:0/134563886 shutdown_connections 2024-02-19T21:55:30.001 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:29.998+0000 7f8b36228700 1 --2- 172.21.15.53:0/134563886 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8b300703b0 0x7f8b3010c380 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:55:30.001 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:29.998+0000 7f8b36228700 1 -- 172.21.15.53:0/134563886 >> 172.21.15.53:0/134563886 conn(0x7f8b300f6390 msgr2=0x7f8b300f8080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:55:30.001 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:29.998+0000 7f8b36228700 1 -- 172.21.15.53:0/134563886 shutdown_connections 2024-02-19T21:55:30.001 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:29.998+0000 7f8b36228700 1 -- 172.21.15.53:0/134563886 wait complete. 2024-02-19T21:55:30.610 INFO:teuthology.orchestra.run.smithi053.stderr:mon is available 2024-02-19T21:55:30.610 INFO:teuthology.orchestra.run.smithi053.stderr:Assimilating anything we can from ceph.conf... 2024-02-19T21:55:30.610 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi053 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/867b6b52-cf71-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpoyd_8g9j:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp0zy4ixsn:/etc/ceph/ceph.conf:z -v /var/lib/ceph/867b6b52-cf71-11ee-95bb-87774f69a715/mon.smithi053:/var/lib/ceph/mon/ceph-smithi053:z docker.io/ceph/ceph:v16.2.4 config assimilate-conf -i /var/lib/ceph/mon/ceph-smithi053/config 2024-02-19T21:55:30.613 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:30 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[37738]: cluster 2024-02-19T21:55:29.401274+0000 mon.smithi053 (mon.0) 2 : cluster [INF] mon.smithi053 is new leader, mons smithi053 in quorum (ranks 0) 2024-02-19T21:55:30.613 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:30 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[37738]: cluster 2024-02-19T21:55:29.401387+0000 mon.smithi053 (mon.0) 3 : cluster [DBG] monmap e1: 1 mons at {smithi053=[v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0]} 2024-02-19T21:55:30.613 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:30 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[37738]: cluster 2024-02-19T21:55:29.402739+0000 mon.smithi053 (mon.0) 4 : cluster [DBG] fsmap 2024-02-19T21:55:30.613 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:30 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[37738]: cluster 2024-02-19T21:55:29.404309+0000 mon.smithi053 (mon.0) 5 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2024-02-19T21:55:30.613 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:30 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[37738]: cluster 2024-02-19T21:55:29.404722+0000 mon.smithi053 (mon.0) 6 : cluster [DBG] mgrmap e1: no daemons active 2024-02-19T21:55:30.613 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:30 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[37738]: audit 2024-02-19T21:55:29.998583+0000 mon.smithi053 (mon.0) 7 : audit [DBG] from='client.? 172.21.15.53:0/134563886' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-02-19T21:55:31.134 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:31.132+0000 7fb78d36d700 1 Processor -- start 2024-02-19T21:55:31.134 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:31.132+0000 7fb78d36d700 1 -- start start 2024-02-19T21:55:31.134 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:31.132+0000 7fb78d36d700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb7880fbea0 0x7fb7880fc2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T21:55:31.134 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:31.132+0000 7fb78d36d700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7880fc7b0 con 0x7fb7880fbea0 2024-02-19T21:55:31.134 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:31.132+0000 7fb786ffd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb7880fbea0 0x7fb7880fc2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:55:31.134 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:31.132+0000 7fb786ffd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb7880fbea0 0x7fb7880fc2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:48858/0 (socket says 172.21.15.53:48858) 2024-02-19T21:55:31.134 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:31.132+0000 7fb786ffd700 1 -- 172.21.15.53:0/2142289559 learned_addr learned my addr 172.21.15.53:0/2142289559 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-02-19T21:55:31.134 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:31.132+0000 7fb786ffd700 1 -- 172.21.15.53:0/2142289559 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7880fc8f0 con 0x7fb7880fbea0 2024-02-19T21:55:31.134 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:31.133+0000 7fb786ffd700 1 --2- 172.21.15.53:0/2142289559 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb7880fbea0 0x7fb7880fc2c0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7fb77800b5f0 tx=0x7fb778006ee0).ready entity=mon.0 client_cookie=fd58abee1ba09473 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:55:31.134 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:31.133+0000 7fb785ffb700 1 -- 172.21.15.53:0/2142289559 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb77800d600 con 0x7fb7880fbea0 2024-02-19T21:55:31.135 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:31.133+0000 7fb785ffb700 1 -- 172.21.15.53:0/2142289559 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb77800d760 con 0x7fb7880fbea0 2024-02-19T21:55:31.135 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:31.133+0000 7fb785ffb700 1 -- 172.21.15.53:0/2142289559 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb77800da30 con 0x7fb7880fbea0 2024-02-19T21:55:31.135 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:31.133+0000 7fb78d36d700 1 -- 172.21.15.53:0/2142289559 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb7880fbea0 msgr2=0x7fb7880fc2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:55:31.135 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:31.133+0000 7fb78d36d700 1 --2- 172.21.15.53:0/2142289559 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb7880fbea0 0x7fb7880fc2c0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7fb77800b5f0 tx=0x7fb778006ee0).stop 2024-02-19T21:55:31.135 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:31.133+0000 7fb78d36d700 1 -- 172.21.15.53:0/2142289559 shutdown_connections 2024-02-19T21:55:31.135 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:31.133+0000 7fb78d36d700 1 --2- 172.21.15.53:0/2142289559 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb7880fbea0 0x7fb7880fc2c0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:55:31.135 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:31.133+0000 7fb78d36d700 1 -- 172.21.15.53:0/2142289559 >> 172.21.15.53:0/2142289559 conn(0x7fb7880f76a0 msgr2=0x7fb7880f9ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:55:31.135 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:31.134+0000 7fb78d36d700 1 -- 172.21.15.53:0/2142289559 shutdown_connections 2024-02-19T21:55:31.135 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:31.134+0000 7fb78d36d700 1 -- 172.21.15.53:0/2142289559 wait complete. 2024-02-19T21:55:31.136 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:31.134+0000 7fb78d36d700 1 Processor -- start 2024-02-19T21:55:31.136 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:31.134+0000 7fb78d36d700 1 -- start start 2024-02-19T21:55:31.136 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:31.135+0000 7fb78d36d700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb7880fbea0 0x7fb78810a320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:55:31.136 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:31.135+0000 7fb78d36d700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb78810a810 con 0x7fb7880fbea0 2024-02-19T21:55:31.137 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:31.135+0000 7fb786ffd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb7880fbea0 0x7fb78810a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:55:31.137 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:31.135+0000 7fb786ffd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb7880fbea0 0x7fb78810a320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:48872/0 (socket says 172.21.15.53:48872) 2024-02-19T21:55:31.137 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:31.135+0000 7fb786ffd700 1 -- 172.21.15.53:0/4095675288 learned_addr learned my addr 172.21.15.53:0/4095675288 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-02-19T21:55:31.137 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:31.135+0000 7fb786ffd700 1 -- 172.21.15.53:0/4095675288 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb77800b200 con 0x7fb7880fbea0 2024-02-19T21:55:31.137 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:31.135+0000 7fb786ffd700 1 --2- 172.21.15.53:0/4095675288 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb7880fbea0 0x7fb78810a320 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7fb778006c70 tx=0x7fb778016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:55:31.137 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:31.135+0000 7fb76f7fe700 1 -- 172.21.15.53:0/4095675288 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb778011b40 con 0x7fb7880fbea0 2024-02-19T21:55:31.137 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:31.136+0000 7fb76f7fe700 1 -- 172.21.15.53:0/4095675288 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb778011ca0 con 0x7fb7880fbea0 2024-02-19T21:55:31.137 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:31.136+0000 7fb76f7fe700 1 -- 172.21.15.53:0/4095675288 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb77801d430 con 0x7fb7880fbea0 2024-02-19T21:55:31.137 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:31.136+0000 7fb78d36d700 1 -- 172.21.15.53:0/4095675288 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb78810aa10 con 0x7fb7880fbea0 2024-02-19T21:55:31.137 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:31.136+0000 7fb78d36d700 1 -- 172.21.15.53:0/4095675288 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb78810aea0 con 0x7fb7880fbea0 2024-02-19T21:55:31.137 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:31.136+0000 7fb76f7fe700 1 -- 172.21.15.53:0/4095675288 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7fb77800f070 con 0x7fb7880fbea0 2024-02-19T21:55:31.137 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:31.136+0000 7fb76f7fe700 1 -- 172.21.15.53:0/4095675288 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fb778027460 con 0x7fb7880fbea0 2024-02-19T21:55:31.138 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:31.137+0000 7fb78d36d700 1 -- 172.21.15.53:0/4095675288 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb788045ad0 con 0x7fb7880fbea0 2024-02-19T21:55:31.139 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:31.139+0000 7fb76f7fe700 1 -- 172.21.15.53:0/4095675288 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7fb778016070 con 0x7fb7880fbea0 2024-02-19T21:55:31.205 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:31.205+0000 7fb78d36d700 1 -- 172.21.15.53:0/4095675288 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7fb7880fd1e0 con 0x7fb7880fbea0 2024-02-19T21:55:31.211 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:31.208+0000 7fb76f7fe700 1 -- 172.21.15.53:0/4095675288 <== mon.0 v2:172.21.15.53:3300/0 7 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fb77801d620 con 0x7fb7880fbea0 2024-02-19T21:55:31.211 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:31.208+0000 7fb76f7fe700 1 -- 172.21.15.53:0/4095675288 <== mon.0 v2:172.21.15.53:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+489 (secure 0 0 0) 0x7fb7780309d0 con 0x7fb7880fbea0 2024-02-19T21:55:31.211 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:31.211 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: [global] 2024-02-19T21:55:31.211 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: fsid = 867b6b52-cf71-11ee-95bb-87774f69a715 2024-02-19T21:55:31.211 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: log_to_journald = false 2024-02-19T21:55:31.211 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: mon_host = [v2:172.21.15.53:3300,v1:172.21.15.53:6789] 2024-02-19T21:55:31.211 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-02-19T21:55:31.211 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-02-19T21:55:31.211 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-02-19T21:55:31.212 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-02-19T21:55:31.212 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:31.212 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: [mgr] 2024-02-19T21:55:31.212 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-02-19T21:55:31.212 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:31.212 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: [osd] 2024-02-19T21:55:31.212 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: mon_osd_backfillfull_ratio = 0.85 2024-02-19T21:55:31.212 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: mon_osd_full_ratio = 0.9 2024-02-19T21:55:31.212 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: mon_osd_nearfull_ratio = 0.8 2024-02-19T21:55:31.212 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-02-19T21:55:31.212 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: osd_memory_target_autotune = true 2024-02-19T21:55:31.212 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-02-19T21:55:31.212 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:31.209+0000 7fb78d36d700 1 -- 172.21.15.53:0/4095675288 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb7880fbea0 msgr2=0x7fb78810a320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:55:31.212 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:31.209+0000 7fb78d36d700 1 --2- 172.21.15.53:0/4095675288 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb7880fbea0 0x7fb78810a320 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7fb778006c70 tx=0x7fb778016000).stop 2024-02-19T21:55:31.213 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:31.209+0000 7fb78d36d700 1 -- 172.21.15.53:0/4095675288 shutdown_connections 2024-02-19T21:55:31.213 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:31.209+0000 7fb78d36d700 1 --2- 172.21.15.53:0/4095675288 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb7880fbea0 0x7fb78810a320 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:55:31.213 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:31.209+0000 7fb78d36d700 1 -- 172.21.15.53:0/4095675288 >> 172.21.15.53:0/4095675288 conn(0x7fb7880f76a0 msgr2=0x7fb7880fec20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:55:31.213 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:31.209+0000 7fb78d36d700 1 -- 172.21.15.53:0/4095675288 shutdown_connections 2024-02-19T21:55:31.213 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:31.209+0000 7fb78d36d700 1 -- 172.21.15.53:0/4095675288 wait complete. 2024-02-19T21:55:31.543 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:31 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[37738]: audit 2024-02-19T21:55:31.206818+0000 mon.smithi053 (mon.0) 8 : audit [INF] from='client.? 172.21.15.53:0/4095675288' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-02-19T21:55:31.543 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:31 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[37738]: audit 2024-02-19T21:55:31.209883+0000 mon.smithi053 (mon.0) 9 : audit [INF] from='client.? 172.21.15.53:0/4095675288' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-02-19T21:55:31.817 INFO:teuthology.orchestra.run.smithi053.stderr:Generating new minimal ceph.conf... 2024-02-19T21:55:31.818 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi053 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/867b6b52-cf71-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpoyd_8g9j:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp0zy4ixsn:/etc/ceph/ceph.conf:z -v /var/lib/ceph/867b6b52-cf71-11ee-95bb-87774f69a715/mon.smithi053:/var/lib/ceph/mon/ceph-smithi053:z docker.io/ceph/ceph:v16.2.4 config generate-minimal-conf -o /var/lib/ceph/mon/ceph-smithi053/config 2024-02-19T21:55:32.346 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:32.339+0000 7f1773a75700 1 Processor -- start 2024-02-19T21:55:32.346 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:32.339+0000 7f1773a75700 1 -- start start 2024-02-19T21:55:32.346 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:32.339+0000 7f1773a75700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f176c0703b0 0x7f176c100ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T21:55:32.346 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:32.339+0000 7f1773a75700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f176c0fc8c0 con 0x7f176c0703b0 2024-02-19T21:55:32.346 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:32.340+0000 7f1771811700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f176c0703b0 0x7f176c100ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:55:32.346 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:32.340+0000 7f1771811700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f176c0703b0 0x7f176c100ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:48882/0 (socket says 172.21.15.53:48882) 2024-02-19T21:55:32.346 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:32.340+0000 7f1771811700 1 -- 172.21.15.53:0/1955188922 learned_addr learned my addr 172.21.15.53:0/1955188922 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-02-19T21:55:32.346 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:32.340+0000 7f1771811700 1 -- 172.21.15.53:0/1955188922 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f176c1013c0 con 0x7f176c0703b0 2024-02-19T21:55:32.347 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:32.340+0000 7f1771811700 1 --2- 172.21.15.53:0/1955188922 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f176c0703b0 0x7f176c100ed0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f176800b5f0 tx=0x7f1768006ee0).ready entity=mon.0 client_cookie=1d17d64d1b8dfa66 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:55:32.347 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:32.340+0000 7f177080f700 1 -- 172.21.15.53:0/1955188922 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f176800d600 con 0x7f176c0703b0 2024-02-19T21:55:32.347 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:32.340+0000 7f177080f700 1 -- 172.21.15.53:0/1955188922 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f176800dbf0 con 0x7f176c0703b0 2024-02-19T21:55:32.347 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:32.340+0000 7f177080f700 1 -- 172.21.15.53:0/1955188922 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1768011cb0 con 0x7f176c0703b0 2024-02-19T21:55:32.347 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:32.341+0000 7f1773a75700 1 -- 172.21.15.53:0/1955188922 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f176c0703b0 msgr2=0x7f176c100ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:55:32.347 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:32.341+0000 7f1773a75700 1 --2- 172.21.15.53:0/1955188922 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f176c0703b0 0x7f176c100ed0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f176800b5f0 tx=0x7f1768006ee0).stop 2024-02-19T21:55:32.347 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:32.341+0000 7f1773a75700 1 -- 172.21.15.53:0/1955188922 shutdown_connections 2024-02-19T21:55:32.347 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:32.341+0000 7f1773a75700 1 --2- 172.21.15.53:0/1955188922 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f176c0703b0 0x7f176c100ed0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:55:32.347 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:32.341+0000 7f1773a75700 1 -- 172.21.15.53:0/1955188922 >> 172.21.15.53:0/1955188922 conn(0x7f176c0f6530 msgr2=0x7f176c0f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:55:32.347 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:32.341+0000 7f1773a75700 1 -- 172.21.15.53:0/1955188922 shutdown_connections 2024-02-19T21:55:32.348 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:32.341+0000 7f1773a75700 1 -- 172.21.15.53:0/1955188922 wait complete. 2024-02-19T21:55:32.348 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:32.342+0000 7f1773a75700 1 Processor -- start 2024-02-19T21:55:32.348 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:32.342+0000 7f1773a75700 1 -- start start 2024-02-19T21:55:32.348 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:32.342+0000 7f1773a75700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f176c0703b0 0x7f176c10cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:55:32.348 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:32.342+0000 7f1773a75700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f176c10d270 con 0x7f176c0703b0 2024-02-19T21:55:32.348 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:32.342+0000 7f1771811700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f176c0703b0 0x7f176c10cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:55:32.348 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:32.342+0000 7f1771811700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f176c0703b0 0x7f176c10cd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:48894/0 (socket says 172.21.15.53:48894) 2024-02-19T21:55:32.348 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:32.342+0000 7f1771811700 1 -- 172.21.15.53:0/55598057 learned_addr learned my addr 172.21.15.53:0/55598057 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-02-19T21:55:32.349 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:32.342+0000 7f1771811700 1 -- 172.21.15.53:0/55598057 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f176800b200 con 0x7f176c0703b0 2024-02-19T21:55:32.349 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:32.342+0000 7f1771811700 1 --2- 172.21.15.53:0/55598057 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f176c0703b0 0x7f176c10cd80 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f1768006800 tx=0x7f1768006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:55:32.349 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:32.343+0000 7f175e7fc700 1 -- 172.21.15.53:0/55598057 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f176801b570 con 0x7f176c0703b0 2024-02-19T21:55:32.349 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:32.343+0000 7f1773a75700 1 -- 172.21.15.53:0/55598057 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f176c10d470 con 0x7f176c0703b0 2024-02-19T21:55:32.349 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:32.343+0000 7f175e7fc700 1 -- 172.21.15.53:0/55598057 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f176801bb60 con 0x7f176c0703b0 2024-02-19T21:55:32.349 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:32.343+0000 7f175e7fc700 1 -- 172.21.15.53:0/55598057 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f176801a9f0 con 0x7f176c0703b0 2024-02-19T21:55:32.349 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:32.343+0000 7f1773a75700 1 -- 172.21.15.53:0/55598057 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f176c10e080 con 0x7f176c0703b0 2024-02-19T21:55:32.349 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:32.343+0000 7f175e7fc700 1 -- 172.21.15.53:0/55598057 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f176800f070 con 0x7f176c0703b0 2024-02-19T21:55:32.349 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:32.343+0000 7f175e7fc700 1 -- 172.21.15.53:0/55598057 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f1768025d80 con 0x7f176c0703b0 2024-02-19T21:55:32.349 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:32.344+0000 7f1773a75700 1 -- 172.21.15.53:0/55598057 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f176c059100 con 0x7f176c0703b0 2024-02-19T21:55:32.350 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:32.347+0000 7f175e7fc700 1 -- 172.21.15.53:0/55598057 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f1768016070 con 0x7f176c0703b0 2024-02-19T21:55:32.414 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:32.414+0000 7f1773a75700 1 -- 172.21.15.53:0/55598057 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f176c102040 con 0x7f176c0703b0 2024-02-19T21:55:32.414 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:32.414+0000 7f175e7fc700 1 -- 172.21.15.53:0/55598057 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+175 (secure 0 0 0) 0x7f1768025460 con 0x7f176c0703b0 2024-02-19T21:55:32.415 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:32.415+0000 7f1773a75700 1 -- 172.21.15.53:0/55598057 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f176c0703b0 msgr2=0x7f176c10cd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:55:32.415 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:32.415+0000 7f1773a75700 1 --2- 172.21.15.53:0/55598057 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f176c0703b0 0x7f176c10cd80 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f1768006800 tx=0x7f1768006ee0).stop 2024-02-19T21:55:32.416 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:32.415+0000 7f1773a75700 1 -- 172.21.15.53:0/55598057 shutdown_connections 2024-02-19T21:55:32.416 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:32.415+0000 7f1773a75700 1 --2- 172.21.15.53:0/55598057 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f176c0703b0 0x7f176c10cd80 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:55:32.416 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:32.415+0000 7f1773a75700 1 -- 172.21.15.53:0/55598057 >> 172.21.15.53:0/55598057 conn(0x7f176c0f6530 msgr2=0x7f176c0f7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:55:32.416 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:32.415+0000 7f1773a75700 1 -- 172.21.15.53:0/55598057 shutdown_connections 2024-02-19T21:55:32.416 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:32.415+0000 7f1773a75700 1 -- 172.21.15.53:0/55598057 wait complete. 2024-02-19T21:55:32.746 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:32 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[37738]: audit 2024-02-19T21:55:32.415536+0000 mon.smithi053 (mon.0) 10 : audit [DBG] from='client.? 172.21.15.53:0/55598057' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T21:55:32.992 INFO:teuthology.orchestra.run.smithi053.stderr:Restarting the monitor... 2024-02-19T21:55:32.992 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: systemctl restart ceph-867b6b52-cf71-11ee-95bb-87774f69a715@mon.smithi053 2024-02-19T21:55:33.293 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:33 smithi053 systemd[1]: Stopping Ceph mon.smithi053 for 867b6b52-cf71-11ee-95bb-87774f69a715... 2024-02-19T21:55:33.294 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:33 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[37738]: debug 2024-02-19T21:55:33.124+0000 7f083256e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi053 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-02-19T21:55:33.294 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:33 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[37738]: debug 2024-02-19T21:55:33.124+0000 7f083256e700 -1 mon.smithi053@0(leader) e1 *** Got Signal Terminated *** 2024-02-19T21:55:33.969 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:33 smithi053 podman[38061]: ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053 2024-02-19T21:55:34.289 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:33 smithi053 systemd[1]: ceph-867b6b52-cf71-11ee-95bb-87774f69a715@mon.smithi053.service: Succeeded. 2024-02-19T21:55:34.289 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:33 smithi053 systemd[1]: Stopped Ceph mon.smithi053 for 867b6b52-cf71-11ee-95bb-87774f69a715. 2024-02-19T21:55:34.289 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:33 smithi053 systemd[1]: Starting Ceph mon.smithi053 for 867b6b52-cf71-11ee-95bb-87774f69a715... 2024-02-19T21:55:34.466 INFO:teuthology.orchestra.run.smithi053.stderr:Setting mon public_network to 172.21.0.0/20 2024-02-19T21:55:34.466 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi053 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/867b6b52-cf71-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpoyd_8g9j:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp0zy4ixsn:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mon public_network 172.21.0.0/20 2024-02-19T21:55:34.700 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.406+0000 7efebd698700 0 set uid:gid to 167:167 (ceph:ceph) 2024-02-19T21:55:34.700 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.406+0000 7efebd698700 0 ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable), process ceph-mon, pid 7 2024-02-19T21:55:34.700 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.406+0000 7efebd698700 0 pidfile_write: ignore empty --pid-file 2024-02-19T21:55:34.701 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.411+0000 7efebd698700 0 load: jerasure load: lrc load: isa 2024-02-19T21:55:34.701 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: RocksDB version: 6.8.1 2024-02-19T21:55:34.701 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: 2024-02-19T21:55:34.701 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-02-19T21:55:34.701 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Compile date May 13 2021 2024-02-19T21:55:34.701 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: DB SUMMARY 2024-02-19T21:55:34.701 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: 2024-02-19T21:55:34.701 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: CURRENT file: CURRENT 2024-02-19T21:55:34.701 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: 2024-02-19T21:55:34.701 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: IDENTITY file: IDENTITY 2024-02-19T21:55:34.701 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: 2024-02-19T21:55:34.701 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: MANIFEST file: MANIFEST-000005 size: 131 Bytes 2024-02-19T21:55:34.701 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: 2024-02-19T21:55:34.701 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi053/store.db dir, Total Num: 1, files: 000004.sst 2024-02-19T21:55:34.701 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: 2024-02-19T21:55:34.702 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi053/store.db: 000006.log size: 98698 ; 2024-02-19T21:55:34.702 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: 2024-02-19T21:55:34.702 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.error_if_exists: 0 2024-02-19T21:55:34.702 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.create_if_missing: 0 2024-02-19T21:55:34.702 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.paranoid_checks: 1 2024-02-19T21:55:34.702 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.env: 0x55afb21e41c0 2024-02-19T21:55:34.702 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.fs: Posix File System 2024-02-19T21:55:34.702 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.info_log: 0x55afb312b960 2024-02-19T21:55:34.702 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.max_file_opening_threads: 16 2024-02-19T21:55:34.702 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.statistics: (nil) 2024-02-19T21:55:34.702 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.use_fsync: 0 2024-02-19T21:55:34.702 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.max_log_file_size: 0 2024-02-19T21:55:34.702 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-19T21:55:34.702 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.log_file_time_to_roll: 0 2024-02-19T21:55:34.703 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.keep_log_file_num: 1000 2024-02-19T21:55:34.703 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.recycle_log_file_num: 0 2024-02-19T21:55:34.703 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.allow_fallocate: 1 2024-02-19T21:55:34.703 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.allow_mmap_reads: 0 2024-02-19T21:55:34.703 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.allow_mmap_writes: 0 2024-02-19T21:55:34.703 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.use_direct_reads: 0 2024-02-19T21:55:34.703 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-19T21:55:34.703 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.create_missing_column_families: 0 2024-02-19T21:55:34.703 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.db_log_dir: 2024-02-19T21:55:34.703 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi053/store.db 2024-02-19T21:55:34.703 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.table_cache_numshardbits: 6 2024-02-19T21:55:34.703 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.max_subcompactions: 1 2024-02-19T21:55:34.703 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.max_background_flushes: -1 2024-02-19T21:55:34.703 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.WAL_ttl_seconds: 0 2024-02-19T21:55:34.703 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.WAL_size_limit_MB: 0 2024-02-19T21:55:34.704 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-02-19T21:55:34.704 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-19T21:55:34.704 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.is_fd_close_on_exec: 1 2024-02-19T21:55:34.704 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.advise_random_on_open: 1 2024-02-19T21:55:34.704 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.db_write_buffer_size: 0 2024-02-19T21:55:34.704 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.write_buffer_manager: 0x55afb3da0030 2024-02-19T21:55:34.704 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-19T21:55:34.704 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-02-19T21:55:34.704 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-19T21:55:34.704 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.use_adaptive_mutex: 0 2024-02-19T21:55:34.704 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.rate_limiter: (nil) 2024-02-19T21:55:34.704 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-19T21:55:34.704 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.wal_recovery_mode: 2 2024-02-19T21:55:34.704 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.enable_thread_tracking: 0 2024-02-19T21:55:34.705 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.enable_pipelined_write: 0 2024-02-19T21:55:34.705 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.unordered_write: 0 2024-02-19T21:55:34.705 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-19T21:55:34.705 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-19T21:55:34.705 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-19T21:55:34.705 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-19T21:55:34.705 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.row_cache: None 2024-02-19T21:55:34.705 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.wal_filter: None 2024-02-19T21:55:34.705 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-19T21:55:34.705 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.allow_ingest_behind: 0 2024-02-19T21:55:34.705 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.preserve_deletes: 0 2024-02-19T21:55:34.705 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.two_write_queues: 0 2024-02-19T21:55:34.705 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.manual_wal_flush: 0 2024-02-19T21:55:34.705 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.atomic_flush: 0 2024-02-19T21:55:34.705 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-19T21:55:34.706 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.persist_stats_to_disk: 0 2024-02-19T21:55:34.706 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.write_dbid_to_manifest: 0 2024-02-19T21:55:34.706 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.log_readahead_size: 0 2024-02-19T21:55:34.706 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2024-02-19T21:55:34.706 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.max_background_jobs: 2 2024-02-19T21:55:34.706 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.max_background_compactions: -1 2024-02-19T21:55:34.706 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-19T21:55:34.706 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-19T21:55:34.706 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.delayed_write_rate : 16777216 2024-02-19T21:55:34.706 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.max_total_wal_size: 0 2024-02-19T21:55:34.706 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-19T21:55:34.706 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.stats_dump_period_sec: 600 2024-02-19T21:55:34.706 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.stats_persist_period_sec: 600 2024-02-19T21:55:34.706 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-19T21:55:34.706 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.max_open_files: -1 2024-02-19T21:55:34.707 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.bytes_per_sync: 0 2024-02-19T21:55:34.707 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.wal_bytes_per_sync: 0 2024-02-19T21:55:34.707 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.strict_bytes_per_sync: 0 2024-02-19T21:55:34.707 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.compaction_readahead_size: 0 2024-02-19T21:55:34.707 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Compression algorithms supported: 2024-02-19T21:55:34.707 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-19T21:55:34.707 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: kZSTD supported: 0 2024-02-19T21:55:34.707 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: kXpressCompression supported: 0 2024-02-19T21:55:34.707 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: kLZ4HCCompression supported: 1 2024-02-19T21:55:34.707 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: kLZ4Compression supported: 1 2024-02-19T21:55:34.707 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: kBZip2Compression supported: 0 2024-02-19T21:55:34.707 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: kZlibCompression supported: 1 2024-02-19T21:55:34.707 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: kSnappyCompression supported: 1 2024-02-19T21:55:34.708 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Fast CRC32 supported: Supported on x86 2024-02-19T21:55:34.709 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi053/store.db/MANIFEST-000005 2024-02-19T21:55:34.709 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: 2024-02-19T21:55:34.709 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2024-02-19T21:55:34.709 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: 2024-02-19T21:55:34.709 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-19T21:55:34.710 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.merge_operator: 2024-02-19T21:55:34.710 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.compaction_filter: None 2024-02-19T21:55:34.710 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.compaction_filter_factory: None 2024-02-19T21:55:34.710 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.memtable_factory: SkipListFactory 2024-02-19T21:55:34.710 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.table_factory: BlockBasedTable 2024-02-19T21:55:34.710 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55afb3043cb0) 2024-02-19T21:55:34.710 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cache_index_and_filter_blocks: 1 2024-02-19T21:55:34.710 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cache_index_and_filter_blocks_with_high_priority: 0 2024-02-19T21:55:34.710 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-19T21:55:34.710 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: pin_top_level_index_and_filter: 1 2024-02-19T21:55:34.710 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: index_type: 0 2024-02-19T21:55:34.710 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: data_block_index_type: 0 2024-02-19T21:55:34.710 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: index_shortening: 1 2024-02-19T21:55:34.710 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: data_block_hash_table_util_ratio: 0.750000 2024-02-19T21:55:34.711 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: hash_index_allow_collision: 1 2024-02-19T21:55:34.711 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: checksum: 1 2024-02-19T21:55:34.711 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: no_block_cache: 0 2024-02-19T21:55:34.711 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: block_cache: 0x55afb3078f10 2024-02-19T21:55:34.711 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: block_cache_name: BinnedLRUCache 2024-02-19T21:55:34.711 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: block_cache_options: 2024-02-19T21:55:34.711 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: capacity : 536870912 2024-02-19T21:55:34.711 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: num_shard_bits : 4 2024-02-19T21:55:34.711 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: strict_capacity_limit : 0 2024-02-19T21:55:34.711 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: high_pri_pool_ratio: 0.000 2024-02-19T21:55:34.711 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: block_cache_compressed: (nil) 2024-02-19T21:55:34.711 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: persistent_cache: (nil) 2024-02-19T21:55:34.711 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: block_size: 4096 2024-02-19T21:55:34.711 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: block_size_deviation: 10 2024-02-19T21:55:34.711 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: block_restart_interval: 16 2024-02-19T21:55:34.712 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: index_block_restart_interval: 1 2024-02-19T21:55:34.712 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: metadata_block_size: 4096 2024-02-19T21:55:34.712 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: partition_filters: 0 2024-02-19T21:55:34.712 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: use_delta_encoding: 1 2024-02-19T21:55:34.712 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: filter_policy: rocksdb.BuiltinBloomFilter 2024-02-19T21:55:34.712 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: whole_key_filtering: 1 2024-02-19T21:55:34.712 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: verify_compression: 0 2024-02-19T21:55:34.712 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: read_amp_bytes_per_bit: 0 2024-02-19T21:55:34.712 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: format_version: 2 2024-02-19T21:55:34.712 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: enable_index_compression: 1 2024-02-19T21:55:34.712 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: block_align: 0 2024-02-19T21:55:34.712 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: 2024-02-19T21:55:34.713 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.write_buffer_size: 33554432 2024-02-19T21:55:34.713 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.max_write_buffer_number: 2 2024-02-19T21:55:34.713 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.compression: NoCompression 2024-02-19T21:55:34.713 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.bottommost_compression: Disabled 2024-02-19T21:55:34.713 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.prefix_extractor: nullptr 2024-02-19T21:55:34.713 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-19T21:55:34.713 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.num_levels: 7 2024-02-19T21:55:34.713 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-19T21:55:34.713 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-19T21:55:34.713 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-02-19T21:55:34.713 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-19T21:55:34.713 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-19T21:55:34.713 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-19T21:55:34.713 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-19T21:55:34.714 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-19T21:55:34.714 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-19T21:55:34.714 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.compression_opts.window_bits: -14 2024-02-19T21:55:34.714 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.compression_opts.level: 32767 2024-02-19T21:55:34.714 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.compression_opts.strategy: 0 2024-02-19T21:55:34.714 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-19T21:55:34.714 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-19T21:55:34.714 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.compression_opts.enabled: false 2024-02-19T21:55:34.714 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-19T21:55:34.714 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-19T21:55:34.714 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-19T21:55:34.714 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.target_file_size_base: 67108864 2024-02-19T21:55:34.714 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.target_file_size_multiplier: 1 2024-02-19T21:55:34.714 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-19T21:55:34.715 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-19T21:55:34.715 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-19T21:55:34.715 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-19T21:55:34.715 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-19T21:55:34.715 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-19T21:55:34.715 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-19T21:55:34.715 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-19T21:55:34.715 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-19T21:55:34.715 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-19T21:55:34.715 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-19T21:55:34.715 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-19T21:55:34.715 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.arena_block_size: 4194304 2024-02-19T21:55:34.715 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-19T21:55:34.715 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-19T21:55:34.715 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-02-19T21:55:34.715 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.disable_auto_compactions: 0 2024-02-19T21:55:34.716 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-19T21:55:34.716 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-19T21:55:34.716 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-19T21:55:34.716 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-19T21:55:34.716 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-19T21:55:34.716 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-19T21:55:34.716 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-19T21:55:34.716 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-19T21:55:34.716 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-19T21:55:34.716 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-19T21:55:34.716 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.table_properties_collectors: 2024-02-19T21:55:34.716 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.inplace_update_support: 0 2024-02-19T21:55:34.716 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.inplace_update_num_locks: 10000 2024-02-19T21:55:34.717 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-19T21:55:34.717 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-19T21:55:34.717 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.memtable_huge_page_size: 0 2024-02-19T21:55:34.717 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.bloom_locality: 0 2024-02-19T21:55:34.717 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.max_successive_merges: 0 2024-02-19T21:55:34.717 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.optimize_filters_for_hits: 0 2024-02-19T21:55:34.717 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.paranoid_file_checks: 0 2024-02-19T21:55:34.717 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.force_consistency_checks: 0 2024-02-19T21:55:34.717 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.report_bg_io_stats: 0 2024-02-19T21:55:34.717 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.ttl: 2592000 2024-02-19T21:55:34.717 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.412+0000 7efebd698700 4 rocksdb: Options.periodic_compaction_seconds: 0 2024-02-19T21:55:34.717 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.413+0000 7efebd698700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi053/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 5, log_number is 4,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-02-19T21:55:34.717 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: 2024-02-19T21:55:34.717 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.413+0000 7efebd698700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 4 2024-02-19T21:55:34.717 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: 2024-02-19T21:55:34.718 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.413+0000 7efebd698700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1708379734414545, "job": 1, "event": "recovery_started", "log_files": [6]} 2024-02-19T21:55:34.718 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.413+0000 7efebd698700 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #6 mode 2 2024-02-19T21:55:34.718 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.413+0000 7efebd698700 3 rocksdb: [le/block_based/filter_policy.cc:584] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-02-19T21:55:34.718 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.414+0000 7efebd698700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1708379734415625, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 7, "file_size": 94272, "table_properties": {"data_size": 92360, "index_size": 389, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 709, "raw_key_size": 12563, "raw_average_key_size": 49, "raw_value_size": 85391, "raw_average_value_size": 336, "num_data_blocks": 12, "num_entries": 254, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1708379734, "oldest_key_time": 3, "file_creation_time": 0}} 2024-02-19T21:55:34.718 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.414+0000 7efebd698700 4 rocksdb: [version_set.cc:3826] Creating manifest 8 2024-02-19T21:55:34.718 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: 2024-02-19T21:55:34.718 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.415+0000 7efebd698700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1708379734416121, "job": 1, "event": "recovery_finished"} 2024-02-19T21:55:34.718 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.415+0000 7efebd698700 4 rocksdb: DB pointer 0x55afb313d800 2024-02-19T21:55:34.718 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.415+0000 7efea44b0700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2024-02-19T21:55:34.718 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.415+0000 7efea44b0700 4 rocksdb: [db_impl/db_impl.cc:851] 2024-02-19T21:55:34.718 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: ** DB Stats ** 2024-02-19T21:55:34.718 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: Uptime(secs): 0.0 total, 0.0 interval 2024-02-19T21:55:34.718 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-19T21:55:34.718 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-19T21:55:34.719 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-19T21:55:34.719 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-19T21:55:34.719 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-02-19T21:55:34.719 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-19T21:55:34.719 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: 2024-02-19T21:55:34.719 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: ** Compaction Stats [default] ** 2024-02-19T21:55:34.719 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-19T21:55:34.719 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-19T21:55:34.719 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: L0 2/0 93.75 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 123.8 0.00 0.00 1 0.001 0 0 2024-02-19T21:55:34.719 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: Sum 2/0 93.75 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 123.8 0.00 0.00 1 0.001 0 0 2024-02-19T21:55:34.719 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 123.8 0.00 0.00 1 0.001 0 0 2024-02-19T21:55:34.719 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: 2024-02-19T21:55:34.719 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: ** Compaction Stats [default] ** 2024-02-19T21:55:34.719 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-19T21:55:34.719 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-19T21:55:34.720 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 123.8 0.00 0.00 1 0.001 0 0 2024-02-19T21:55:34.720 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: Uptime(secs): 0.0 total, 0.0 interval 2024-02-19T21:55:34.720 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: Flush(GB): cumulative 0.000, interval 0.000 2024-02-19T21:55:34.720 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-19T21:55:34.720 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: AddFile(Total Files): cumulative 0, interval 0 2024-02-19T21:55:34.720 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: AddFile(L0 Files): cumulative 0, interval 0 2024-02-19T21:55:34.720 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: AddFile(Keys): cumulative 0, interval 0 2024-02-19T21:55:34.720 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: Cumulative compaction: 0.00 GB write, 26.49 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-19T21:55:34.720 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: Interval compaction: 0.00 GB write, 26.49 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-19T21:55:34.720 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-19T21:55:34.720 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: 2024-02-19T21:55:34.720 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: ** File Read Latency Histogram By Level [default] ** 2024-02-19T21:55:34.720 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: 2024-02-19T21:55:34.720 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: ** Compaction Stats [default] ** 2024-02-19T21:55:34.721 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-19T21:55:34.721 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-19T21:55:34.721 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: L0 2/0 93.75 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 123.8 0.00 0.00 1 0.001 0 0 2024-02-19T21:55:34.721 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: Sum 2/0 93.75 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 123.8 0.00 0.00 1 0.001 0 0 2024-02-19T21:55:34.721 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-19T21:55:34.721 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: 2024-02-19T21:55:34.721 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: ** Compaction Stats [default] ** 2024-02-19T21:55:34.721 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-19T21:55:34.721 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-19T21:55:34.721 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 123.8 0.00 0.00 1 0.001 0 0 2024-02-19T21:55:34.721 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: Uptime(secs): 0.0 total, 0.0 interval 2024-02-19T21:55:34.721 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: Flush(GB): cumulative 0.000, interval 0.000 2024-02-19T21:55:34.721 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-19T21:55:34.721 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: AddFile(Total Files): cumulative 0, interval 0 2024-02-19T21:55:34.722 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: AddFile(L0 Files): cumulative 0, interval 0 2024-02-19T21:55:34.722 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: AddFile(Keys): cumulative 0, interval 0 2024-02-19T21:55:34.722 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: Cumulative compaction: 0.00 GB write, 26.20 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-19T21:55:34.722 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-19T21:55:34.722 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-19T21:55:34.722 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: 2024-02-19T21:55:34.722 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: ** File Read Latency Histogram By Level [default] ** 2024-02-19T21:55:34.722 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: 2024-02-19T21:55:34.722 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.416+0000 7efebd698700 0 starting mon.smithi053 rank 0 at public addrs [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] at bind addrs [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi053 fsid 867b6b52-cf71-11ee-95bb-87774f69a715 2024-02-19T21:55:34.722 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.417+0000 7efebd698700 1 mon.smithi053@-1(???) e1 preinit fsid 867b6b52-cf71-11ee-95bb-87774f69a715 2024-02-19T21:55:34.722 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.417+0000 7efebd698700 0 mon.smithi053@-1(???).mds e1 new map 2024-02-19T21:55:34.722 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.417+0000 7efebd698700 0 mon.smithi053@-1(???).mds e1 print_map 2024-02-19T21:55:34.722 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: e1 2024-02-19T21:55:34.722 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: enable_multiple, ever_enabled_multiple: 1,1 2024-02-19T21:55:34.722 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-19T21:55:34.723 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: legacy client fscid: -1 2024-02-19T21:55:34.723 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: 2024-02-19T21:55:34.723 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: No filesystems configured 2024-02-19T21:55:34.723 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: 2024-02-19T21:55:34.723 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.418+0000 7efebd698700 0 mon.smithi053@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-02-19T21:55:34.723 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.418+0000 7efebd698700 0 mon.smithi053@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-19T21:55:34.723 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.418+0000 7efebd698700 0 mon.smithi053@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-19T21:55:34.723 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.418+0000 7efebd698700 0 mon.smithi053@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-19T21:55:34.723 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:55:32.415536+0000 mon.smithi053 (mon.0) 10 : audit [DBG] from='client.? 172.21.15.53:0/55598057' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T21:55:34.723 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.418+0000 7efebd698700 1 mon.smithi053@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-02-19T21:55:34.723 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.419+0000 7efebd698700 20 expand_channel_meta expand map: {default=false} 2024-02-19T21:55:34.723 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.419+0000 7efebd698700 20 expand_channel_meta from 'false' to 'false' 2024-02-19T21:55:34.723 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.419+0000 7efebd698700 20 expand_channel_meta expanded map: {default=false} 2024-02-19T21:55:34.723 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.419+0000 7efebd698700 20 expand_channel_meta expand map: {default=info} 2024-02-19T21:55:34.723 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.419+0000 7efebd698700 20 expand_channel_meta from 'info' to 'info' 2024-02-19T21:55:34.724 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.419+0000 7efebd698700 20 expand_channel_meta expanded map: {default=info} 2024-02-19T21:55:34.724 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.419+0000 7efebd698700 20 expand_channel_meta expand map: {default=daemon} 2024-02-19T21:55:34.724 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.419+0000 7efebd698700 20 expand_channel_meta from 'daemon' to 'daemon' 2024-02-19T21:55:34.724 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.419+0000 7efebd698700 20 expand_channel_meta expanded map: {default=daemon} 2024-02-19T21:55:34.724 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.419+0000 7efebd698700 20 expand_channel_meta expand map: {default=debug} 2024-02-19T21:55:34.724 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.419+0000 7efebd698700 20 expand_channel_meta from 'debug' to 'debug' 2024-02-19T21:55:34.724 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T21:55:34.419+0000 7efebd698700 20 expand_channel_meta expanded map: {default=debug} 2024-02-19T21:55:34.724 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 bash[38135]: e7fa9587a8e806654369aec61f373c94da503231f4654671c666e5ad410a555e 2024-02-19T21:55:34.724 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 systemd[1]: Started Ceph mon.smithi053 for 867b6b52-cf71-11ee-95bb-87774f69a715. 2024-02-19T21:55:34.724 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T21:55:34.434767+0000 mon.smithi053 (mon.0) 1 : cluster [INF] mon.smithi053 is new leader, mons smithi053 in quorum (ranks 0) 2024-02-19T21:55:34.724 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T21:55:34.434815+0000 mon.smithi053 (mon.0) 2 : cluster [DBG] monmap e1: 1 mons at {smithi053=[v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0]} 2024-02-19T21:55:34.724 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T21:55:34.434855+0000 mon.smithi053 (mon.0) 3 : cluster [DBG] fsmap 2024-02-19T21:55:34.724 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T21:55:34.434881+0000 mon.smithi053 (mon.0) 4 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2024-02-19T21:55:34.724 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T21:55:34.435225+0000 mon.smithi053 (mon.0) 5 : cluster [DBG] mgrmap e1: no daemons active 2024-02-19T21:55:35.029 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:35.026+0000 7f429ff89700 1 Processor -- start 2024-02-19T21:55:35.029 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:35.026+0000 7f429ff89700 1 -- start start 2024-02-19T21:55:35.030 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:35.026+0000 7f429ff89700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f42980703f0 0x7f4298100e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T21:55:35.030 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:35.027+0000 7f429ff89700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42980fc880 con 0x7f42980703f0 2024-02-19T21:55:35.030 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:35.027+0000 7f429dd25700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f42980703f0 0x7f4298100e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:55:35.030 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:35.027+0000 7f429dd25700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f42980703f0 0x7f4298100e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:48904/0 (socket says 172.21.15.53:48904) 2024-02-19T21:55:35.030 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:35.027+0000 7f429dd25700 1 -- 172.21.15.53:0/3371279083 learned_addr learned my addr 172.21.15.53:0/3371279083 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-02-19T21:55:35.030 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:35.027+0000 7f429dd25700 1 -- 172.21.15.53:0/3371279083 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4298101380 con 0x7f42980703f0 2024-02-19T21:55:35.030 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:35.027+0000 7f429dd25700 1 --2- 172.21.15.53:0/3371279083 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f42980703f0 0x7f4298100e90 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f429400b5f0 tx=0x7f4294006ee0).ready entity=mon.0 client_cookie=4c91dbc645a12e5f server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:55:35.030 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:35.027+0000 7f429cd23700 1 -- 172.21.15.53:0/3371279083 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f429400d600 con 0x7f42980703f0 2024-02-19T21:55:35.030 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:35.028+0000 7f429cd23700 1 -- 172.21.15.53:0/3371279083 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f429400dbf0 con 0x7f42980703f0 2024-02-19T21:55:35.030 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:35.028+0000 7f429cd23700 1 -- 172.21.15.53:0/3371279083 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4294011ce0 con 0x7f42980703f0 2024-02-19T21:55:35.030 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:35.028+0000 7f429ff89700 1 -- 172.21.15.53:0/3371279083 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f42980703f0 msgr2=0x7f4298100e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:55:35.030 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:35.028+0000 7f429ff89700 1 --2- 172.21.15.53:0/3371279083 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f42980703f0 0x7f4298100e90 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f429400b5f0 tx=0x7f4294006ee0).stop 2024-02-19T21:55:35.030 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:35.028+0000 7f429ff89700 1 -- 172.21.15.53:0/3371279083 shutdown_connections 2024-02-19T21:55:35.030 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:35.028+0000 7f429ff89700 1 --2- 172.21.15.53:0/3371279083 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f42980703f0 0x7f4298100e90 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:55:35.031 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:35.028+0000 7f429ff89700 1 -- 172.21.15.53:0/3371279083 >> 172.21.15.53:0/3371279083 conn(0x7f42980f6530 msgr2=0x7f42980f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:55:35.031 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:35.028+0000 7f429ff89700 1 -- 172.21.15.53:0/3371279083 shutdown_connections 2024-02-19T21:55:35.031 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:35.028+0000 7f429ff89700 1 -- 172.21.15.53:0/3371279083 wait complete. 2024-02-19T21:55:35.031 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:35.029+0000 7f429ff89700 1 Processor -- start 2024-02-19T21:55:35.031 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:35.029+0000 7f429ff89700 1 -- start start 2024-02-19T21:55:35.031 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:35.029+0000 7f429ff89700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f42980703f0 0x7f429810cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:55:35.031 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:35.029+0000 7f429ff89700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f429810d280 con 0x7f42980703f0 2024-02-19T21:55:35.031 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:35.029+0000 7f429dd25700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f42980703f0 0x7f429810cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:55:35.031 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:35.029+0000 7f429dd25700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f42980703f0 0x7f429810cd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:48912/0 (socket says 172.21.15.53:48912) 2024-02-19T21:55:35.031 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:35.029+0000 7f429dd25700 1 -- 172.21.15.53:0/399344419 learned_addr learned my addr 172.21.15.53:0/399344419 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-02-19T21:55:35.031 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:35.029+0000 7f429dd25700 1 -- 172.21.15.53:0/399344419 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f429400b200 con 0x7f42980703f0 2024-02-19T21:55:35.031 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:35.029+0000 7f429dd25700 1 --2- 172.21.15.53:0/399344419 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f42980703f0 0x7f429810cd90 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f4294011ed0 tx=0x7f4294006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:55:35.032 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:35.030+0000 7f428a7fc700 1 -- 172.21.15.53:0/399344419 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f429401b570 con 0x7f42980703f0 2024-02-19T21:55:35.032 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:35.030+0000 7f429ff89700 1 -- 172.21.15.53:0/399344419 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f429810d480 con 0x7f42980703f0 2024-02-19T21:55:35.033 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:35.030+0000 7f428a7fc700 1 -- 172.21.15.53:0/399344419 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f429401bb60 con 0x7f42980703f0 2024-02-19T21:55:35.033 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:35.030+0000 7f428a7fc700 1 -- 172.21.15.53:0/399344419 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f429401aad0 con 0x7f42980703f0 2024-02-19T21:55:35.033 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:35.030+0000 7f429ff89700 1 -- 172.21.15.53:0/399344419 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f429810e090 con 0x7f42980703f0 2024-02-19T21:55:35.033 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:35.030+0000 7f428a7fc700 1 -- 172.21.15.53:0/399344419 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f429400f070 con 0x7f42980703f0 2024-02-19T21:55:35.033 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:35.030+0000 7f428a7fc700 1 -- 172.21.15.53:0/399344419 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f4294025cd0 con 0x7f42980703f0 2024-02-19T21:55:35.033 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:35.031+0000 7f429ff89700 1 -- 172.21.15.53:0/399344419 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4298059100 con 0x7f42980703f0 2024-02-19T21:55:35.033 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:35.033+0000 7f428a7fc700 1 -- 172.21.15.53:0/399344419 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f4294016070 con 0x7f42980703f0 2024-02-19T21:55:35.099 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:35.099+0000 7f429ff89700 1 -- 172.21.15.53:0/399344419 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7f4298102110 con 0x7f42980703f0 2024-02-19T21:55:35.101 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:35.102+0000 7f428a7fc700 1 -- 172.21.15.53:0/399344419 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7f4294025460 con 0x7f42980703f0 2024-02-19T21:55:35.103 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:35.102+0000 7f429ff89700 1 -- 172.21.15.53:0/399344419 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f42980703f0 msgr2=0x7f429810cd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:55:35.103 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:35.102+0000 7f429ff89700 1 --2- 172.21.15.53:0/399344419 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f42980703f0 0x7f429810cd90 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f4294011ed0 tx=0x7f4294006ee0).stop 2024-02-19T21:55:35.103 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:35.102+0000 7f429ff89700 1 -- 172.21.15.53:0/399344419 shutdown_connections 2024-02-19T21:55:35.103 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:35.102+0000 7f429ff89700 1 --2- 172.21.15.53:0/399344419 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f42980703f0 0x7f429810cd90 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:55:35.103 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:35.102+0000 7f429ff89700 1 -- 172.21.15.53:0/399344419 >> 172.21.15.53:0/399344419 conn(0x7f42980f6530 msgr2=0x7f42980f7ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:55:35.103 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:35.103+0000 7f429ff89700 1 -- 172.21.15.53:0/399344419 shutdown_connections 2024-02-19T21:55:35.103 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:35.103+0000 7f429ff89700 1 -- 172.21.15.53:0/399344419 wait complete. 2024-02-19T21:55:35.655 INFO:teuthology.orchestra.run.smithi053.stderr:Wrote config to /etc/ceph/ceph.conf 2024-02-19T21:55:35.655 INFO:teuthology.orchestra.run.smithi053.stderr:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-02-19T21:55:35.655 INFO:teuthology.orchestra.run.smithi053.stderr:Creating mgr... 2024-02-19T21:55:35.656 INFO:teuthology.orchestra.run.smithi053.stderr:Verifying port 9283 ... 2024-02-19T21:55:35.658 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: systemctl daemon-reload 2024-02-19T21:55:35.758 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: systemctl stop ceph-867b6b52-cf71-11ee-95bb-87774f69a715@mgr.smithi053.fyhcdi 2024-02-19T21:55:35.784 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: systemctl reset-failed ceph-867b6b52-cf71-11ee-95bb-87774f69a715@mgr.smithi053.fyhcdi 2024-02-19T21:55:35.790 INFO:teuthology.orchestra.run.smithi053.stderr:systemctl: Failed to reset failed state of unit ceph-867b6b52-cf71-11ee-95bb-87774f69a715@mgr.smithi053.fyhcdi.service: Unit ceph-867b6b52-cf71-11ee-95bb-87774f69a715@mgr.smithi053.fyhcdi.service not loaded. 2024-02-19T21:55:35.791 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: systemctl enable ceph-867b6b52-cf71-11ee-95bb-87774f69a715@mgr.smithi053.fyhcdi 2024-02-19T21:55:35.797 INFO:teuthology.orchestra.run.smithi053.stderr:systemctl: Created symlink /etc/systemd/system/ceph-867b6b52-cf71-11ee-95bb-87774f69a715.target.wants/ceph-867b6b52-cf71-11ee-95bb-87774f69a715@mgr.smithi053.fyhcdi.service → /etc/systemd/system/ceph-867b6b52-cf71-11ee-95bb-87774f69a715@.service. 2024-02-19T21:55:35.884 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: systemctl start ceph-867b6b52-cf71-11ee-95bb-87774f69a715@mgr.smithi053.fyhcdi 2024-02-19T21:55:36.431 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:36 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:55:35.103131+0000 mon.smithi053 (mon.0) 6 : audit [INF] from='client.? 172.21.15.53:0/399344419' entity='client.admin' 2024-02-19T21:55:36.435 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: systemctl is-enabled firewalld.service 2024-02-19T21:55:36.443 INFO:teuthology.orchestra.run.smithi053.stderr:systemctl: disabled 2024-02-19T21:55:36.444 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: systemctl is-active firewalld.service 2024-02-19T21:55:36.452 INFO:teuthology.orchestra.run.smithi053.stderr:systemctl: inactive 2024-02-19T21:55:36.453 INFO:teuthology.orchestra.run.smithi053.stderr:firewalld.service is not enabled 2024-02-19T21:55:36.453 INFO:teuthology.orchestra.run.smithi053.stderr:Not possible to enable service . firewalld.service is not available 2024-02-19T21:55:36.453 INFO:teuthology.orchestra.run.smithi053.stderr:Not possible to open ports <[]>. firewalld.service is not available 2024-02-19T21:55:36.453 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: systemctl is-enabled firewalld.service 2024-02-19T21:55:36.461 INFO:teuthology.orchestra.run.smithi053.stderr:systemctl: disabled 2024-02-19T21:55:36.462 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: systemctl is-active firewalld.service 2024-02-19T21:55:36.470 INFO:teuthology.orchestra.run.smithi053.stderr:systemctl: inactive 2024-02-19T21:55:36.471 INFO:teuthology.orchestra.run.smithi053.stderr:firewalld.service is not enabled 2024-02-19T21:55:36.471 INFO:teuthology.orchestra.run.smithi053.stderr:Not possible to open ports <[9283]>. firewalld.service is not available 2024-02-19T21:55:36.471 INFO:teuthology.orchestra.run.smithi053.stderr:Waiting for mgr to start... 2024-02-19T21:55:36.471 INFO:teuthology.orchestra.run.smithi053.stderr:Waiting for mgr... 2024-02-19T21:55:36.472 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi053 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/867b6b52-cf71-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpoyd_8g9j:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp0zy4ixsn:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2024-02-19T21:55:37.032 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:37.030+0000 7f498be62700 1 Processor -- start 2024-02-19T21:55:37.032 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:37.030+0000 7f498be62700 1 -- start start 2024-02-19T21:55:37.032 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:37.030+0000 7f498be62700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f49840703f0 0x7f49840fea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T21:55:37.032 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:37.030+0000 7f498be62700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49840fef80 con 0x7f49840703f0 2024-02-19T21:55:37.032 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:37.031+0000 7f4989bfe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f49840703f0 0x7f49840fea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:55:37.032 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:37.031+0000 7f4989bfe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f49840703f0 0x7f49840fea90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:51570/0 (socket says 172.21.15.53:51570) 2024-02-19T21:55:37.034 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:37.031+0000 7f4989bfe700 1 -- 172.21.15.53:0/3109330188 learned_addr learned my addr 172.21.15.53:0/3109330188 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-02-19T21:55:37.034 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:37.031+0000 7f4989bfe700 1 -- 172.21.15.53:0/3109330188 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49840ff0c0 con 0x7f49840703f0 2024-02-19T21:55:37.034 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:37.031+0000 7f4989bfe700 1 --2- 172.21.15.53:0/3109330188 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f49840703f0 0x7f49840fea90 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f498000b5f0 tx=0x7f4980006ee0).ready entity=mon.0 client_cookie=edd53098d7821d52 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:55:37.034 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:37.031+0000 7f4988bfc700 1 -- 172.21.15.53:0/3109330188 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f498000d600 con 0x7f49840703f0 2024-02-19T21:55:37.034 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:37.032+0000 7f4988bfc700 1 -- 172.21.15.53:0/3109330188 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f498000dbf0 con 0x7f49840703f0 2024-02-19T21:55:37.034 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:37.032+0000 7f4988bfc700 1 -- 172.21.15.53:0/3109330188 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4980011ce0 con 0x7f49840703f0 2024-02-19T21:55:37.034 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:37.032+0000 7f498be62700 1 -- 172.21.15.53:0/3109330188 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f49840703f0 msgr2=0x7f49840fea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:55:37.034 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:37.032+0000 7f498be62700 1 --2- 172.21.15.53:0/3109330188 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f49840703f0 0x7f49840fea90 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f498000b5f0 tx=0x7f4980006ee0).stop 2024-02-19T21:55:37.034 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:37.032+0000 7f498be62700 1 -- 172.21.15.53:0/3109330188 shutdown_connections 2024-02-19T21:55:37.034 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:37.032+0000 7f498be62700 1 --2- 172.21.15.53:0/3109330188 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f49840703f0 0x7f49840fea90 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:55:37.035 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:37.032+0000 7f498be62700 1 -- 172.21.15.53:0/3109330188 >> 172.21.15.53:0/3109330188 conn(0x7f49840f6530 msgr2=0x7f49840f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:55:37.035 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:37.032+0000 7f498be62700 1 -- 172.21.15.53:0/3109330188 shutdown_connections 2024-02-19T21:55:37.035 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:37.032+0000 7f498be62700 1 -- 172.21.15.53:0/3109330188 wait complete. 2024-02-19T21:55:37.035 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:37.033+0000 7f498be62700 1 Processor -- start 2024-02-19T21:55:37.035 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:37.033+0000 7f498be62700 1 -- start start 2024-02-19T21:55:37.037 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:37.033+0000 7f498be62700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f49840703f0 0x7f498410ab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:55:37.037 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:37.033+0000 7f498be62700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f498410b080 con 0x7f49840703f0 2024-02-19T21:55:37.037 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:37.033+0000 7f4989bfe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f49840703f0 0x7f498410ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:55:37.037 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:37.033+0000 7f4989bfe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f49840703f0 0x7f498410ab90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:51586/0 (socket says 172.21.15.53:51586) 2024-02-19T21:55:37.037 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:37.033+0000 7f4989bfe700 1 -- 172.21.15.53:0/4157547095 learned_addr learned my addr 172.21.15.53:0/4157547095 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-02-19T21:55:37.037 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:37.034+0000 7f4989bfe700 1 -- 172.21.15.53:0/4157547095 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f498000b200 con 0x7f49840703f0 2024-02-19T21:55:37.037 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:37.034+0000 7f4989bfe700 1 --2- 172.21.15.53:0/4157547095 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f49840703f0 0x7f498410ab90 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f49800115c0 tx=0x7f4980006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:55:37.037 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:37.034+0000 7f49727fc700 1 -- 172.21.15.53:0/4157547095 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f498001b410 con 0x7f49840703f0 2024-02-19T21:55:37.037 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:37.034+0000 7f498be62700 1 -- 172.21.15.53:0/4157547095 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f498410b280 con 0x7f49840703f0 2024-02-19T21:55:37.037 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:37.034+0000 7f49727fc700 1 -- 172.21.15.53:0/4157547095 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f498001ba00 con 0x7f49840703f0 2024-02-19T21:55:37.037 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:37.034+0000 7f49727fc700 1 -- 172.21.15.53:0/4157547095 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f498001aaa0 con 0x7f49840703f0 2024-02-19T21:55:37.037 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:37.034+0000 7f498be62700 1 -- 172.21.15.53:0/4157547095 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f498410b710 con 0x7f49840703f0 2024-02-19T21:55:37.037 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:37.034+0000 7f49727fc700 1 -- 172.21.15.53:0/4157547095 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f498000f070 con 0x7f49840703f0 2024-02-19T21:55:37.037 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:37.034+0000 7f49727fc700 1 -- 172.21.15.53:0/4157547095 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f4980024d70 con 0x7f49840703f0 2024-02-19T21:55:37.038 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:37.035+0000 7f498be62700 1 -- 172.21.15.53:0/4157547095 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4984045ad0 con 0x7f49840703f0 2024-02-19T21:55:37.038 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:37.037+0000 7f49727fc700 1 -- 172.21.15.53:0/4157547095 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f4980016070 con 0x7f49840703f0 2024-02-19T21:55:37.107 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:37.107+0000 7f498be62700 1 -- 172.21.15.53:0/4157547095 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f49840ffe00 con 0x7f49840703f0 2024-02-19T21:55:37.108 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:37.107+0000 7f49727fc700 1 -- 172.21.15.53:0/4157547095 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1230 (secure 0 0 0) 0x7f4980024410 con 0x7f49840703f0 2024-02-19T21:55:37.108 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:37.108 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: { 2024-02-19T21:55:37.111 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "fsid": "867b6b52-cf71-11ee-95bb-87774f69a715", 2024-02-19T21:55:37.111 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "health": { 2024-02-19T21:55:37.111 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-02-19T21:55:37.111 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "checks": {}, 2024-02-19T21:55:37.111 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "mutes": [] 2024-02-19T21:55:37.111 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: }, 2024-02-19T21:55:37.111 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-02-19T21:55:37.111 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "quorum": [ 2024-02-19T21:55:37.111 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 0 2024-02-19T21:55:37.112 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: ], 2024-02-19T21:55:37.112 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "quorum_names": [ 2024-02-19T21:55:37.112 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "smithi053" 2024-02-19T21:55:37.112 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: ], 2024-02-19T21:55:37.112 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "quorum_age": 2, 2024-02-19T21:55:37.112 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "monmap": { 2024-02-19T21:55:37.112 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-19T21:55:37.112 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2024-02-19T21:55:37.112 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "num_mons": 1 2024-02-19T21:55:37.112 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: }, 2024-02-19T21:55:37.112 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "osdmap": { 2024-02-19T21:55:37.112 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-19T21:55:37.112 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "num_osds": 0, 2024-02-19T21:55:37.112 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-02-19T21:55:37.113 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-02-19T21:55:37.113 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-02-19T21:55:37.113 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-02-19T21:55:37.113 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-02-19T21:55:37.113 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: }, 2024-02-19T21:55:37.113 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "pgmap": { 2024-02-19T21:55:37.113 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-02-19T21:55:37.118 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-02-19T21:55:37.118 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "num_pools": 0, 2024-02-19T21:55:37.118 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "num_objects": 0, 2024-02-19T21:55:37.118 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-02-19T21:55:37.118 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-02-19T21:55:37.118 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-02-19T21:55:37.118 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "bytes_total": 0 2024-02-19T21:55:37.118 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: }, 2024-02-19T21:55:37.118 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "fsmap": { 2024-02-19T21:55:37.118 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-19T21:55:37.118 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "by_rank": [], 2024-02-19T21:55:37.119 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "up:standby": 0 2024-02-19T21:55:37.119 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: }, 2024-02-19T21:55:37.119 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "mgrmap": { 2024-02-19T21:55:37.119 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "available": false, 2024-02-19T21:55:37.119 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-02-19T21:55:37.119 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "modules": [ 2024-02-19T21:55:37.119 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "iostat", 2024-02-19T21:55:37.119 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "restful" 2024-02-19T21:55:37.119 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: ], 2024-02-19T21:55:37.119 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "services": {} 2024-02-19T21:55:37.119 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: }, 2024-02-19T21:55:37.119 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "servicemap": { 2024-02-19T21:55:37.119 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-19T21:55:37.119 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "modified": "2024-02-19T21:55:29.402051+0000", 2024-02-19T21:55:37.119 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "services": {} 2024-02-19T21:55:37.120 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: }, 2024-02-19T21:55:37.120 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "progress_events": {} 2024-02-19T21:55:37.120 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: } 2024-02-19T21:55:37.120 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:37.108+0000 7f498be62700 1 -- 172.21.15.53:0/4157547095 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f49840703f0 msgr2=0x7f498410ab90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:55:37.120 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:37.108+0000 7f498be62700 1 --2- 172.21.15.53:0/4157547095 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f49840703f0 0x7f498410ab90 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f49800115c0 tx=0x7f4980006ee0).stop 2024-02-19T21:55:37.120 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:37.108+0000 7f498be62700 1 -- 172.21.15.53:0/4157547095 shutdown_connections 2024-02-19T21:55:37.120 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:37.108+0000 7f498be62700 1 --2- 172.21.15.53:0/4157547095 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f49840703f0 0x7f498410ab90 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:55:37.120 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:37.108+0000 7f498be62700 1 -- 172.21.15.53:0/4157547095 >> 172.21.15.53:0/4157547095 conn(0x7f49840f6530 msgr2=0x7f49840f7f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:55:37.120 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:37.108+0000 7f498be62700 1 -- 172.21.15.53:0/4157547095 shutdown_connections 2024-02-19T21:55:37.120 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:37.108+0000 7f498be62700 1 -- 172.21.15.53:0/4157547095 wait complete. 2024-02-19T21:55:37.543 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:37 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:55:37.108698+0000 mon.smithi053 (mon.0) 7 : audit [DBG] from='client.? 172.21.15.53:0/4157547095' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-19T21:55:37.624 INFO:teuthology.orchestra.run.smithi053.stderr:mgr not available, waiting (1/15)... 2024-02-19T21:55:39.626 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi053 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/867b6b52-cf71-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpoyd_8g9j:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp0zy4ixsn:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2024-02-19T21:55:40.205 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:40.188+0000 7fc5d643d700 1 Processor -- start 2024-02-19T21:55:40.205 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:40.188+0000 7fc5d643d700 1 -- start start 2024-02-19T21:55:40.205 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:40.188+0000 7fc5d643d700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc5d00fad90 0x7fc5d00fb1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T21:55:40.205 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:40.188+0000 7fc5d643d700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5d00fb6a0 con 0x7fc5d00fad90 2024-02-19T21:55:40.205 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:40.189+0000 7fc5cffff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc5d00fad90 0x7fc5d00fb1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:55:40.205 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:40.189+0000 7fc5cffff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc5d00fad90 0x7fc5d00fb1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:51600/0 (socket says 172.21.15.53:51600) 2024-02-19T21:55:40.206 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:40.189+0000 7fc5cffff700 1 -- 172.21.15.53:0/1599830220 learned_addr learned my addr 172.21.15.53:0/1599830220 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-02-19T21:55:40.206 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:40.189+0000 7fc5cffff700 1 -- 172.21.15.53:0/1599830220 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5d00fb7e0 con 0x7fc5d00fad90 2024-02-19T21:55:40.206 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:40.189+0000 7fc5cffff700 1 --2- 172.21.15.53:0/1599830220 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc5d00fad90 0x7fc5d00fb1b0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7fc5c000b5f0 tx=0x7fc5c0006ee0).ready entity=mon.0 client_cookie=51b609a94591e1e9 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:55:40.206 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:40.189+0000 7fc5ceffd700 1 -- 172.21.15.53:0/1599830220 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc5c000d600 con 0x7fc5d00fad90 2024-02-19T21:55:40.206 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:40.189+0000 7fc5ceffd700 1 -- 172.21.15.53:0/1599830220 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fc5c000dbf0 con 0x7fc5d00fad90 2024-02-19T21:55:40.206 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:40.189+0000 7fc5ceffd700 1 -- 172.21.15.53:0/1599830220 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc5c0011cb0 con 0x7fc5d00fad90 2024-02-19T21:55:40.206 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:40.190+0000 7fc5d643d700 1 -- 172.21.15.53:0/1599830220 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc5d00fad90 msgr2=0x7fc5d00fb1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:55:40.206 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:40.190+0000 7fc5d643d700 1 --2- 172.21.15.53:0/1599830220 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc5d00fad90 0x7fc5d00fb1b0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7fc5c000b5f0 tx=0x7fc5c0006ee0).stop 2024-02-19T21:55:40.206 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:40.190+0000 7fc5d643d700 1 -- 172.21.15.53:0/1599830220 shutdown_connections 2024-02-19T21:55:40.206 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:40.190+0000 7fc5d643d700 1 --2- 172.21.15.53:0/1599830220 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc5d00fad90 0x7fc5d00fb1b0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:55:40.206 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:40.190+0000 7fc5d643d700 1 -- 172.21.15.53:0/1599830220 >> 172.21.15.53:0/1599830220 conn(0x7fc5d00f6530 msgr2=0x7fc5d00f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:55:40.206 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:40.190+0000 7fc5d643d700 1 -- 172.21.15.53:0/1599830220 shutdown_connections 2024-02-19T21:55:40.206 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:40.190+0000 7fc5d643d700 1 -- 172.21.15.53:0/1599830220 wait complete. 2024-02-19T21:55:40.206 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:40.190+0000 7fc5d643d700 1 Processor -- start 2024-02-19T21:55:40.206 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:40.191+0000 7fc5d643d700 1 -- start start 2024-02-19T21:55:40.207 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:40.191+0000 7fc5d643d700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc5d00fad90 0x7fc5d0108920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:55:40.207 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:40.191+0000 7fc5d643d700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5d0108e10 con 0x7fc5d00fad90 2024-02-19T21:55:40.207 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:40.191+0000 7fc5cffff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc5d00fad90 0x7fc5d0108920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:55:40.207 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:40.191+0000 7fc5cffff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc5d00fad90 0x7fc5d0108920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:51612/0 (socket says 172.21.15.53:51612) 2024-02-19T21:55:40.207 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:40.191+0000 7fc5cffff700 1 -- 172.21.15.53:0/3379355269 learned_addr learned my addr 172.21.15.53:0/3379355269 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-02-19T21:55:40.207 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:40.191+0000 7fc5cffff700 1 -- 172.21.15.53:0/3379355269 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5c000b200 con 0x7fc5d00fad90 2024-02-19T21:55:40.207 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:40.191+0000 7fc5cffff700 1 --2- 172.21.15.53:0/3379355269 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc5d00fad90 0x7fc5d0108920 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7fc5c0011ea0 tx=0x7fc5c0006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:55:40.207 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:40.191+0000 7fc5ccff9700 1 -- 172.21.15.53:0/3379355269 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc5c001b570 con 0x7fc5d00fad90 2024-02-19T21:55:40.207 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:40.192+0000 7fc5ccff9700 1 -- 172.21.15.53:0/3379355269 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fc5c001bb60 con 0x7fc5d00fad90 2024-02-19T21:55:40.207 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:40.192+0000 7fc5d643d700 1 -- 172.21.15.53:0/3379355269 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5d0109010 con 0x7fc5d00fad90 2024-02-19T21:55:40.208 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:40.192+0000 7fc5ccff9700 1 -- 172.21.15.53:0/3379355269 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc5c001a8d0 con 0x7fc5d00fad90 2024-02-19T21:55:40.208 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:40.192+0000 7fc5d643d700 1 -- 172.21.15.53:0/3379355269 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5d01094a0 con 0x7fc5d00fad90 2024-02-19T21:55:40.208 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:40.192+0000 7fc5ccff9700 1 -- 172.21.15.53:0/3379355269 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7fc5c000f070 con 0x7fc5d00fad90 2024-02-19T21:55:40.208 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:40.192+0000 7fc5ccff9700 1 -- 172.21.15.53:0/3379355269 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fc5c0025d50 con 0x7fc5d00fad90 2024-02-19T21:55:40.208 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:40.193+0000 7fc5d643d700 1 -- 172.21.15.53:0/3379355269 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5d0045ad0 con 0x7fc5d00fad90 2024-02-19T21:55:40.209 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:40.195+0000 7fc5ccff9700 1 -- 172.21.15.53:0/3379355269 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7fc5c0016070 con 0x7fc5d00fad90 2024-02-19T21:55:40.273 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:40.266+0000 7fc5d643d700 1 -- 172.21.15.53:0/3379355269 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fc5d00fcae0 con 0x7fc5d00fad90 2024-02-19T21:55:40.273 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:40.266+0000 7fc5ccff9700 1 -- 172.21.15.53:0/3379355269 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1230 (secure 0 0 0) 0x7fc5c0025460 con 0x7fc5d00fad90 2024-02-19T21:55:40.274 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:40.274 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: { 2024-02-19T21:55:40.274 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "fsid": "867b6b52-cf71-11ee-95bb-87774f69a715", 2024-02-19T21:55:40.274 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "health": { 2024-02-19T21:55:40.274 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-02-19T21:55:40.274 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "checks": {}, 2024-02-19T21:55:40.274 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "mutes": [] 2024-02-19T21:55:40.274 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: }, 2024-02-19T21:55:40.274 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-02-19T21:55:40.274 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "quorum": [ 2024-02-19T21:55:40.274 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 0 2024-02-19T21:55:40.274 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: ], 2024-02-19T21:55:40.274 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "quorum_names": [ 2024-02-19T21:55:40.274 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "smithi053" 2024-02-19T21:55:40.274 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: ], 2024-02-19T21:55:40.275 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "quorum_age": 5, 2024-02-19T21:55:40.275 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "monmap": { 2024-02-19T21:55:40.275 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-19T21:55:40.275 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2024-02-19T21:55:40.275 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "num_mons": 1 2024-02-19T21:55:40.275 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: }, 2024-02-19T21:55:40.275 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "osdmap": { 2024-02-19T21:55:40.275 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-19T21:55:40.275 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "num_osds": 0, 2024-02-19T21:55:40.275 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-02-19T21:55:40.275 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-02-19T21:55:40.275 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-02-19T21:55:40.276 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-02-19T21:55:40.276 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-02-19T21:55:40.276 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: }, 2024-02-19T21:55:40.276 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "pgmap": { 2024-02-19T21:55:40.276 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-02-19T21:55:40.276 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-02-19T21:55:40.276 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "num_pools": 0, 2024-02-19T21:55:40.276 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "num_objects": 0, 2024-02-19T21:55:40.276 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-02-19T21:55:40.277 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-02-19T21:55:40.277 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-02-19T21:55:40.277 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "bytes_total": 0 2024-02-19T21:55:40.277 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: }, 2024-02-19T21:55:40.277 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "fsmap": { 2024-02-19T21:55:40.277 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-19T21:55:40.277 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "by_rank": [], 2024-02-19T21:55:40.277 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "up:standby": 0 2024-02-19T21:55:40.277 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: }, 2024-02-19T21:55:40.277 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "mgrmap": { 2024-02-19T21:55:40.277 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "available": false, 2024-02-19T21:55:40.277 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-02-19T21:55:40.277 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "modules": [ 2024-02-19T21:55:40.278 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "iostat", 2024-02-19T21:55:40.278 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "restful" 2024-02-19T21:55:40.278 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: ], 2024-02-19T21:55:40.278 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "services": {} 2024-02-19T21:55:40.278 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: }, 2024-02-19T21:55:40.278 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "servicemap": { 2024-02-19T21:55:40.278 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-19T21:55:40.278 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "modified": "2024-02-19T21:55:29.402051+0000", 2024-02-19T21:55:40.278 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "services": {} 2024-02-19T21:55:40.278 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: }, 2024-02-19T21:55:40.278 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "progress_events": {} 2024-02-19T21:55:40.278 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: } 2024-02-19T21:55:40.278 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:40.267+0000 7fc5d643d700 1 -- 172.21.15.53:0/3379355269 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc5d00fad90 msgr2=0x7fc5d0108920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:55:40.278 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:40.267+0000 7fc5d643d700 1 --2- 172.21.15.53:0/3379355269 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc5d00fad90 0x7fc5d0108920 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7fc5c0011ea0 tx=0x7fc5c0006ee0).stop 2024-02-19T21:55:40.278 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:40.267+0000 7fc5d643d700 1 -- 172.21.15.53:0/3379355269 shutdown_connections 2024-02-19T21:55:40.279 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:40.267+0000 7fc5d643d700 1 --2- 172.21.15.53:0/3379355269 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc5d00fad90 0x7fc5d0108920 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:55:40.279 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:40.267+0000 7fc5d643d700 1 -- 172.21.15.53:0/3379355269 >> 172.21.15.53:0/3379355269 conn(0x7fc5d00f6530 msgr2=0x7fc5d00ff9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:55:40.279 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:40.268+0000 7fc5d643d700 1 -- 172.21.15.53:0/3379355269 shutdown_connections 2024-02-19T21:55:40.279 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:40.268+0000 7fc5d643d700 1 -- 172.21.15.53:0/3379355269 wait complete. 2024-02-19T21:55:40.543 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:40 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:55:40.267888+0000 mon.smithi053 (mon.0) 8 : audit [DBG] from='client.? 172.21.15.53:0/3379355269' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-19T21:55:40.847 INFO:teuthology.orchestra.run.smithi053.stderr:mgr not available, waiting (2/15)... 2024-02-19T21:55:42.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:41 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T21:55:41.530998+0000 mon.smithi053 (mon.0) 9 : cluster [INF] Activating manager daemon smithi053.fyhcdi 2024-02-19T21:55:42.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:41 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T21:55:41.533622+0000 mon.smithi053 (mon.0) 10 : cluster [DBG] mgrmap e2: smithi053.fyhcdi(active, starting, since 0.00271722s) 2024-02-19T21:55:42.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:41 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:55:41.534717+0000 mon.smithi053 (mon.0) 11 : audit [DBG] from='mgr.14100 172.21.15.53:0/2394073443' entity='mgr.smithi053.fyhcdi' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-19T21:55:42.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:41 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:55:41.534898+0000 mon.smithi053 (mon.0) 12 : audit [DBG] from='mgr.14100 172.21.15.53:0/2394073443' entity='mgr.smithi053.fyhcdi' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-19T21:55:42.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:41 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:55:41.535040+0000 mon.smithi053 (mon.0) 13 : audit [DBG] from='mgr.14100 172.21.15.53:0/2394073443' entity='mgr.smithi053.fyhcdi' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-19T21:55:42.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:41 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:55:41.535180+0000 mon.smithi053 (mon.0) 14 : audit [DBG] from='mgr.14100 172.21.15.53:0/2394073443' entity='mgr.smithi053.fyhcdi' cmd=[{"prefix": "mon metadata", "id": "smithi053"}]: dispatch 2024-02-19T21:55:42.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:41 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:55:41.535540+0000 mon.smithi053 (mon.0) 15 : audit [DBG] from='mgr.14100 172.21.15.53:0/2394073443' entity='mgr.smithi053.fyhcdi' cmd=[{"prefix": "mgr metadata", "who": "smithi053.fyhcdi", "id": "smithi053.fyhcdi"}]: dispatch 2024-02-19T21:55:42.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:41 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T21:55:41.549535+0000 mon.smithi053 (mon.0) 16 : cluster [INF] Manager daemon smithi053.fyhcdi is now available 2024-02-19T21:55:42.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:41 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:55:41.556187+0000 mon.smithi053 (mon.0) 17 : audit [INF] from='mgr.14100 172.21.15.53:0/2394073443' entity='mgr.smithi053.fyhcdi' 2024-02-19T21:55:42.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:41 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:55:41.561658+0000 mon.smithi053 (mon.0) 18 : audit [INF] from='mgr.14100 172.21.15.53:0/2394073443' entity='mgr.smithi053.fyhcdi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi053.fyhcdi/mirror_snapshot_schedule"}]: dispatch 2024-02-19T21:55:42.045 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:41 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:55:41.562793+0000 mon.smithi053 (mon.0) 19 : audit [INF] from='mgr.14100 172.21.15.53:0/2394073443' entity='mgr.smithi053.fyhcdi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi053.fyhcdi/trash_purge_schedule"}]: dispatch 2024-02-19T21:55:42.045 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:41 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:55:41.567956+0000 mon.smithi053 (mon.0) 20 : audit [INF] from='mgr.14100 172.21.15.53:0/2394073443' entity='mgr.smithi053.fyhcdi' 2024-02-19T21:55:42.045 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:41 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:55:41.570031+0000 mon.smithi053 (mon.0) 21 : audit [INF] from='mgr.14100 172.21.15.53:0/2394073443' entity='mgr.smithi053.fyhcdi' 2024-02-19T21:55:42.850 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi053 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/867b6b52-cf71-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpoyd_8g9j:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp0zy4ixsn:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2024-02-19T21:55:43.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:43 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T21:55:42.537656+0000 mon.smithi053 (mon.0) 22 : cluster [DBG] mgrmap e3: smithi053.fyhcdi(active, since 1.00675s) 2024-02-19T21:55:45.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:45 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T21:55:44.537233+0000 mon.smithi053 (mon.0) 23 : cluster [DBG] mgrmap e4: smithi053.fyhcdi(active, since 3s) 2024-02-19T21:55:46.612 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:45.983+0000 7f46df4fe700 1 Processor -- start 2024-02-19T21:55:46.612 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:45.984+0000 7f46df4fe700 1 -- start start 2024-02-19T21:55:46.612 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:45.984+0000 7f46df4fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f46d80703d0 0x7f46d8100eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T21:55:46.612 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:45.984+0000 7f46df4fe700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46d80fc8a0 con 0x7f46d80703d0 2024-02-19T21:55:46.612 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:45.984+0000 7f46dd29a700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f46d80703d0 0x7f46d8100eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:55:46.612 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:45.984+0000 7f46dd29a700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f46d80703d0 0x7f46d8100eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:51664/0 (socket says 172.21.15.53:51664) 2024-02-19T21:55:46.612 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:45.984+0000 7f46dd29a700 1 -- 172.21.15.53:0/450828964 learned_addr learned my addr 172.21.15.53:0/450828964 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-02-19T21:55:46.613 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:45.984+0000 7f46dd29a700 1 -- 172.21.15.53:0/450828964 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46d81013a0 con 0x7f46d80703d0 2024-02-19T21:55:46.613 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:45.984+0000 7f46dd29a700 1 --2- 172.21.15.53:0/450828964 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f46d80703d0 0x7f46d8100eb0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f46d400b5f0 tx=0x7f46d4006ee0).ready entity=mon.0 client_cookie=7a180a061815fedd server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:55:46.613 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:45.985+0000 7f46cbfff700 1 -- 172.21.15.53:0/450828964 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f46d400d600 con 0x7f46d80703d0 2024-02-19T21:55:46.613 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:45.985+0000 7f46cbfff700 1 -- 172.21.15.53:0/450828964 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f46d400dbf0 con 0x7f46d80703d0 2024-02-19T21:55:46.613 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:45.985+0000 7f46cbfff700 1 -- 172.21.15.53:0/450828964 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f46d4011cb0 con 0x7f46d80703d0 2024-02-19T21:55:46.613 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:45.985+0000 7f46df4fe700 1 -- 172.21.15.53:0/450828964 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f46d80703d0 msgr2=0x7f46d8100eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:55:46.626 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:45.985+0000 7f46df4fe700 1 --2- 172.21.15.53:0/450828964 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f46d80703d0 0x7f46d8100eb0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f46d400b5f0 tx=0x7f46d4006ee0).stop 2024-02-19T21:55:46.627 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:45.985+0000 7f46df4fe700 1 -- 172.21.15.53:0/450828964 shutdown_connections 2024-02-19T21:55:46.627 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:45.985+0000 7f46df4fe700 1 --2- 172.21.15.53:0/450828964 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f46d80703d0 0x7f46d8100eb0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:55:46.627 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:45.985+0000 7f46df4fe700 1 -- 172.21.15.53:0/450828964 >> 172.21.15.53:0/450828964 conn(0x7f46d80f6530 msgr2=0x7f46d80f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:55:46.627 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:45.985+0000 7f46df4fe700 1 -- 172.21.15.53:0/450828964 shutdown_connections 2024-02-19T21:55:46.627 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:45.985+0000 7f46df4fe700 1 -- 172.21.15.53:0/450828964 wait complete. 2024-02-19T21:55:46.627 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:45.986+0000 7f46df4fe700 1 Processor -- start 2024-02-19T21:55:46.627 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:45.986+0000 7f46df4fe700 1 -- start start 2024-02-19T21:55:46.627 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:45.986+0000 7f46df4fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f46d80703d0 0x7f46d810cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:55:46.627 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:45.986+0000 7f46df4fe700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46d810d2a0 con 0x7f46d80703d0 2024-02-19T21:55:46.627 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:45.986+0000 7f46dd29a700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f46d80703d0 0x7f46d810cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:55:46.627 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:45.986+0000 7f46dd29a700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f46d80703d0 0x7f46d810cdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:51676/0 (socket says 172.21.15.53:51676) 2024-02-19T21:55:46.627 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:45.986+0000 7f46dd29a700 1 -- 172.21.15.53:0/2237720513 learned_addr learned my addr 172.21.15.53:0/2237720513 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-02-19T21:55:46.627 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:45.986+0000 7f46dd29a700 1 -- 172.21.15.53:0/2237720513 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46d400b200 con 0x7f46d80703d0 2024-02-19T21:55:46.628 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:45.987+0000 7f46dd29a700 1 --2- 172.21.15.53:0/2237720513 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f46d80703d0 0x7f46d810cdb0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f46d4006a20 tx=0x7f46d4006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:55:46.628 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:45.987+0000 7f46c9ffb700 1 -- 172.21.15.53:0/2237720513 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f46d401b570 con 0x7f46d80703d0 2024-02-19T21:55:46.628 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:45.987+0000 7f46df4fe700 1 -- 172.21.15.53:0/2237720513 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46d810d4a0 con 0x7f46d80703d0 2024-02-19T21:55:46.628 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:45.987+0000 7f46c9ffb700 1 -- 172.21.15.53:0/2237720513 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f46d401bb60 con 0x7f46d80703d0 2024-02-19T21:55:46.628 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:45.987+0000 7f46c9ffb700 1 -- 172.21.15.53:0/2237720513 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f46d401aa20 con 0x7f46d80703d0 2024-02-19T21:55:46.628 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:45.987+0000 7f46df4fe700 1 -- 172.21.15.53:0/2237720513 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46d810e0e0 con 0x7f46d80703d0 2024-02-19T21:55:46.628 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:45.988+0000 7f46c9ffb700 1 -- 172.21.15.53:0/2237720513 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 4) v1 ==== 36109+0+0 (secure 0 0 0) 0x7f46d400f070 con 0x7f46d80703d0 2024-02-19T21:55:46.628 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:45.988+0000 7f46c9ffb700 1 --2- 172.21.15.53:0/2237720513 >> [v2:172.21.15.53:6800/1836174550,v1:172.21.15.53:6801/1836174550] conn(0x7f46c002fdc0 0x7f46c0032280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:55:46.628 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:45.988+0000 7f46c9ffb700 1 -- 172.21.15.53:0/2237720513 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f46d4041220 con 0x7f46d80703d0 2024-02-19T21:55:46.628 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:45.988+0000 7f46dca99700 1 --2- 172.21.15.53:0/2237720513 >> [v2:172.21.15.53:6800/1836174550,v1:172.21.15.53:6801/1836174550] conn(0x7f46c002fdc0 0x7f46c0032280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:55:46.628 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:45.988+0000 7f46df4fe700 1 -- 172.21.15.53:0/2237720513 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46d810d930 con 0x7f46d80703d0 2024-02-19T21:55:46.628 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:45.988+0000 7f46dca99700 1 --2- 172.21.15.53:0/2237720513 >> [v2:172.21.15.53:6800/1836174550,v1:172.21.15.53:6801/1836174550] conn(0x7f46c002fdc0 0x7f46c0032280 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f46cc0096a0 tx=0x7f46cc006b40).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:55:46.628 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:45.993+0000 7f46c9ffb700 1 -- 172.21.15.53:0/2237720513 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f46d4022960 con 0x7f46d80703d0 2024-02-19T21:55:46.628 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:46.210+0000 7f46df4fe700 1 -- 172.21.15.53:0/2237720513 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f46d81020c0 con 0x7f46d80703d0 2024-02-19T21:55:46.629 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:46.210+0000 7f46c9ffb700 1 -- 172.21.15.53:0/2237720513 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1230 (secure 0 0 0) 0x7f46d4023e90 con 0x7f46d80703d0 2024-02-19T21:55:46.629 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:46.212+0000 7f46df4fe700 1 -- 172.21.15.53:0/2237720513 >> [v2:172.21.15.53:6800/1836174550,v1:172.21.15.53:6801/1836174550] conn(0x7f46c002fdc0 msgr2=0x7f46c0032280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:55:46.629 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:46.212+0000 7f46df4fe700 1 --2- 172.21.15.53:0/2237720513 >> [v2:172.21.15.53:6800/1836174550,v1:172.21.15.53:6801/1836174550] conn(0x7f46c002fdc0 0x7f46c0032280 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f46cc0096a0 tx=0x7f46cc006b40).stop 2024-02-19T21:55:46.629 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:46.212+0000 7f46df4fe700 1 -- 172.21.15.53:0/2237720513 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f46d80703d0 msgr2=0x7f46d810cdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:55:46.629 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:46.212+0000 7f46df4fe700 1 --2- 172.21.15.53:0/2237720513 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f46d80703d0 0x7f46d810cdb0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f46d4006a20 tx=0x7f46d4006ee0).stop 2024-02-19T21:55:46.629 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:46.212+0000 7f46df4fe700 1 -- 172.21.15.53:0/2237720513 shutdown_connections 2024-02-19T21:55:46.629 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:46.212+0000 7f46df4fe700 1 --2- 172.21.15.53:0/2237720513 >> [v2:172.21.15.53:6800/1836174550,v1:172.21.15.53:6801/1836174550] conn(0x7f46c002fdc0 0x7f46c0032280 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:55:46.629 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:46.212+0000 7f46df4fe700 1 --2- 172.21.15.53:0/2237720513 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f46d80703d0 0x7f46d810cdb0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:55:46.629 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:46.212+0000 7f46df4fe700 1 -- 172.21.15.53:0/2237720513 >> 172.21.15.53:0/2237720513 conn(0x7f46d80f6530 msgr2=0x7f46d80f7f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:55:46.629 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:46.212+0000 7f46df4fe700 1 -- 172.21.15.53:0/2237720513 shutdown_connections 2024-02-19T21:55:46.629 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:46.212+0000 7f46df4fe700 1 -- 172.21.15.53:0/2237720513 wait complete. 2024-02-19T21:55:46.629 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:46.629 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: { 2024-02-19T21:55:46.629 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "fsid": "867b6b52-cf71-11ee-95bb-87774f69a715", 2024-02-19T21:55:46.629 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "health": { 2024-02-19T21:55:46.630 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-02-19T21:55:46.630 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "checks": {}, 2024-02-19T21:55:46.630 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "mutes": [] 2024-02-19T21:55:46.630 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: }, 2024-02-19T21:55:46.630 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-02-19T21:55:46.630 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "quorum": [ 2024-02-19T21:55:46.630 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 0 2024-02-19T21:55:46.630 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: ], 2024-02-19T21:55:46.630 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "quorum_names": [ 2024-02-19T21:55:46.630 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "smithi053" 2024-02-19T21:55:46.630 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: ], 2024-02-19T21:55:46.630 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "quorum_age": 11, 2024-02-19T21:55:46.630 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "monmap": { 2024-02-19T21:55:46.630 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-19T21:55:46.630 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2024-02-19T21:55:46.631 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "num_mons": 1 2024-02-19T21:55:46.631 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: }, 2024-02-19T21:55:46.631 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "osdmap": { 2024-02-19T21:55:46.631 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-19T21:55:46.631 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "num_osds": 0, 2024-02-19T21:55:46.631 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-02-19T21:55:46.631 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-02-19T21:55:46.631 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-02-19T21:55:46.631 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-02-19T21:55:46.631 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-02-19T21:55:46.631 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: }, 2024-02-19T21:55:46.631 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "pgmap": { 2024-02-19T21:55:46.631 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-02-19T21:55:46.631 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-02-19T21:55:46.631 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "num_pools": 0, 2024-02-19T21:55:46.631 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "num_objects": 0, 2024-02-19T21:55:46.632 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-02-19T21:55:46.632 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-02-19T21:55:46.632 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-02-19T21:55:46.632 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "bytes_total": 0 2024-02-19T21:55:46.632 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: }, 2024-02-19T21:55:46.632 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "fsmap": { 2024-02-19T21:55:46.632 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-19T21:55:46.632 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "by_rank": [], 2024-02-19T21:55:46.632 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "up:standby": 0 2024-02-19T21:55:46.632 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: }, 2024-02-19T21:55:46.632 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "mgrmap": { 2024-02-19T21:55:46.632 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "available": true, 2024-02-19T21:55:46.632 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-02-19T21:55:46.632 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "modules": [ 2024-02-19T21:55:46.632 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "iostat", 2024-02-19T21:55:46.633 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "restful" 2024-02-19T21:55:46.633 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: ], 2024-02-19T21:55:46.633 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "services": {} 2024-02-19T21:55:46.633 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: }, 2024-02-19T21:55:46.633 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "servicemap": { 2024-02-19T21:55:46.633 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-19T21:55:46.633 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "modified": "2024-02-19T21:55:29.402051+0000", 2024-02-19T21:55:46.633 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "services": {} 2024-02-19T21:55:46.633 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: }, 2024-02-19T21:55:46.633 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "progress_events": {} 2024-02-19T21:55:46.633 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: } 2024-02-19T21:55:46.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:46 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:55:46.211563+0000 mon.smithi053 (mon.0) 24 : audit [DBG] from='client.? 172.21.15.53:0/2237720513' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-19T21:55:51.109 INFO:teuthology.orchestra.run.smithi053.stderr:mgr is available 2024-02-19T21:55:51.109 INFO:teuthology.orchestra.run.smithi053.stderr:Enabling cephadm module... 2024-02-19T21:55:51.109 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi053 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/867b6b52-cf71-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpoyd_8g9j:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp0zy4ixsn:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable cephadm 2024-02-19T21:55:53.773 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:53.769+0000 7fcb431b7700 1 Processor -- start 2024-02-19T21:55:53.773 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:53.769+0000 7fcb431b7700 1 -- start start 2024-02-19T21:55:53.773 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:53.769+0000 7fcb431b7700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcb3c070430 0x7fcb3c0fe890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T21:55:53.773 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:53.769+0000 7fcb431b7700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb3c0fed80 con 0x7fcb3c070430 2024-02-19T21:55:53.773 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:53.769+0000 7fcb421b5700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcb3c070430 0x7fcb3c0fe890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:55:53.773 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:53.769+0000 7fcb421b5700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcb3c070430 0x7fcb3c0fe890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:38924/0 (socket says 172.21.15.53:38924) 2024-02-19T21:55:53.773 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:53.769+0000 7fcb421b5700 1 -- 172.21.15.53:0/1729646826 learned_addr learned my addr 172.21.15.53:0/1729646826 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-02-19T21:55:53.773 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:53.770+0000 7fcb421b5700 1 -- 172.21.15.53:0/1729646826 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb3c0feec0 con 0x7fcb3c070430 2024-02-19T21:55:53.774 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:53.770+0000 7fcb421b5700 1 --2- 172.21.15.53:0/1729646826 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcb3c070430 0x7fcb3c0fe890 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7fcb3800b5f0 tx=0x7fcb38006ee0).ready entity=mon.0 client_cookie=ae3a71f4c537ce14 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:55:53.774 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:53.770+0000 7fcb411b3700 1 -- 172.21.15.53:0/1729646826 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcb3800d600 con 0x7fcb3c070430 2024-02-19T21:55:53.774 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:53.770+0000 7fcb411b3700 1 -- 172.21.15.53:0/1729646826 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fcb3800dbf0 con 0x7fcb3c070430 2024-02-19T21:55:53.774 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:53.770+0000 7fcb411b3700 1 -- 172.21.15.53:0/1729646826 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcb38011cb0 con 0x7fcb3c070430 2024-02-19T21:55:53.774 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:53.770+0000 7fcb431b7700 1 -- 172.21.15.53:0/1729646826 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcb3c070430 msgr2=0x7fcb3c0fe890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:55:53.774 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:53.770+0000 7fcb431b7700 1 --2- 172.21.15.53:0/1729646826 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcb3c070430 0x7fcb3c0fe890 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7fcb3800b5f0 tx=0x7fcb38006ee0).stop 2024-02-19T21:55:53.774 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:53.771+0000 7fcb431b7700 1 -- 172.21.15.53:0/1729646826 shutdown_connections 2024-02-19T21:55:53.774 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:53.771+0000 7fcb431b7700 1 --2- 172.21.15.53:0/1729646826 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcb3c070430 0x7fcb3c0fe890 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:55:53.774 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:53.771+0000 7fcb431b7700 1 -- 172.21.15.53:0/1729646826 >> 172.21.15.53:0/1729646826 conn(0x7fcb3c0f6390 msgr2=0x7fcb3c0f87b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:55:53.774 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:53.771+0000 7fcb431b7700 1 -- 172.21.15.53:0/1729646826 shutdown_connections 2024-02-19T21:55:53.774 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:53.771+0000 7fcb431b7700 1 -- 172.21.15.53:0/1729646826 wait complete. 2024-02-19T21:55:53.774 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:53.771+0000 7fcb431b7700 1 Processor -- start 2024-02-19T21:55:53.774 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:53.771+0000 7fcb431b7700 1 -- start start 2024-02-19T21:55:53.775 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:53.771+0000 7fcb431b7700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcb3c070430 0x7fcb3c0fc310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:55:53.775 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:53.771+0000 7fcb431b7700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb3c0fa790 con 0x7fcb3c070430 2024-02-19T21:55:53.775 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:53.772+0000 7fcb421b5700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcb3c070430 0x7fcb3c0fc310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:55:53.775 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:53.772+0000 7fcb421b5700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcb3c070430 0x7fcb3c0fc310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:38938/0 (socket says 172.21.15.53:38938) 2024-02-19T21:55:53.775 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:53.772+0000 7fcb421b5700 1 -- 172.21.15.53:0/1427801890 learned_addr learned my addr 172.21.15.53:0/1427801890 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-02-19T21:55:53.775 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:53.772+0000 7fcb421b5700 1 -- 172.21.15.53:0/1427801890 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb3800b200 con 0x7fcb3c070430 2024-02-19T21:55:53.775 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:53.772+0000 7fcb421b5700 1 --2- 172.21.15.53:0/1427801890 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcb3c070430 0x7fcb3c0fc310 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7fcb38006800 tx=0x7fcb38006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:55:53.775 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:53.772+0000 7fcb2affd700 1 -- 172.21.15.53:0/1427801890 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcb3801b570 con 0x7fcb3c070430 2024-02-19T21:55:53.775 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:53.772+0000 7fcb2affd700 1 -- 172.21.15.53:0/1427801890 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fcb3801bb60 con 0x7fcb3c070430 2024-02-19T21:55:53.775 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:53.772+0000 7fcb431b7700 1 -- 172.21.15.53:0/1427801890 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb3c0fa930 con 0x7fcb3c070430 2024-02-19T21:55:53.775 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:53.772+0000 7fcb2affd700 1 -- 172.21.15.53:0/1427801890 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcb3801a9f0 con 0x7fcb3c070430 2024-02-19T21:55:53.775 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:53.772+0000 7fcb431b7700 1 -- 172.21.15.53:0/1427801890 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb3c0fadc0 con 0x7fcb3c070430 2024-02-19T21:55:53.775 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:53.773+0000 7fcb2affd700 1 -- 172.21.15.53:0/1427801890 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 4) v1 ==== 36109+0+0 (secure 0 0 0) 0x7fcb3800f070 con 0x7fcb3c070430 2024-02-19T21:55:53.776 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:53.773+0000 7fcb2affd700 1 --2- 172.21.15.53:0/1427801890 >> [v2:172.21.15.53:6800/1836174550,v1:172.21.15.53:6801/1836174550] conn(0x7fcb2402fe10 0x7fcb240322d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:55:53.776 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:53.773+0000 7fcb2affd700 1 -- 172.21.15.53:0/1427801890 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fcb380412a0 con 0x7fcb3c070430 2024-02-19T21:55:53.776 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:53.773+0000 7fcb431b7700 1 -- 172.21.15.53:0/1427801890 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb3c045ad0 con 0x7fcb3c070430 2024-02-19T21:55:53.776 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:53.773+0000 7fcb419b4700 1 --2- 172.21.15.53:0/1427801890 >> [v2:172.21.15.53:6800/1836174550,v1:172.21.15.53:6801/1836174550] conn(0x7fcb2402fe10 0x7fcb240322d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:55:53.777 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:53.774+0000 7fcb419b4700 1 --2- 172.21.15.53:0/1427801890 >> [v2:172.21.15.53:6800/1836174550,v1:172.21.15.53:6801/1836174550] conn(0x7fcb2402fe10 0x7fcb240322d0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fcb2c0096a0 tx=0x7fcb2c006b40).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:55:53.778 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:53.778+0000 7fcb2affd700 1 -- 172.21.15.53:0/1427801890 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcb38016070 con 0x7fcb3c070430 2024-02-19T21:55:53.977 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:53.977+0000 7fcb431b7700 1 -- 172.21.15.53:0/1427801890 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7fcb3c027a00 con 0x7fcb3c070430 2024-02-19T21:55:54.030 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:54.029+0000 7fcb2affd700 1 -- 172.21.15.53:0/1427801890 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mgrmap(e 5) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fcb3801ab50 con 0x7fcb3c070430 2024-02-19T21:55:54.030 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:54.031+0000 7fcb2affd700 1 -- 172.21.15.53:0/1427801890 <== mon.0 v2:172.21.15.53:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7fcb380247d0 con 0x7fcb3c070430 2024-02-19T21:55:54.034 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:54.033+0000 7fcb431b7700 1 -- 172.21.15.53:0/1427801890 >> [v2:172.21.15.53:6800/1836174550,v1:172.21.15.53:6801/1836174550] conn(0x7fcb2402fe10 msgr2=0x7fcb240322d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:55:54.034 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:54.033+0000 7fcb431b7700 1 --2- 172.21.15.53:0/1427801890 >> [v2:172.21.15.53:6800/1836174550,v1:172.21.15.53:6801/1836174550] conn(0x7fcb2402fe10 0x7fcb240322d0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fcb2c0096a0 tx=0x7fcb2c006b40).stop 2024-02-19T21:55:54.034 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:54.033+0000 7fcb431b7700 1 -- 172.21.15.53:0/1427801890 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcb3c070430 msgr2=0x7fcb3c0fc310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:55:54.034 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:54.033+0000 7fcb431b7700 1 --2- 172.21.15.53:0/1427801890 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcb3c070430 0x7fcb3c0fc310 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7fcb38006800 tx=0x7fcb38006ee0).stop 2024-02-19T21:55:54.035 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:54.033+0000 7fcb431b7700 1 -- 172.21.15.53:0/1427801890 shutdown_connections 2024-02-19T21:55:54.035 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:54.033+0000 7fcb431b7700 1 --2- 172.21.15.53:0/1427801890 >> [v2:172.21.15.53:6800/1836174550,v1:172.21.15.53:6801/1836174550] conn(0x7fcb2402fe10 0x7fcb240322d0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:55:54.035 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:54.033+0000 7fcb431b7700 1 --2- 172.21.15.53:0/1427801890 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcb3c070430 0x7fcb3c0fc310 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:55:54.035 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:54.033+0000 7fcb431b7700 1 -- 172.21.15.53:0/1427801890 >> 172.21.15.53:0/1427801890 conn(0x7fcb3c0f6390 msgr2=0x7fcb3c0f7d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:55:54.035 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:54.033+0000 7fcb431b7700 1 -- 172.21.15.53:0/1427801890 shutdown_connections 2024-02-19T21:55:54.035 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:54.034+0000 7fcb431b7700 1 -- 172.21.15.53:0/1427801890 wait complete. 2024-02-19T21:55:54.293 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:54 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:55:53.978941+0000 mon.smithi053 (mon.0) 25 : audit [INF] from='client.? 172.21.15.53:0/1427801890' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-02-19T21:55:54.621 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi053 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/867b6b52-cf71-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpoyd_8g9j:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp0zy4ixsn:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr stat 2024-02-19T21:55:55.151 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:55.142+0000 7fa8c5362700 1 Processor -- start 2024-02-19T21:55:55.151 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:55.142+0000 7fa8c5362700 1 -- start start 2024-02-19T21:55:55.151 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:55.142+0000 7fa8c5362700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa8c00703f0 0x7fa8c0100cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T21:55:55.151 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:55.142+0000 7fa8c5362700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8c00fc6c0 con 0x7fa8c00703f0 2024-02-19T21:55:55.151 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:55.142+0000 7fa8bffff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa8c00703f0 0x7fa8c0100cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:55:55.151 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:55.142+0000 7fa8bffff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa8c00703f0 0x7fa8c0100cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:38960/0 (socket says 172.21.15.53:38960) 2024-02-19T21:55:55.151 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:55.142+0000 7fa8bffff700 1 -- 172.21.15.53:0/878915753 learned_addr learned my addr 172.21.15.53:0/878915753 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-02-19T21:55:55.151 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:55.143+0000 7fa8bffff700 1 -- 172.21.15.53:0/878915753 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8c01011c0 con 0x7fa8c00703f0 2024-02-19T21:55:55.152 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:55.143+0000 7fa8bffff700 1 --2- 172.21.15.53:0/878915753 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa8c00703f0 0x7fa8c0100cd0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7fa8b000b5f0 tx=0x7fa8b0006ee0).ready entity=mon.0 client_cookie=8edd970330568df4 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:55:55.152 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:55.143+0000 7fa8beffd700 1 -- 172.21.15.53:0/878915753 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa8b000d600 con 0x7fa8c00703f0 2024-02-19T21:55:55.152 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:55.143+0000 7fa8beffd700 1 -- 172.21.15.53:0/878915753 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fa8b000dbf0 con 0x7fa8c00703f0 2024-02-19T21:55:55.152 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:55.143+0000 7fa8beffd700 1 -- 172.21.15.53:0/878915753 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa8b0011ce0 con 0x7fa8c00703f0 2024-02-19T21:55:55.152 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:55.143+0000 7fa8c5362700 1 -- 172.21.15.53:0/878915753 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa8c00703f0 msgr2=0x7fa8c0100cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:55:55.152 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:55.143+0000 7fa8c5362700 1 --2- 172.21.15.53:0/878915753 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa8c00703f0 0x7fa8c0100cd0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7fa8b000b5f0 tx=0x7fa8b0006ee0).stop 2024-02-19T21:55:55.152 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:55.143+0000 7fa8c5362700 1 -- 172.21.15.53:0/878915753 shutdown_connections 2024-02-19T21:55:55.152 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:55.143+0000 7fa8c5362700 1 --2- 172.21.15.53:0/878915753 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa8c00703f0 0x7fa8c0100cd0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:55:55.152 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:55.143+0000 7fa8c5362700 1 -- 172.21.15.53:0/878915753 >> 172.21.15.53:0/878915753 conn(0x7fa8c00f6390 msgr2=0x7fa8c00f87b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:55:55.152 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:55.144+0000 7fa8c5362700 1 -- 172.21.15.53:0/878915753 shutdown_connections 2024-02-19T21:55:55.152 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:55.144+0000 7fa8c5362700 1 -- 172.21.15.53:0/878915753 wait complete. 2024-02-19T21:55:55.152 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:55.144+0000 7fa8c5362700 1 Processor -- start 2024-02-19T21:55:55.152 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:55.144+0000 7fa8c5362700 1 -- start start 2024-02-19T21:55:55.153 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:55.144+0000 7fa8c5362700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa8c00703f0 0x7fa8c010cb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:55:55.153 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:55.144+0000 7fa8c5362700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8c010d070 con 0x7fa8c00703f0 2024-02-19T21:55:55.153 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:55.145+0000 7fa8bffff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa8c00703f0 0x7fa8c010cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:55:55.153 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:55.145+0000 7fa8bffff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa8c00703f0 0x7fa8c010cb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:38976/0 (socket says 172.21.15.53:38976) 2024-02-19T21:55:55.153 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:55.145+0000 7fa8bffff700 1 -- 172.21.15.53:0/2612724391 learned_addr learned my addr 172.21.15.53:0/2612724391 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-02-19T21:55:55.153 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:55.145+0000 7fa8bffff700 1 -- 172.21.15.53:0/2612724391 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8b000b200 con 0x7fa8c00703f0 2024-02-19T21:55:55.153 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:55.145+0000 7fa8bffff700 1 --2- 172.21.15.53:0/2612724391 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa8c00703f0 0x7fa8c010cb80 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7fa8b0011ed0 tx=0x7fa8b0006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:55:55.153 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:55.145+0000 7fa8bcff9700 1 -- 172.21.15.53:0/2612724391 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa8b001b570 con 0x7fa8c00703f0 2024-02-19T21:55:55.153 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:55.145+0000 7fa8c5362700 1 -- 172.21.15.53:0/2612724391 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8c010d270 con 0x7fa8c00703f0 2024-02-19T21:55:55.153 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:55.145+0000 7fa8bcff9700 1 -- 172.21.15.53:0/2612724391 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fa8b001bb60 con 0x7fa8c00703f0 2024-02-19T21:55:55.153 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:55.145+0000 7fa8bcff9700 1 -- 172.21.15.53:0/2612724391 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa8b001aad0 con 0x7fa8c00703f0 2024-02-19T21:55:55.153 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:55.145+0000 7fa8c5362700 1 -- 172.21.15.53:0/2612724391 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8c010d700 con 0x7fa8c00703f0 2024-02-19T21:55:55.153 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:55.146+0000 7fa8bcff9700 1 -- 172.21.15.53:0/2612724391 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fa8b000f070 con 0x7fa8c00703f0 2024-02-19T21:55:55.153 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:55.146+0000 7fa8c5362700 1 -- 172.21.15.53:0/2612724391 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8c0045ad0 con 0x7fa8c00703f0 2024-02-19T21:55:55.154 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:55.146+0000 7fa8bcff9700 1 --2- 172.21.15.53:0/2612724391 >> [v2:172.21.15.53:6800/1836174550,v1:172.21.15.53:6801/1836174550] conn(0x7fa8ac02fe10 0x7fa8ac0322d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:55:55.154 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:55.146+0000 7fa8bcff9700 1 -- 172.21.15.53:0/2612724391 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fa8b0041320 con 0x7fa8c00703f0 2024-02-19T21:55:55.154 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:55.146+0000 7fa8bf7fe700 1 -- 172.21.15.53:0/2612724391 >> [v2:172.21.15.53:6800/1836174550,v1:172.21.15.53:6801/1836174550] conn(0x7fa8ac02fe10 msgr2=0x7fa8ac0322d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.53:6800/1836174550 2024-02-19T21:55:55.154 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:55.146+0000 7fa8bf7fe700 1 --2- 172.21.15.53:0/2612724391 >> [v2:172.21.15.53:6800/1836174550,v1:172.21.15.53:6801/1836174550] conn(0x7fa8ac02fe10 0x7fa8ac0322d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-02-19T21:55:55.154 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:55.151+0000 7fa8bcff9700 1 -- 172.21.15.53:0/2612724391 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa8b0010360 con 0x7fa8c00703f0 2024-02-19T21:55:55.293 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:55 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:55:54.032130+0000 mon.smithi053 (mon.0) 26 : audit [INF] from='client.? 172.21.15.53:0/1427801890' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-02-19T21:55:55.293 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:55 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T21:55:54.032174+0000 mon.smithi053 (mon.0) 27 : cluster [DBG] mgrmap e5: smithi053.fyhcdi(active, since 12s) 2024-02-19T21:55:55.347 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:55.347+0000 7fa8bf7fe700 1 -- 172.21.15.53:0/2612724391 >> [v2:172.21.15.53:6800/1836174550,v1:172.21.15.53:6801/1836174550] conn(0x7fa8ac02fe10 msgr2=0x7fa8ac0322d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.53:6800/1836174550 2024-02-19T21:55:55.347 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:55.347+0000 7fa8bf7fe700 1 --2- 172.21.15.53:0/2612724391 >> [v2:172.21.15.53:6800/1836174550,v1:172.21.15.53:6801/1836174550] conn(0x7fa8ac02fe10 0x7fa8ac0322d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2024-02-19T21:55:55.394 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:55.393+0000 7fa8c5362700 1 -- 172.21.15.53:0/2612724391 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7fa8c0106680 con 0x7fa8c00703f0 2024-02-19T21:55:55.394 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:55.394+0000 7fa8bcff9700 1 -- 172.21.15.53:0/2612724391 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+103 (secure 0 0 0) 0x7fa8b0016070 con 0x7fa8c00703f0 2024-02-19T21:55:55.395 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: { 2024-02-19T21:55:55.395 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "epoch": 5, 2024-02-19T21:55:55.395 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "available": true, 2024-02-19T21:55:55.395 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "active_name": "smithi053.fyhcdi", 2024-02-19T21:55:55.395 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "num_standby": 0 2024-02-19T21:55:55.395 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: } 2024-02-19T21:55:55.396 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:55.396+0000 7fa8c5362700 1 -- 172.21.15.53:0/2612724391 >> [v2:172.21.15.53:6800/1836174550,v1:172.21.15.53:6801/1836174550] conn(0x7fa8ac02fe10 msgr2=0x7fa8ac0322d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-19T21:55:55.396 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:55.396+0000 7fa8c5362700 1 --2- 172.21.15.53:0/2612724391 >> [v2:172.21.15.53:6800/1836174550,v1:172.21.15.53:6801/1836174550] conn(0x7fa8ac02fe10 0x7fa8ac0322d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-19T21:55:55.396 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:55.396+0000 7fa8c5362700 1 -- 172.21.15.53:0/2612724391 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa8c00703f0 msgr2=0x7fa8c010cb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:55:55.396 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:55.396+0000 7fa8c5362700 1 --2- 172.21.15.53:0/2612724391 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa8c00703f0 0x7fa8c010cb80 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7fa8b0011ed0 tx=0x7fa8b0006ee0).stop 2024-02-19T21:55:55.397 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:55.396+0000 7fa8c5362700 1 -- 172.21.15.53:0/2612724391 shutdown_connections 2024-02-19T21:55:55.397 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:55.396+0000 7fa8c5362700 1 --2- 172.21.15.53:0/2612724391 >> [v2:172.21.15.53:6800/1836174550,v1:172.21.15.53:6801/1836174550] conn(0x7fa8ac02fe10 0x7fa8ac0322d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-19T21:55:55.397 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:55.396+0000 7fa8c5362700 1 --2- 172.21.15.53:0/2612724391 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa8c00703f0 0x7fa8c010cb80 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:55:55.397 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:55.396+0000 7fa8c5362700 1 -- 172.21.15.53:0/2612724391 >> 172.21.15.53:0/2612724391 conn(0x7fa8c00f6390 msgr2=0x7fa8c00f7d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:55:55.397 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:55.396+0000 7fa8c5362700 1 -- 172.21.15.53:0/2612724391 shutdown_connections 2024-02-19T21:55:55.397 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:55.396+0000 7fa8c5362700 1 -- 172.21.15.53:0/2612724391 wait complete. 2024-02-19T21:55:56.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:55:56 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:55:55.395269+0000 mon.smithi053 (mon.0) 28 : audit [DBG] from='client.? 172.21.15.53:0/2612724391' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-02-19T21:55:56.062 INFO:teuthology.orchestra.run.smithi053.stderr:Waiting for the mgr to restart... 2024-02-19T21:55:56.062 INFO:teuthology.orchestra.run.smithi053.stderr:Waiting for mgr epoch 5... 2024-02-19T21:55:56.062 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi053 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/867b6b52-cf71-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpoyd_8g9j:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp0zy4ixsn:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 tell mgr mgr_status 2024-02-19T21:55:56.596 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:56.595+0000 7f0e26b45700 1 Processor -- start 2024-02-19T21:55:56.597 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:56.595+0000 7f0e26b45700 1 -- start start 2024-02-19T21:55:56.597 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:56.595+0000 7f0e26b45700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0e20070430 0x7f0e200fea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T21:55:56.597 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:56.595+0000 7f0e26b45700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e200fef40 con 0x7f0e20070430 2024-02-19T21:55:56.597 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:56.596+0000 7f0e248e1700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0e20070430 0x7f0e200fea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:55:56.597 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:56.596+0000 7f0e248e1700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0e20070430 0x7f0e200fea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:47688/0 (socket says 172.21.15.53:47688) 2024-02-19T21:55:56.597 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:56.596+0000 7f0e248e1700 1 -- 172.21.15.53:0/3428157022 learned_addr learned my addr 172.21.15.53:0/3428157022 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-02-19T21:55:56.597 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:56.596+0000 7f0e248e1700 1 -- 172.21.15.53:0/3428157022 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e200ff080 con 0x7f0e20070430 2024-02-19T21:55:56.597 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:56.596+0000 7f0e248e1700 1 --2- 172.21.15.53:0/3428157022 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0e20070430 0x7f0e200fea50 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f0e1400b5f0 tx=0x7f0e14006ee0).ready entity=mon.0 client_cookie=70eb58d4127de55a server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:55:56.597 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:56.596+0000 7f0e1f7fe700 1 -- 172.21.15.53:0/3428157022 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0e1400d600 con 0x7f0e20070430 2024-02-19T21:55:56.599 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:56.596+0000 7f0e1f7fe700 1 -- 172.21.15.53:0/3428157022 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f0e1400dbf0 con 0x7f0e20070430 2024-02-19T21:55:56.599 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:56.596+0000 7f0e1f7fe700 1 -- 172.21.15.53:0/3428157022 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0e14011cb0 con 0x7f0e20070430 2024-02-19T21:55:56.599 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:56.597+0000 7f0e26b45700 1 -- 172.21.15.53:0/3428157022 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0e20070430 msgr2=0x7f0e200fea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:55:56.599 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:56.597+0000 7f0e26b45700 1 --2- 172.21.15.53:0/3428157022 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0e20070430 0x7f0e200fea50 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f0e1400b5f0 tx=0x7f0e14006ee0).stop 2024-02-19T21:55:56.599 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:56.597+0000 7f0e26b45700 1 -- 172.21.15.53:0/3428157022 shutdown_connections 2024-02-19T21:55:56.599 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:56.597+0000 7f0e26b45700 1 --2- 172.21.15.53:0/3428157022 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0e20070430 0x7f0e200fea50 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:55:56.599 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:56.597+0000 7f0e26b45700 1 -- 172.21.15.53:0/3428157022 >> 172.21.15.53:0/3428157022 conn(0x7f0e200f6530 msgr2=0x7f0e200f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:55:56.599 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:56.597+0000 7f0e26b45700 1 -- 172.21.15.53:0/3428157022 shutdown_connections 2024-02-19T21:55:56.599 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:56.597+0000 7f0e26b45700 1 -- 172.21.15.53:0/3428157022 wait complete. 2024-02-19T21:55:56.599 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:56.597+0000 7f0e26b45700 1 Processor -- start 2024-02-19T21:55:56.599 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:56.597+0000 7f0e26b45700 1 -- start start 2024-02-19T21:55:56.599 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:56.598+0000 7f0e26b45700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0e20070430 0x7f0e200fc4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:55:56.600 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:56.598+0000 7f0e26b45700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e200fa950 con 0x7f0e20070430 2024-02-19T21:55:56.600 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:56.598+0000 7f0e248e1700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0e20070430 0x7f0e200fc4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:55:56.600 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:56.598+0000 7f0e248e1700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0e20070430 0x7f0e200fc4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:47704/0 (socket says 172.21.15.53:47704) 2024-02-19T21:55:56.600 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:56.598+0000 7f0e248e1700 1 -- 172.21.15.53:0/2348645661 learned_addr learned my addr 172.21.15.53:0/2348645661 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-02-19T21:55:56.600 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:56.598+0000 7f0e248e1700 1 -- 172.21.15.53:0/2348645661 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e1400b200 con 0x7f0e20070430 2024-02-19T21:55:56.600 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:56.598+0000 7f0e248e1700 1 --2- 172.21.15.53:0/2348645661 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0e20070430 0x7f0e200fc4d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f0e1400df90 tx=0x7f0e14006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:55:56.600 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:56.598+0000 7f0e1d7fa700 1 -- 172.21.15.53:0/2348645661 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0e1401b410 con 0x7f0e20070430 2024-02-19T21:55:56.600 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:56.599+0000 7f0e26b45700 1 -- 172.21.15.53:0/2348645661 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e200faaf0 con 0x7f0e20070430 2024-02-19T21:55:56.600 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:56.599+0000 7f0e1d7fa700 1 -- 172.21.15.53:0/2348645661 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f0e1401ba00 con 0x7f0e20070430 2024-02-19T21:55:56.600 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:56.599+0000 7f0e1d7fa700 1 -- 172.21.15.53:0/2348645661 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0e1401aa30 con 0x7f0e20070430 2024-02-19T21:55:56.600 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:56.599+0000 7f0e26b45700 1 -- 172.21.15.53:0/2348645661 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e200faf80 con 0x7f0e20070430 2024-02-19T21:55:56.601 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:56.600+0000 7f0e1d7fa700 1 -- 172.21.15.53:0/2348645661 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f0e1400f070 con 0x7f0e20070430 2024-02-19T21:55:56.601 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:56.600+0000 7f0e1d7fa700 1 --2- 172.21.15.53:0/2348645661 >> [v2:172.21.15.53:6800/1836174550,v1:172.21.15.53:6801/1836174550] conn(0x7f0e0402fe10 0x7f0e040322d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:55:56.601 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:56.600+0000 7f0e1d7fa700 1 -- 172.21.15.53:0/2348645661 --> [v2:172.21.15.53:6800/1836174550,v1:172.21.15.53:6801/1836174550] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0e04032990 con 0x7f0e0402fe10 2024-02-19T21:55:56.601 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:56.600+0000 7f0e1d7fa700 1 -- 172.21.15.53:0/2348645661 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f0e14041140 con 0x7f0e20070430 2024-02-19T21:55:56.601 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:56.600+0000 7f0e1ffff700 1 -- 172.21.15.53:0/2348645661 >> [v2:172.21.15.53:6800/1836174550,v1:172.21.15.53:6801/1836174550] conn(0x7f0e0402fe10 msgr2=0x7f0e040322d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.53:6800/1836174550 2024-02-19T21:55:56.602 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:56.600+0000 7f0e1ffff700 1 --2- 172.21.15.53:0/2348645661 >> [v2:172.21.15.53:6800/1836174550,v1:172.21.15.53:6801/1836174550] conn(0x7f0e0402fe10 0x7f0e040322d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-02-19T21:55:56.801 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:56.801+0000 7f0e1ffff700 1 -- 172.21.15.53:0/2348645661 >> [v2:172.21.15.53:6800/1836174550,v1:172.21.15.53:6801/1836174550] conn(0x7f0e0402fe10 msgr2=0x7f0e040322d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.53:6800/1836174550 2024-02-19T21:55:56.801 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:56.801+0000 7f0e1ffff700 1 --2- 172.21.15.53:0/2348645661 >> [v2:172.21.15.53:6800/1836174550,v1:172.21.15.53:6801/1836174550] conn(0x7f0e0402fe10 0x7f0e040322d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2024-02-19T21:55:57.202 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:57.202+0000 7f0e1ffff700 1 -- 172.21.15.53:0/2348645661 >> [v2:172.21.15.53:6800/1836174550,v1:172.21.15.53:6801/1836174550] conn(0x7f0e0402fe10 msgr2=0x7f0e040322d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.53:6800/1836174550 2024-02-19T21:55:57.202 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:57.202+0000 7f0e1ffff700 1 --2- 172.21.15.53:0/2348645661 >> [v2:172.21.15.53:6800/1836174550,v1:172.21.15.53:6801/1836174550] conn(0x7f0e0402fe10 0x7f0e040322d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2024-02-19T21:55:58.003 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:58.003+0000 7f0e1ffff700 1 -- 172.21.15.53:0/2348645661 >> [v2:172.21.15.53:6800/1836174550,v1:172.21.15.53:6801/1836174550] conn(0x7f0e0402fe10 msgr2=0x7f0e040322d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.53:6800/1836174550 2024-02-19T21:55:58.003 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:58.003+0000 7f0e1ffff700 1 --2- 172.21.15.53:0/2348645661 >> [v2:172.21.15.53:6800/1836174550,v1:172.21.15.53:6801/1836174550] conn(0x7f0e0402fe10 0x7f0e040322d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2024-02-19T21:55:59.265 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:59.264+0000 7f0e1d7fa700 1 -- 172.21.15.53:0/2348645661 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mgrmap(e 6) v1 ==== 35970+0+0 (secure 0 0 0) 0x7f0e1401ab90 con 0x7f0e20070430 2024-02-19T21:55:59.265 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:59.265+0000 7f0e1d7fa700 1 -- 172.21.15.53:0/2348645661 >> [v2:172.21.15.53:6800/1836174550,v1:172.21.15.53:6801/1836174550] conn(0x7f0e0402fe10 msgr2=0x7f0e040322d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-19T21:55:59.265 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:55:59.265+0000 7f0e1d7fa700 1 --2- 172.21.15.53:0/2348645661 >> [v2:172.21.15.53:6800/1836174550,v1:172.21.15.53:6801/1836174550] conn(0x7f0e0402fe10 0x7f0e040322d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-19T21:56:00.267 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:00.267+0000 7f0e1d7fa700 1 -- 172.21.15.53:0/2348645661 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mgrmap(e 7) v1 ==== 36040+0+0 (secure 0 0 0) 0x7f0e1401b570 con 0x7f0e20070430 2024-02-19T21:56:00.267 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:00.267+0000 7f0e1d7fa700 1 --2- 172.21.15.53:0/2348645661 >> [v2:172.21.15.53:6800/1863695699,v1:172.21.15.53:6801/1863695699] conn(0x7f0e04033930 0x7f0e04035d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:56:00.267 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:00.267+0000 7f0e1d7fa700 1 -- 172.21.15.53:0/2348645661 --> [v2:172.21.15.53:6800/1863695699,v1:172.21.15.53:6801/1863695699] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0e04032990 con 0x7f0e04033930 2024-02-19T21:56:00.268 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:00.268+0000 7f0e1ffff700 1 --2- 172.21.15.53:0/2348645661 >> [v2:172.21.15.53:6800/1863695699,v1:172.21.15.53:6801/1863695699] conn(0x7f0e04033930 0x7f0e04035d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:56:00.268 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:00.269+0000 7f0e1ffff700 1 --2- 172.21.15.53:0/2348645661 >> [v2:172.21.15.53:6800/1863695699,v1:172.21.15.53:6801/1863695699] conn(0x7f0e04033930 0x7f0e04035d20 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f0e0c0036d0 tx=0x7f0e0c006ca0).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:56:00.268 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:00.269+0000 7f0e1d7fa700 1 -- 172.21.15.53:0/2348645661 <== mgr.14114 v2:172.21.15.53:6800/1863695699 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6438 (secure 0 0 0) 0x7f0e04032990 con 0x7f0e04033930 2024-02-19T21:56:00.275 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:00.276+0000 7f0e26b45700 1 -- 172.21.15.53:0/2348645661 --> [v2:172.21.15.53:6800/1863695699,v1:172.21.15.53:6801/1863695699] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f0e20059100 con 0x7f0e04033930 2024-02-19T21:56:00.276 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:00.276+0000 7f0e1d7fa700 1 -- 172.21.15.53:0/2348645661 <== mgr.14114 v2:172.21.15.53:6800/1863695699 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f0e20059100 con 0x7f0e04033930 2024-02-19T21:56:00.276 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: { 2024-02-19T21:56:00.276 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "mgrmap_epoch": 7, 2024-02-19T21:56:00.276 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "initialized": true 2024-02-19T21:56:00.276 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: } 2024-02-19T21:56:00.278 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:00.277+0000 7f0e26b45700 1 -- 172.21.15.53:0/2348645661 >> [v2:172.21.15.53:6800/1863695699,v1:172.21.15.53:6801/1863695699] conn(0x7f0e04033930 msgr2=0x7f0e04035d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:56:00.278 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:00.277+0000 7f0e26b45700 1 --2- 172.21.15.53:0/2348645661 >> [v2:172.21.15.53:6800/1863695699,v1:172.21.15.53:6801/1863695699] conn(0x7f0e04033930 0x7f0e04035d20 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f0e0c0036d0 tx=0x7f0e0c006ca0).stop 2024-02-19T21:56:00.278 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:00.277+0000 7f0e26b45700 1 -- 172.21.15.53:0/2348645661 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0e20070430 msgr2=0x7f0e200fc4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:56:00.278 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:00.277+0000 7f0e26b45700 1 --2- 172.21.15.53:0/2348645661 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0e20070430 0x7f0e200fc4d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f0e1400df90 tx=0x7f0e14006ee0).stop 2024-02-19T21:56:00.278 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:00.277+0000 7f0e26b45700 1 -- 172.21.15.53:0/2348645661 shutdown_connections 2024-02-19T21:56:00.278 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:00.277+0000 7f0e26b45700 1 --2- 172.21.15.53:0/2348645661 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0e20070430 0x7f0e200fc4d0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:56:00.278 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:00.277+0000 7f0e26b45700 1 --2- 172.21.15.53:0/2348645661 >> [v2:172.21.15.53:6800/1863695699,v1:172.21.15.53:6801/1863695699] conn(0x7f0e04033930 0x7f0e04035d20 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:56:00.278 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:00.277+0000 7f0e26b45700 1 --2- 172.21.15.53:0/2348645661 >> [v2:172.21.15.53:6800/1836174550,v1:172.21.15.53:6801/1836174550] conn(0x7f0e0402fe10 0x7f0e040322d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-19T21:56:00.278 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:00.277+0000 7f0e26b45700 1 -- 172.21.15.53:0/2348645661 >> 172.21.15.53:0/2348645661 conn(0x7f0e200f6530 msgr2=0x7f0e200f7f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:56:00.279 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:00.277+0000 7f0e26b45700 1 -- 172.21.15.53:0/2348645661 shutdown_connections 2024-02-19T21:56:00.279 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:00.277+0000 7f0e26b45700 1 -- 172.21.15.53:0/2348645661 wait complete. 2024-02-19T21:56:00.543 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:56:00 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T21:55:59.214107+0000 mon.smithi053 (mon.0) 29 : cluster [INF] Active manager daemon smithi053.fyhcdi restarted 2024-02-19T21:56:00.544 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:56:00 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T21:55:59.214457+0000 mon.smithi053 (mon.0) 30 : cluster [INF] Activating manager daemon smithi053.fyhcdi 2024-02-19T21:56:00.544 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:56:00 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T21:55:59.216107+0000 mon.smithi053 (mon.0) 31 : cluster [DBG] osdmap e2: 0 total, 0 up, 0 in 2024-02-19T21:56:00.544 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:56:00 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T21:55:59.266665+0000 mon.smithi053 (mon.0) 32 : cluster [DBG] mgrmap e6: smithi053.fyhcdi(active, starting, since 0.0522779s) 2024-02-19T21:56:00.544 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:56:00 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:55:59.267773+0000 mon.smithi053 (mon.0) 33 : audit [DBG] from='mgr.14114 172.21.15.53:0/1866081671' entity='mgr.smithi053.fyhcdi' cmd=[{"prefix": "mon metadata", "id": "smithi053"}]: dispatch 2024-02-19T21:56:00.544 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:56:00 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:55:59.268375+0000 mon.smithi053 (mon.0) 34 : audit [DBG] from='mgr.14114 172.21.15.53:0/1866081671' entity='mgr.smithi053.fyhcdi' cmd=[{"prefix": "mgr metadata", "who": "smithi053.fyhcdi", "id": "smithi053.fyhcdi"}]: dispatch 2024-02-19T21:56:00.544 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:56:00 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:55:59.269054+0000 mon.smithi053 (mon.0) 35 : audit [DBG] from='mgr.14114 172.21.15.53:0/1866081671' entity='mgr.smithi053.fyhcdi' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-19T21:56:00.544 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:56:00 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:55:59.269213+0000 mon.smithi053 (mon.0) 36 : audit [DBG] from='mgr.14114 172.21.15.53:0/1866081671' entity='mgr.smithi053.fyhcdi' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-19T21:56:00.544 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:56:00 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:55:59.269344+0000 mon.smithi053 (mon.0) 37 : audit [DBG] from='mgr.14114 172.21.15.53:0/1866081671' entity='mgr.smithi053.fyhcdi' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-19T21:56:00.544 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:56:00 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T21:55:59.271440+0000 mon.smithi053 (mon.0) 38 : cluster [INF] Manager daemon smithi053.fyhcdi is now available 2024-02-19T21:56:00.544 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:56:00 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:55:59.284546+0000 mon.smithi053 (mon.0) 39 : audit [INF] from='mgr.14114 172.21.15.53:0/1866081671' entity='mgr.smithi053.fyhcdi' 2024-02-19T21:56:00.545 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:56:00 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:55:59.287309+0000 mon.smithi053 (mon.0) 40 : audit [INF] from='mgr.14114 172.21.15.53:0/1866081671' entity='mgr.smithi053.fyhcdi' 2024-02-19T21:56:00.545 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:56:00 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:55:59.289733+0000 mon.smithi053 (mon.0) 41 : audit [INF] from='mgr.14114 172.21.15.53:0/1866081671' entity='mgr.smithi053.fyhcdi' 2024-02-19T21:56:00.545 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:56:00 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:55:59.382820+0000 mon.smithi053 (mon.0) 42 : audit [INF] from='mgr.14114 172.21.15.53:0/1866081671' entity='mgr.smithi053.fyhcdi' 2024-02-19T21:56:00.545 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:56:00 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:55:59.383534+0000 mon.smithi053 (mon.0) 43 : audit [DBG] from='mgr.14114 172.21.15.53:0/1866081671' entity='mgr.smithi053.fyhcdi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T21:56:00.545 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:56:00 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:55:59.385249+0000 mon.smithi053 (mon.0) 44 : audit [DBG] from='mgr.14114 172.21.15.53:0/1866081671' entity='mgr.smithi053.fyhcdi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T21:56:00.545 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:56:00 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:55:59.386809+0000 mon.smithi053 (mon.0) 45 : audit [DBG] from='mgr.14114 172.21.15.53:0/1866081671' entity='mgr.smithi053.fyhcdi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T21:56:00.545 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:56:00 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:55:59.397149+0000 mon.smithi053 (mon.0) 46 : audit [INF] from='mgr.14114 172.21.15.53:0/1866081671' entity='mgr.smithi053.fyhcdi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi053.fyhcdi/mirror_snapshot_schedule"}]: dispatch 2024-02-19T21:56:00.545 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:56:00 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:55:59.398341+0000 mon.smithi053 (mon.0) 47 : audit [INF] from='mgr.14114 172.21.15.53:0/1866081671' entity='mgr.smithi053.fyhcdi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi053.fyhcdi/trash_purge_schedule"}]: dispatch 2024-02-19T21:56:00.769 INFO:teuthology.orchestra.run.smithi053.stderr:mgr epoch 5 is available 2024-02-19T21:56:00.769 INFO:teuthology.orchestra.run.smithi053.stderr:Setting orchestrator backend to cephadm... 2024-02-19T21:56:00.769 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi053 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/867b6b52-cf71-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpoyd_8g9j:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp0zy4ixsn:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch set backend cephadm 2024-02-19T21:56:01.274 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:01.272+0000 7fda557f5700 1 Processor -- start 2024-02-19T21:56:01.274 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:01.273+0000 7fda557f5700 1 -- start start 2024-02-19T21:56:01.274 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:01.273+0000 7fda557f5700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fda500fc2e0 0x7fda500fc700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T21:56:01.275 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:01.273+0000 7fda557f5700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda500fcbf0 con 0x7fda500fc2e0 2024-02-19T21:56:01.275 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:01.273+0000 7fda4ffff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fda500fc2e0 0x7fda500fc700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:56:01.275 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:01.273+0000 7fda4ffff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fda500fc2e0 0x7fda500fc700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:47770/0 (socket says 172.21.15.53:47770) 2024-02-19T21:56:01.275 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:01.273+0000 7fda4ffff700 1 -- 172.21.15.53:0/359969590 learned_addr learned my addr 172.21.15.53:0/359969590 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-02-19T21:56:01.275 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:01.273+0000 7fda4ffff700 1 -- 172.21.15.53:0/359969590 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda500fcd30 con 0x7fda500fc2e0 2024-02-19T21:56:01.275 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:01.273+0000 7fda4ffff700 1 --2- 172.21.15.53:0/359969590 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fda500fc2e0 0x7fda500fc700 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7fda4000b5f0 tx=0x7fda40006ee0).ready entity=mon.0 client_cookie=2bd65bf35d06aac server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:56:01.275 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:01.274+0000 7fda4effd700 1 -- 172.21.15.53:0/359969590 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fda4000d600 con 0x7fda500fc2e0 2024-02-19T21:56:01.275 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:01.274+0000 7fda4effd700 1 -- 172.21.15.53:0/359969590 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fda4000dbf0 con 0x7fda500fc2e0 2024-02-19T21:56:01.275 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:01.274+0000 7fda4effd700 1 -- 172.21.15.53:0/359969590 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fda40011ce0 con 0x7fda500fc2e0 2024-02-19T21:56:01.275 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:01.274+0000 7fda557f5700 1 -- 172.21.15.53:0/359969590 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fda500fc2e0 msgr2=0x7fda500fc700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:56:01.275 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:01.274+0000 7fda557f5700 1 --2- 172.21.15.53:0/359969590 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fda500fc2e0 0x7fda500fc700 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7fda4000b5f0 tx=0x7fda40006ee0).stop 2024-02-19T21:56:01.275 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:01.275+0000 7fda557f5700 1 -- 172.21.15.53:0/359969590 shutdown_connections 2024-02-19T21:56:01.276 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:01.275+0000 7fda557f5700 1 --2- 172.21.15.53:0/359969590 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fda500fc2e0 0x7fda500fc700 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:56:01.276 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:01.275+0000 7fda557f5700 1 -- 172.21.15.53:0/359969590 >> 172.21.15.53:0/359969590 conn(0x7fda500f5b70 msgr2=0x7fda500f7f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:56:01.276 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:01.275+0000 7fda557f5700 1 -- 172.21.15.53:0/359969590 shutdown_connections 2024-02-19T21:56:01.276 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:01.275+0000 7fda557f5700 1 -- 172.21.15.53:0/359969590 wait complete. 2024-02-19T21:56:01.276 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:01.275+0000 7fda557f5700 1 Processor -- start 2024-02-19T21:56:01.276 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:01.275+0000 7fda557f5700 1 -- start start 2024-02-19T21:56:01.278 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:01.276+0000 7fda557f5700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fda500fc2e0 0x7fda5010cb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:56:01.278 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:01.276+0000 7fda557f5700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda5010d060 con 0x7fda500fc2e0 2024-02-19T21:56:01.278 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:01.276+0000 7fda4ffff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fda500fc2e0 0x7fda5010cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:56:01.279 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:01.276+0000 7fda4ffff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fda500fc2e0 0x7fda5010cb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:47786/0 (socket says 172.21.15.53:47786) 2024-02-19T21:56:01.279 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:01.276+0000 7fda4ffff700 1 -- 172.21.15.53:0/3345671921 learned_addr learned my addr 172.21.15.53:0/3345671921 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-02-19T21:56:01.279 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:01.276+0000 7fda4ffff700 1 -- 172.21.15.53:0/3345671921 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda4000b200 con 0x7fda500fc2e0 2024-02-19T21:56:01.279 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:01.276+0000 7fda4ffff700 1 --2- 172.21.15.53:0/3345671921 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fda500fc2e0 0x7fda5010cb70 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7fda40000c00 tx=0x7fda40006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:56:01.279 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:01.276+0000 7fda4cff9700 1 -- 172.21.15.53:0/3345671921 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fda4001b410 con 0x7fda500fc2e0 2024-02-19T21:56:01.279 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:01.276+0000 7fda4cff9700 1 -- 172.21.15.53:0/3345671921 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fda4001ba00 con 0x7fda500fc2e0 2024-02-19T21:56:01.279 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:01.276+0000 7fda557f5700 1 -- 172.21.15.53:0/3345671921 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda5010d260 con 0x7fda500fc2e0 2024-02-19T21:56:01.279 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:01.276+0000 7fda4cff9700 1 -- 172.21.15.53:0/3345671921 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fda4001aab0 con 0x7fda500fc2e0 2024-02-19T21:56:01.279 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:01.276+0000 7fda557f5700 1 -- 172.21.15.53:0/3345671921 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda5010d6f0 con 0x7fda500fc2e0 2024-02-19T21:56:01.279 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:01.277+0000 7fda4cff9700 1 -- 172.21.15.53:0/3345671921 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fda4000f070 con 0x7fda500fc2e0 2024-02-19T21:56:01.279 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:01.277+0000 7fda4cff9700 1 --2- 172.21.15.53:0/3345671921 >> [v2:172.21.15.53:6800/1863695699,v1:172.21.15.53:6801/1863695699] conn(0x7fda3c02fe10 0x7fda3c0322d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:56:01.279 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:01.277+0000 7fda4cff9700 1 -- 172.21.15.53:0/3345671921 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7fda40040f80 con 0x7fda500fc2e0 2024-02-19T21:56:01.280 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:01.277+0000 7fda557f5700 1 -- 172.21.15.53:0/3345671921 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda50045ad0 con 0x7fda500fc2e0 2024-02-19T21:56:01.280 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:01.278+0000 7fda4f7fe700 1 --2- 172.21.15.53:0/3345671921 >> [v2:172.21.15.53:6800/1863695699,v1:172.21.15.53:6801/1863695699] conn(0x7fda3c02fe10 0x7fda3c0322d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:56:01.280 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:01.278+0000 7fda4f7fe700 1 --2- 172.21.15.53:0/3345671921 >> [v2:172.21.15.53:6800/1863695699,v1:172.21.15.53:6801/1863695699] conn(0x7fda3c02fe10 0x7fda3c0322d0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fda440096a0 tx=0x7fda44006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:56:01.281 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:01.282+0000 7fda4cff9700 1 -- 172.21.15.53:0/3345671921 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fda40016070 con 0x7fda500fc2e0 2024-02-19T21:56:01.472 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:01.473+0000 7fda557f5700 1 -- 172.21.15.53:0/3345671921 --> [v2:172.21.15.53:6800/1863695699,v1:172.21.15.53:6801/1863695699] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7fda5010eca0 con 0x7fda3c02fe10 2024-02-19T21:56:01.477 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:01.478+0000 7fda4cff9700 1 -- 172.21.15.53:0/3345671921 <== mgr.14114 v2:172.21.15.53:6800/1863695699 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fda5010eca0 con 0x7fda3c02fe10 2024-02-19T21:56:01.480 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:01.480+0000 7fda557f5700 1 -- 172.21.15.53:0/3345671921 >> [v2:172.21.15.53:6800/1863695699,v1:172.21.15.53:6801/1863695699] conn(0x7fda3c02fe10 msgr2=0x7fda3c0322d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:56:01.480 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:01.480+0000 7fda557f5700 1 --2- 172.21.15.53:0/3345671921 >> [v2:172.21.15.53:6800/1863695699,v1:172.21.15.53:6801/1863695699] conn(0x7fda3c02fe10 0x7fda3c0322d0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fda440096a0 tx=0x7fda44006b40).stop 2024-02-19T21:56:01.480 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:01.480+0000 7fda557f5700 1 -- 172.21.15.53:0/3345671921 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fda500fc2e0 msgr2=0x7fda5010cb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:56:01.480 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:01.480+0000 7fda557f5700 1 --2- 172.21.15.53:0/3345671921 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fda500fc2e0 0x7fda5010cb70 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7fda40000c00 tx=0x7fda40006ee0).stop 2024-02-19T21:56:01.480 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:01.480+0000 7fda557f5700 1 -- 172.21.15.53:0/3345671921 shutdown_connections 2024-02-19T21:56:01.480 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:01.480+0000 7fda557f5700 1 --2- 172.21.15.53:0/3345671921 >> [v2:172.21.15.53:6800/1863695699,v1:172.21.15.53:6801/1863695699] conn(0x7fda3c02fe10 0x7fda3c0322d0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:56:01.480 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:01.480+0000 7fda557f5700 1 --2- 172.21.15.53:0/3345671921 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fda500fc2e0 0x7fda5010cb70 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:56:01.480 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:01.480+0000 7fda557f5700 1 -- 172.21.15.53:0/3345671921 >> 172.21.15.53:0/3345671921 conn(0x7fda500f5b70 msgr2=0x7fda500f74c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:56:01.481 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:01.480+0000 7fda557f5700 1 -- 172.21.15.53:0/3345671921 shutdown_connections 2024-02-19T21:56:01.481 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:01.480+0000 7fda557f5700 1 -- 172.21.15.53:0/3345671921 wait complete. 2024-02-19T21:56:01.544 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:56:01 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:56:00.269513+0000 mgr.smithi053.fyhcdi (mgr.14114) 1 : audit [DBG] from='client.14118 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-02-19T21:56:01.544 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:56:01 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T21:56:00.270270+0000 mon.smithi053 (mon.0) 48 : cluster [DBG] mgrmap e7: smithi053.fyhcdi(active, since 1.05588s) 2024-02-19T21:56:01.544 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:56:01 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:56:00.276758+0000 mgr.smithi053.fyhcdi (mgr.14114) 2 : audit [DBG] from='client.14118 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-02-19T21:56:02.042 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi053 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/867b6b52-cf71-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpoyd_8g9j:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp0zy4ixsn:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm set-user root 2024-02-19T21:56:02.543 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:56:02 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T21:56:01.270886+0000 mon.smithi053 (mon.0) 49 : cluster [DBG] mgrmap e8: smithi053.fyhcdi(active, since 2s) 2024-02-19T21:56:02.543 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:56:02 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:56:01.474314+0000 mgr.smithi053.fyhcdi (mgr.14114) 3 : audit [DBG] from='client.14124 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T21:56:02.544 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:56:02 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:56:01.478079+0000 mon.smithi053 (mon.0) 50 : audit [INF] from='mgr.14114 172.21.15.53:0/1866081671' entity='mgr.smithi053.fyhcdi' 2024-02-19T21:56:02.544 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:56:02 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:56:01.481519+0000 mon.smithi053 (mon.0) 51 : audit [DBG] from='mgr.14114 172.21.15.53:0/1866081671' entity='mgr.smithi053.fyhcdi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T21:56:02.594 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:02.592+0000 7faacd627700 1 Processor -- start 2024-02-19T21:56:02.594 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:02.592+0000 7faacd627700 1 -- start start 2024-02-19T21:56:02.594 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:02.592+0000 7faacd627700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faac8070430 0x7faac80fea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T21:56:02.594 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:02.593+0000 7faacd627700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faac80fef40 con 0x7faac8070430 2024-02-19T21:56:02.594 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:02.593+0000 7faac6ffd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faac8070430 0x7faac80fea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:56:02.594 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:02.593+0000 7faac6ffd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faac8070430 0x7faac80fea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:47796/0 (socket says 172.21.15.53:47796) 2024-02-19T21:56:02.595 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:02.593+0000 7faac6ffd700 1 -- 172.21.15.53:0/3698943924 learned_addr learned my addr 172.21.15.53:0/3698943924 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-02-19T21:56:02.595 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:02.593+0000 7faac6ffd700 1 -- 172.21.15.53:0/3698943924 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faac80ff080 con 0x7faac8070430 2024-02-19T21:56:02.595 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:02.593+0000 7faac6ffd700 1 --2- 172.21.15.53:0/3698943924 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faac8070430 0x7faac80fea50 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7faab000b5f0 tx=0x7faab0006ee0).ready entity=mon.0 client_cookie=9b74ca8333663f84 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:56:02.595 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:02.593+0000 7faac5ffb700 1 -- 172.21.15.53:0/3698943924 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faab000d600 con 0x7faac8070430 2024-02-19T21:56:02.595 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:02.593+0000 7faac5ffb700 1 -- 172.21.15.53:0/3698943924 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7faab000dbf0 con 0x7faac8070430 2024-02-19T21:56:02.595 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:02.593+0000 7faac5ffb700 1 -- 172.21.15.53:0/3698943924 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faab0011cb0 con 0x7faac8070430 2024-02-19T21:56:02.595 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:02.594+0000 7faacd627700 1 -- 172.21.15.53:0/3698943924 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faac8070430 msgr2=0x7faac80fea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:56:02.595 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:02.594+0000 7faacd627700 1 --2- 172.21.15.53:0/3698943924 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faac8070430 0x7faac80fea50 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7faab000b5f0 tx=0x7faab0006ee0).stop 2024-02-19T21:56:02.595 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:02.594+0000 7faacd627700 1 -- 172.21.15.53:0/3698943924 shutdown_connections 2024-02-19T21:56:02.595 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:02.594+0000 7faacd627700 1 --2- 172.21.15.53:0/3698943924 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faac8070430 0x7faac80fea50 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:56:02.595 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:02.594+0000 7faacd627700 1 -- 172.21.15.53:0/3698943924 >> 172.21.15.53:0/3698943924 conn(0x7faac80f6530 msgr2=0x7faac80f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:56:02.595 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:02.594+0000 7faacd627700 1 -- 172.21.15.53:0/3698943924 shutdown_connections 2024-02-19T21:56:02.595 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:02.594+0000 7faacd627700 1 -- 172.21.15.53:0/3698943924 wait complete. 2024-02-19T21:56:02.595 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:02.595+0000 7faacd627700 1 Processor -- start 2024-02-19T21:56:02.596 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:02.595+0000 7faacd627700 1 -- start start 2024-02-19T21:56:02.596 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:02.595+0000 7faacd627700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faac8070430 0x7faac80fc4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:56:02.596 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:02.595+0000 7faacd627700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faac80fa950 con 0x7faac8070430 2024-02-19T21:56:02.596 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:02.595+0000 7faac6ffd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faac8070430 0x7faac80fc4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:56:02.596 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:02.595+0000 7faac6ffd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faac8070430 0x7faac80fc4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:47804/0 (socket says 172.21.15.53:47804) 2024-02-19T21:56:02.597 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:02.595+0000 7faac6ffd700 1 -- 172.21.15.53:0/287133846 learned_addr learned my addr 172.21.15.53:0/287133846 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-02-19T21:56:02.597 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:02.595+0000 7faac6ffd700 1 -- 172.21.15.53:0/287133846 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faab000b200 con 0x7faac8070430 2024-02-19T21:56:02.598 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:02.595+0000 7faac6ffd700 1 --2- 172.21.15.53:0/287133846 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faac8070430 0x7faac80fc4d0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7faab0006800 tx=0x7faab0006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:56:02.598 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:02.596+0000 7faabf7fe700 1 -- 172.21.15.53:0/287133846 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faab001b570 con 0x7faac8070430 2024-02-19T21:56:02.598 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:02.596+0000 7faacd627700 1 -- 172.21.15.53:0/287133846 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faac80faaf0 con 0x7faac8070430 2024-02-19T21:56:02.598 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:02.596+0000 7faabf7fe700 1 -- 172.21.15.53:0/287133846 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7faab001bb60 con 0x7faac8070430 2024-02-19T21:56:02.598 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:02.596+0000 7faabf7fe700 1 -- 172.21.15.53:0/287133846 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faab001a9f0 con 0x7faac8070430 2024-02-19T21:56:02.598 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:02.596+0000 7faacd627700 1 -- 172.21.15.53:0/287133846 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faac80faf80 con 0x7faac8070430 2024-02-19T21:56:02.598 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:02.596+0000 7faabf7fe700 1 -- 172.21.15.53:0/287133846 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7faab000f070 con 0x7faac8070430 2024-02-19T21:56:02.598 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:02.597+0000 7faabf7fe700 1 --2- 172.21.15.53:0/287133846 >> [v2:172.21.15.53:6800/1863695699,v1:172.21.15.53:6801/1863695699] conn(0x7faaa802fe10 0x7faaa80322d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:56:02.598 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:02.597+0000 7faabf7fe700 1 -- 172.21.15.53:0/287133846 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7faab00422b0 con 0x7faac8070430 2024-02-19T21:56:02.598 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:02.597+0000 7faac67fc700 1 --2- 172.21.15.53:0/287133846 >> [v2:172.21.15.53:6800/1863695699,v1:172.21.15.53:6801/1863695699] conn(0x7faaa802fe10 0x7faaa80322d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:56:02.598 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:02.597+0000 7faacd627700 1 -- 172.21.15.53:0/287133846 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faac8059100 con 0x7faac8070430 2024-02-19T21:56:02.598 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:02.597+0000 7faac67fc700 1 --2- 172.21.15.53:0/287133846 >> [v2:172.21.15.53:6800/1863695699,v1:172.21.15.53:6801/1863695699] conn(0x7faaa802fe10 0x7faaa80322d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7faab80096a0 tx=0x7faab8006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:56:02.600 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:02.601+0000 7faabf7fe700 1 -- 172.21.15.53:0/287133846 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7faab0016070 con 0x7faac8070430 2024-02-19T21:56:02.760 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:02.761+0000 7faacd627700 1 -- 172.21.15.53:0/287133846 --> [v2:172.21.15.53:6800/1863695699,v1:172.21.15.53:6801/1863695699] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7faac8104af0 con 0x7faaa802fe10 2024-02-19T21:56:02.761 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:02.762+0000 7faabf7fe700 1 -- 172.21.15.53:0/287133846 <== mgr.14114 v2:172.21.15.53:6800/1863695699 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7faac8104af0 con 0x7faaa802fe10 2024-02-19T21:56:02.761 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: value unchanged 2024-02-19T21:56:02.763 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:02.764+0000 7faacd627700 1 -- 172.21.15.53:0/287133846 >> [v2:172.21.15.53:6800/1863695699,v1:172.21.15.53:6801/1863695699] conn(0x7faaa802fe10 msgr2=0x7faaa80322d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:56:02.763 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:02.764+0000 7faacd627700 1 --2- 172.21.15.53:0/287133846 >> [v2:172.21.15.53:6800/1863695699,v1:172.21.15.53:6801/1863695699] conn(0x7faaa802fe10 0x7faaa80322d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7faab80096a0 tx=0x7faab8006b40).stop 2024-02-19T21:56:02.763 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:02.764+0000 7faacd627700 1 -- 172.21.15.53:0/287133846 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faac8070430 msgr2=0x7faac80fc4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:56:02.763 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:02.764+0000 7faacd627700 1 --2- 172.21.15.53:0/287133846 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faac8070430 0x7faac80fc4d0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7faab0006800 tx=0x7faab0006ee0).stop 2024-02-19T21:56:02.764 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:02.764+0000 7faacd627700 1 -- 172.21.15.53:0/287133846 shutdown_connections 2024-02-19T21:56:02.764 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:02.764+0000 7faacd627700 1 --2- 172.21.15.53:0/287133846 >> [v2:172.21.15.53:6800/1863695699,v1:172.21.15.53:6801/1863695699] conn(0x7faaa802fe10 0x7faaa80322d0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:56:02.764 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:02.764+0000 7faacd627700 1 --2- 172.21.15.53:0/287133846 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faac8070430 0x7faac80fc4d0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:56:02.764 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:02.764+0000 7faacd627700 1 -- 172.21.15.53:0/287133846 >> 172.21.15.53:0/287133846 conn(0x7faac80f6530 msgr2=0x7faac80f7f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:56:02.764 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:02.764+0000 7faacd627700 1 -- 172.21.15.53:0/287133846 shutdown_connections 2024-02-19T21:56:02.764 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:02.764+0000 7faacd627700 1 -- 172.21.15.53:0/287133846 wait complete. 2024-02-19T21:56:03.416 INFO:teuthology.orchestra.run.smithi053.stderr:Generating ssh key... 2024-02-19T21:56:03.417 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi053 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/867b6b52-cf71-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpoyd_8g9j:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp0zy4ixsn:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm generate-key 2024-02-19T21:56:03.419 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:56:03 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:56:02.762407+0000 mgr.smithi053.fyhcdi (mgr.14114) 4 : audit [DBG] from='client.14126 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T21:56:03.967 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:03.924+0000 7fb45ebc2700 1 Processor -- start 2024-02-19T21:56:03.967 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:03.924+0000 7fb45ebc2700 1 -- start start 2024-02-19T21:56:03.967 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:03.924+0000 7fb45ebc2700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb4580703b0 0x7fb458100ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T21:56:03.967 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:03.924+0000 7fb45ebc2700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4580fc8c0 con 0x7fb4580703b0 2024-02-19T21:56:03.967 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:03.924+0000 7fb45c95e700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb4580703b0 0x7fb458100ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:56:03.967 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:03.924+0000 7fb45c95e700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb4580703b0 0x7fb458100ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:47810/0 (socket says 172.21.15.53:47810) 2024-02-19T21:56:03.967 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:03.924+0000 7fb45c95e700 1 -- 172.21.15.53:0/1529324741 learned_addr learned my addr 172.21.15.53:0/1529324741 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-02-19T21:56:03.967 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:03.924+0000 7fb45c95e700 1 -- 172.21.15.53:0/1529324741 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4581013c0 con 0x7fb4580703b0 2024-02-19T21:56:03.967 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:03.924+0000 7fb45c95e700 1 --2- 172.21.15.53:0/1529324741 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb4580703b0 0x7fb458100ed0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7fb44c00b5f0 tx=0x7fb44c006ee0).ready entity=mon.0 client_cookie=20c46abcfb65f4e5 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:56:03.967 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:03.925+0000 7fb4577fe700 1 -- 172.21.15.53:0/1529324741 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb44c00d600 con 0x7fb4580703b0 2024-02-19T21:56:03.967 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:03.925+0000 7fb4577fe700 1 -- 172.21.15.53:0/1529324741 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fb44c00dbf0 con 0x7fb4580703b0 2024-02-19T21:56:03.968 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:03.925+0000 7fb4577fe700 1 -- 172.21.15.53:0/1529324741 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb44c011cb0 con 0x7fb4580703b0 2024-02-19T21:56:03.968 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:03.925+0000 7fb45ebc2700 1 -- 172.21.15.53:0/1529324741 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb4580703b0 msgr2=0x7fb458100ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:56:03.968 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:03.925+0000 7fb45ebc2700 1 --2- 172.21.15.53:0/1529324741 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb4580703b0 0x7fb458100ed0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7fb44c00b5f0 tx=0x7fb44c006ee0).stop 2024-02-19T21:56:03.968 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:03.925+0000 7fb45ebc2700 1 -- 172.21.15.53:0/1529324741 shutdown_connections 2024-02-19T21:56:03.968 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:03.925+0000 7fb45ebc2700 1 --2- 172.21.15.53:0/1529324741 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb4580703b0 0x7fb458100ed0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:56:03.968 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:03.925+0000 7fb45ebc2700 1 -- 172.21.15.53:0/1529324741 >> 172.21.15.53:0/1529324741 conn(0x7fb4580f6530 msgr2=0x7fb4580f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:56:03.968 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:03.925+0000 7fb45ebc2700 1 -- 172.21.15.53:0/1529324741 shutdown_connections 2024-02-19T21:56:03.968 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:03.925+0000 7fb45ebc2700 1 -- 172.21.15.53:0/1529324741 wait complete. 2024-02-19T21:56:03.968 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:03.926+0000 7fb45ebc2700 1 Processor -- start 2024-02-19T21:56:03.968 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:03.926+0000 7fb45ebc2700 1 -- start start 2024-02-19T21:56:03.968 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:03.926+0000 7fb45ebc2700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb4580703b0 0x7fb45810cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:56:03.968 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:03.926+0000 7fb45ebc2700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb45810d270 con 0x7fb4580703b0 2024-02-19T21:56:03.968 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:03.926+0000 7fb45c95e700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb4580703b0 0x7fb45810cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:56:03.968 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:03.926+0000 7fb45c95e700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb4580703b0 0x7fb45810cd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:47814/0 (socket says 172.21.15.53:47814) 2024-02-19T21:56:03.969 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:03.926+0000 7fb45c95e700 1 -- 172.21.15.53:0/2235485019 learned_addr learned my addr 172.21.15.53:0/2235485019 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-02-19T21:56:03.969 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:03.926+0000 7fb45c95e700 1 -- 172.21.15.53:0/2235485019 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb44c00b200 con 0x7fb4580703b0 2024-02-19T21:56:03.969 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:03.927+0000 7fb45c95e700 1 --2- 172.21.15.53:0/2235485019 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb4580703b0 0x7fb45810cd80 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7fb44c006800 tx=0x7fb44c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:56:03.969 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:03.927+0000 7fb4557fa700 1 -- 172.21.15.53:0/2235485019 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb44c01b570 con 0x7fb4580703b0 2024-02-19T21:56:03.969 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:03.927+0000 7fb4557fa700 1 -- 172.21.15.53:0/2235485019 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fb44c01bb60 con 0x7fb4580703b0 2024-02-19T21:56:03.969 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:03.927+0000 7fb4557fa700 1 -- 172.21.15.53:0/2235485019 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb44c01a9f0 con 0x7fb4580703b0 2024-02-19T21:56:03.969 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:03.927+0000 7fb45ebc2700 1 -- 172.21.15.53:0/2235485019 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb45810d470 con 0x7fb4580703b0 2024-02-19T21:56:03.969 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:03.927+0000 7fb45ebc2700 1 -- 172.21.15.53:0/2235485019 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb45810e080 con 0x7fb4580703b0 2024-02-19T21:56:03.969 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:03.927+0000 7fb4557fa700 1 -- 172.21.15.53:0/2235485019 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fb44c00f070 con 0x7fb4580703b0 2024-02-19T21:56:03.969 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:03.928+0000 7fb4557fa700 1 --2- 172.21.15.53:0/2235485019 >> [v2:172.21.15.53:6800/1863695699,v1:172.21.15.53:6801/1863695699] conn(0x7fb43c02fe70 0x7fb43c032330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:56:03.969 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:03.928+0000 7fb4557fa700 1 -- 172.21.15.53:0/2235485019 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7fb44c041370 con 0x7fb4580703b0 2024-02-19T21:56:03.969 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:03.928+0000 7fb45ebc2700 1 -- 172.21.15.53:0/2235485019 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb458059100 con 0x7fb4580703b0 2024-02-19T21:56:03.969 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:03.928+0000 7fb457fff700 1 --2- 172.21.15.53:0/2235485019 >> [v2:172.21.15.53:6800/1863695699,v1:172.21.15.53:6801/1863695699] conn(0x7fb43c02fe70 0x7fb43c032330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:56:03.969 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:03.928+0000 7fb457fff700 1 --2- 172.21.15.53:0/2235485019 >> [v2:172.21.15.53:6800/1863695699,v1:172.21.15.53:6801/1863695699] conn(0x7fb43c02fe70 0x7fb43c032330 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7fb4440096a0 tx=0x7fb444006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:56:03.969 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:03.932+0000 7fb4557fa700 1 -- 172.21.15.53:0/2235485019 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb44c016070 con 0x7fb4580703b0 2024-02-19T21:56:04.090 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:04.091+0000 7fb45ebc2700 1 -- 172.21.15.53:0/2235485019 --> [v2:172.21.15.53:6800/1863695699,v1:172.21.15.53:6801/1863695699] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7fb458102180 con 0x7fb43c02fe70 2024-02-19T21:56:04.201 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:04.201+0000 7fb4557fa700 1 -- 172.21.15.53:0/2235485019 <== mgr.14114 v2:172.21.15.53:6800/1863695699 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fb458102180 con 0x7fb43c02fe70 2024-02-19T21:56:04.203 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:04.203+0000 7fb45ebc2700 1 -- 172.21.15.53:0/2235485019 >> [v2:172.21.15.53:6800/1863695699,v1:172.21.15.53:6801/1863695699] conn(0x7fb43c02fe70 msgr2=0x7fb43c032330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:56:04.203 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:04.203+0000 7fb45ebc2700 1 --2- 172.21.15.53:0/2235485019 >> [v2:172.21.15.53:6800/1863695699,v1:172.21.15.53:6801/1863695699] conn(0x7fb43c02fe70 0x7fb43c032330 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7fb4440096a0 tx=0x7fb444006b40).stop 2024-02-19T21:56:04.203 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:04.203+0000 7fb45ebc2700 1 -- 172.21.15.53:0/2235485019 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb4580703b0 msgr2=0x7fb45810cd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:56:04.203 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:04.203+0000 7fb45ebc2700 1 --2- 172.21.15.53:0/2235485019 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb4580703b0 0x7fb45810cd80 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7fb44c006800 tx=0x7fb44c006ee0).stop 2024-02-19T21:56:04.203 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:04.203+0000 7fb45ebc2700 1 -- 172.21.15.53:0/2235485019 shutdown_connections 2024-02-19T21:56:04.203 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:04.203+0000 7fb45ebc2700 1 --2- 172.21.15.53:0/2235485019 >> [v2:172.21.15.53:6800/1863695699,v1:172.21.15.53:6801/1863695699] conn(0x7fb43c02fe70 0x7fb43c032330 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:56:04.203 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:04.203+0000 7fb45ebc2700 1 --2- 172.21.15.53:0/2235485019 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb4580703b0 0x7fb45810cd80 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:56:04.203 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:04.203+0000 7fb45ebc2700 1 -- 172.21.15.53:0/2235485019 >> 172.21.15.53:0/2235485019 conn(0x7fb4580f6530 msgr2=0x7fb4580f7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:56:04.203 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:04.203+0000 7fb45ebc2700 1 -- 172.21.15.53:0/2235485019 shutdown_connections 2024-02-19T21:56:04.204 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:04.203+0000 7fb45ebc2700 1 -- 172.21.15.53:0/2235485019 wait complete. 2024-02-19T21:56:04.797 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi053 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/867b6b52-cf71-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpoyd_8g9j:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp0zy4ixsn:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm get-pub-key 2024-02-19T21:56:05.325 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:05.322+0000 7fb4c7c57700 1 Processor -- start 2024-02-19T21:56:05.326 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:05.323+0000 7fb4c7c57700 1 -- start start 2024-02-19T21:56:05.326 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:05.323+0000 7fb4c7c57700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb4c00703b0 0x7fb4c0100ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T21:56:05.326 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:05.323+0000 7fb4c7c57700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4c00fc8c0 con 0x7fb4c00703b0 2024-02-19T21:56:05.326 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:05.323+0000 7fb4c59f3700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb4c00703b0 0x7fb4c0100ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:56:05.326 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:05.323+0000 7fb4c59f3700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb4c00703b0 0x7fb4c0100ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:47826/0 (socket says 172.21.15.53:47826) 2024-02-19T21:56:05.326 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:05.323+0000 7fb4c59f3700 1 -- 172.21.15.53:0/2093900321 learned_addr learned my addr 172.21.15.53:0/2093900321 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-02-19T21:56:05.326 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:05.323+0000 7fb4c59f3700 1 -- 172.21.15.53:0/2093900321 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4c01013c0 con 0x7fb4c00703b0 2024-02-19T21:56:05.326 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:05.323+0000 7fb4c59f3700 1 --2- 172.21.15.53:0/2093900321 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb4c00703b0 0x7fb4c0100ed0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7fb4b400b5f0 tx=0x7fb4b4006ee0).ready entity=mon.0 client_cookie=b1b7fa5204491530 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:56:05.326 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:05.324+0000 7fb4c49f1700 1 -- 172.21.15.53:0/2093900321 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb4b400d600 con 0x7fb4c00703b0 2024-02-19T21:56:05.326 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:05.324+0000 7fb4c49f1700 1 -- 172.21.15.53:0/2093900321 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fb4b400dbf0 con 0x7fb4c00703b0 2024-02-19T21:56:05.326 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:05.324+0000 7fb4c49f1700 1 -- 172.21.15.53:0/2093900321 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb4b4011cb0 con 0x7fb4c00703b0 2024-02-19T21:56:05.326 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:05.324+0000 7fb4c7c57700 1 -- 172.21.15.53:0/2093900321 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb4c00703b0 msgr2=0x7fb4c0100ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:56:05.326 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:05.324+0000 7fb4c7c57700 1 --2- 172.21.15.53:0/2093900321 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb4c00703b0 0x7fb4c0100ed0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7fb4b400b5f0 tx=0x7fb4b4006ee0).stop 2024-02-19T21:56:05.327 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:05.324+0000 7fb4c7c57700 1 -- 172.21.15.53:0/2093900321 shutdown_connections 2024-02-19T21:56:05.327 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:05.324+0000 7fb4c7c57700 1 --2- 172.21.15.53:0/2093900321 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb4c00703b0 0x7fb4c0100ed0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:56:05.327 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:05.324+0000 7fb4c7c57700 1 -- 172.21.15.53:0/2093900321 >> 172.21.15.53:0/2093900321 conn(0x7fb4c00f6530 msgr2=0x7fb4c00f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:56:05.327 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:05.324+0000 7fb4c7c57700 1 -- 172.21.15.53:0/2093900321 shutdown_connections 2024-02-19T21:56:05.327 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:05.324+0000 7fb4c7c57700 1 -- 172.21.15.53:0/2093900321 wait complete. 2024-02-19T21:56:05.327 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:05.325+0000 7fb4c7c57700 1 Processor -- start 2024-02-19T21:56:05.327 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:05.325+0000 7fb4c7c57700 1 -- start start 2024-02-19T21:56:05.327 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:05.325+0000 7fb4c7c57700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb4c00703b0 0x7fb4c010cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:56:05.327 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:05.325+0000 7fb4c7c57700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4c010d2c0 con 0x7fb4c00703b0 2024-02-19T21:56:05.327 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:05.325+0000 7fb4c59f3700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb4c00703b0 0x7fb4c010cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:56:05.327 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:05.325+0000 7fb4c59f3700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb4c00703b0 0x7fb4c010cdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:47838/0 (socket says 172.21.15.53:47838) 2024-02-19T21:56:05.327 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:05.325+0000 7fb4c59f3700 1 -- 172.21.15.53:0/794761944 learned_addr learned my addr 172.21.15.53:0/794761944 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-02-19T21:56:05.327 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:05.326+0000 7fb4c59f3700 1 -- 172.21.15.53:0/794761944 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4b400b200 con 0x7fb4c00703b0 2024-02-19T21:56:05.329 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:05.326+0000 7fb4c59f3700 1 --2- 172.21.15.53:0/794761944 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb4c00703b0 0x7fb4c010cdd0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7fb4b400df90 tx=0x7fb4b4006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:56:05.329 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:05.326+0000 7fb4b27fc700 1 -- 172.21.15.53:0/794761944 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb4b401b410 con 0x7fb4c00703b0 2024-02-19T21:56:05.329 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:05.326+0000 7fb4b27fc700 1 -- 172.21.15.53:0/794761944 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fb4b401ba00 con 0x7fb4c00703b0 2024-02-19T21:56:05.329 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:05.326+0000 7fb4c7c57700 1 -- 172.21.15.53:0/794761944 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4c010d4c0 con 0x7fb4c00703b0 2024-02-19T21:56:05.329 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:05.326+0000 7fb4b27fc700 1 -- 172.21.15.53:0/794761944 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb4b401aa30 con 0x7fb4c00703b0 2024-02-19T21:56:05.329 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:05.326+0000 7fb4c7c57700 1 -- 172.21.15.53:0/794761944 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4c010e100 con 0x7fb4c00703b0 2024-02-19T21:56:05.329 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:05.327+0000 7fb4b27fc700 1 -- 172.21.15.53:0/794761944 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fb4b400f070 con 0x7fb4c00703b0 2024-02-19T21:56:05.329 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:05.327+0000 7fb4b27fc700 1 --2- 172.21.15.53:0/794761944 >> [v2:172.21.15.53:6800/1863695699,v1:172.21.15.53:6801/1863695699] conn(0x7fb4a802fe70 0x7fb4a8032330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:56:05.329 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:05.327+0000 7fb4b27fc700 1 -- 172.21.15.53:0/794761944 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7fb4b4041160 con 0x7fb4c00703b0 2024-02-19T21:56:05.329 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:05.327+0000 7fb4c51f2700 1 --2- 172.21.15.53:0/794761944 >> [v2:172.21.15.53:6800/1863695699,v1:172.21.15.53:6801/1863695699] conn(0x7fb4a802fe70 0x7fb4a8032330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:56:05.329 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:05.327+0000 7fb4c7c57700 1 -- 172.21.15.53:0/794761944 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4c010d950 con 0x7fb4c00703b0 2024-02-19T21:56:05.329 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:05.327+0000 7fb4c51f2700 1 --2- 172.21.15.53:0/794761944 >> [v2:172.21.15.53:6800/1863695699,v1:172.21.15.53:6801/1863695699] conn(0x7fb4a802fe70 0x7fb4a8032330 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7fb4bc0096a0 tx=0x7fb4bc006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:56:05.331 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:05.332+0000 7fb4b27fc700 1 -- 172.21.15.53:0/794761944 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb4b4016070 con 0x7fb4c00703b0 2024-02-19T21:56:05.491 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:05.491+0000 7fb4c7c57700 1 -- 172.21.15.53:0/794761944 --> [v2:172.21.15.53:6800/1863695699,v1:172.21.15.53:6801/1863695699] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7fb4c01021c0 con 0x7fb4a802fe70 2024-02-19T21:56:05.492 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:05.492+0000 7fb4b27fc700 1 -- 172.21.15.53:0/794761944 <== mgr.14114 v2:172.21.15.53:6800/1863695699 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7fb4c01021c0 con 0x7fb4a802fe70 2024-02-19T21:56:05.492 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCkTewxITfw3HAfoqPo+JypGfCGvIqbS/6zUjdSdPTXQmlHWITF5B654zSBv0upEXbeRLCzE8BMFNKrNb2A3vSNUhtusIyw8WuSnCruenuVcHSTNVwyWN8+b02qhsWOSaa+0oIbIxNUQlLsyhpYr003e2aIVLkJMrgeo9xwM2H2A4jHGzxMTKepB+EbQBjrWxC+pgWGiHYUdG0BPxX2mpSfPolHnlzaVRMsfBCMuHM1Ou4uIr/to/BFZGdG/b0E0n7p1T6z46WJ2kNIYJkaWf4+RLLNjm7gNwtEQHcPi0J5axeLu0gghi3lj62K8DeB2utogf2fgc8PSK4xvcsqJkkpXUcAdy7iCIDNSXWkscR2b+cnsQURj1uuIpp0JwNTuhhZLeCQcqK4uD9/9EMqy6hcqmav3z1UXSMnaWZNY9E6AzU6NcF5j2x4+lhQ+jPppxn8c4A1XJ4HYhcnXu3elEnqFVG4mxLFYVb/uqO4JhlYWd54b12/8wesBUdeslCqaOE= ceph-867b6b52-cf71-11ee-95bb-87774f69a715 2024-02-19T21:56:05.494 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:05.494+0000 7fb4c7c57700 1 -- 172.21.15.53:0/794761944 >> [v2:172.21.15.53:6800/1863695699,v1:172.21.15.53:6801/1863695699] conn(0x7fb4a802fe70 msgr2=0x7fb4a8032330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:56:05.495 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:05.494+0000 7fb4c7c57700 1 --2- 172.21.15.53:0/794761944 >> [v2:172.21.15.53:6800/1863695699,v1:172.21.15.53:6801/1863695699] conn(0x7fb4a802fe70 0x7fb4a8032330 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7fb4bc0096a0 tx=0x7fb4bc006b40).stop 2024-02-19T21:56:05.495 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:05.494+0000 7fb4c7c57700 1 -- 172.21.15.53:0/794761944 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb4c00703b0 msgr2=0x7fb4c010cdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:56:05.495 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:05.494+0000 7fb4c7c57700 1 --2- 172.21.15.53:0/794761944 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb4c00703b0 0x7fb4c010cdd0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7fb4b400df90 tx=0x7fb4b4006ee0).stop 2024-02-19T21:56:05.495 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:05.494+0000 7fb4c7c57700 1 -- 172.21.15.53:0/794761944 shutdown_connections 2024-02-19T21:56:05.495 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:05.494+0000 7fb4c7c57700 1 --2- 172.21.15.53:0/794761944 >> [v2:172.21.15.53:6800/1863695699,v1:172.21.15.53:6801/1863695699] conn(0x7fb4a802fe70 0x7fb4a8032330 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:56:05.495 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:05.494+0000 7fb4c7c57700 1 --2- 172.21.15.53:0/794761944 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb4c00703b0 0x7fb4c010cdd0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:56:05.495 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:05.494+0000 7fb4c7c57700 1 -- 172.21.15.53:0/794761944 >> 172.21.15.53:0/794761944 conn(0x7fb4c00f6530 msgr2=0x7fb4c00f7fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:56:05.495 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:05.494+0000 7fb4c7c57700 1 -- 172.21.15.53:0/794761944 shutdown_connections 2024-02-19T21:56:05.495 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:05.494+0000 7fb4c7c57700 1 -- 172.21.15.53:0/794761944 wait complete. 2024-02-19T21:56:05.514 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:56:05 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cephadm 2024-02-19T21:56:04.092434+0000 mgr.smithi053.fyhcdi (mgr.14114) 6 : cephadm [INF] Generating ssh key... 2024-02-19T21:56:05.514 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:56:05 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:56:04.199587+0000 mon.smithi053 (mon.0) 52 : audit [INF] from='mgr.14114 172.21.15.53:0/1866081671' entity='mgr.smithi053.fyhcdi' 2024-02-19T21:56:05.514 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:56:05 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:56:04.201193+0000 mon.smithi053 (mon.0) 53 : audit [INF] from='mgr.14114 172.21.15.53:0/1866081671' entity='mgr.smithi053.fyhcdi' 2024-02-19T21:56:06.256 INFO:teuthology.orchestra.run.smithi053.stderr:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-02-19T21:56:06.256 INFO:teuthology.orchestra.run.smithi053.stderr:Adding key to root@localhost authorized_keys... 2024-02-19T21:56:06.257 INFO:teuthology.orchestra.run.smithi053.stderr:Adding host smithi053... 2024-02-19T21:56:06.257 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi053 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/867b6b52-cf71-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpoyd_8g9j:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp0zy4ixsn:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch host add smithi053 172.21.15.53 2024-02-19T21:56:06.259 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:56:06 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:56:05.493378+0000 mgr.smithi053.fyhcdi (mgr.14114) 7 : audit [DBG] from='client.14130 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T21:56:06.802 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:06.800+0000 7f4b806b5700 1 Processor -- start 2024-02-19T21:56:06.802 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:06.800+0000 7f4b806b5700 1 -- start start 2024-02-19T21:56:06.802 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:06.800+0000 7f4b806b5700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4b78070370 0x7f4b780feb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T21:56:06.802 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:06.800+0000 7f4b806b5700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b78070790 con 0x7f4b78070370 2024-02-19T21:56:06.802 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:06.800+0000 7f4b7e451700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4b78070370 0x7f4b780feb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:56:06.802 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:06.801+0000 7f4b7e451700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4b78070370 0x7f4b780feb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:36010/0 (socket says 172.21.15.53:36010) 2024-02-19T21:56:06.802 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:06.801+0000 7f4b7e451700 1 -- 172.21.15.53:0/3250019615 learned_addr learned my addr 172.21.15.53:0/3250019615 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-02-19T21:56:06.802 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:06.801+0000 7f4b7e451700 1 -- 172.21.15.53:0/3250019615 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b780ff000 con 0x7f4b78070370 2024-02-19T21:56:06.802 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:06.801+0000 7f4b7e451700 1 --2- 172.21.15.53:0/3250019615 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4b78070370 0x7f4b780feb10 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f4b6800b5f0 tx=0x7f4b68006ee0).ready entity=mon.0 client_cookie=7a56653912c73335 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:56:06.802 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:06.801+0000 7f4b7d44f700 1 -- 172.21.15.53:0/3250019615 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4b6800d600 con 0x7f4b78070370 2024-02-19T21:56:06.802 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:06.801+0000 7f4b7d44f700 1 -- 172.21.15.53:0/3250019615 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f4b6800dbf0 con 0x7f4b78070370 2024-02-19T21:56:06.803 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:06.801+0000 7f4b7d44f700 1 -- 172.21.15.53:0/3250019615 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4b68011cb0 con 0x7f4b78070370 2024-02-19T21:56:06.803 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:06.802+0000 7f4b806b5700 1 -- 172.21.15.53:0/3250019615 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4b78070370 msgr2=0x7f4b780feb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:56:06.803 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:06.802+0000 7f4b806b5700 1 --2- 172.21.15.53:0/3250019615 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4b78070370 0x7f4b780feb10 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f4b6800b5f0 tx=0x7f4b68006ee0).stop 2024-02-19T21:56:06.805 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:06.802+0000 7f4b806b5700 1 -- 172.21.15.53:0/3250019615 shutdown_connections 2024-02-19T21:56:06.806 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:06.802+0000 7f4b806b5700 1 --2- 172.21.15.53:0/3250019615 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4b78070370 0x7f4b780feb10 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:56:06.806 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:06.802+0000 7f4b806b5700 1 -- 172.21.15.53:0/3250019615 >> 172.21.15.53:0/3250019615 conn(0x7f4b780f6530 msgr2=0x7f4b780f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:56:06.806 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:06.802+0000 7f4b806b5700 1 -- 172.21.15.53:0/3250019615 shutdown_connections 2024-02-19T21:56:06.806 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:06.802+0000 7f4b806b5700 1 -- 172.21.15.53:0/3250019615 wait complete. 2024-02-19T21:56:06.806 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:06.803+0000 7f4b806b5700 1 Processor -- start 2024-02-19T21:56:06.806 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:06.803+0000 7f4b806b5700 1 -- start start 2024-02-19T21:56:06.806 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:06.803+0000 7f4b806b5700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4b78070370 0x7f4b7810abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:56:06.806 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:06.803+0000 7f4b806b5700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b7810b0a0 con 0x7f4b78070370 2024-02-19T21:56:06.806 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:06.803+0000 7f4b7e451700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4b78070370 0x7f4b7810abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:56:06.806 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:06.803+0000 7f4b7e451700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4b78070370 0x7f4b7810abb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:36014/0 (socket says 172.21.15.53:36014) 2024-02-19T21:56:06.806 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:06.803+0000 7f4b7e451700 1 -- 172.21.15.53:0/2339771793 learned_addr learned my addr 172.21.15.53:0/2339771793 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-02-19T21:56:06.806 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:06.804+0000 7f4b7e451700 1 -- 172.21.15.53:0/2339771793 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b6800b200 con 0x7f4b78070370 2024-02-19T21:56:06.806 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:06.804+0000 7f4b7e451700 1 --2- 172.21.15.53:0/2339771793 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4b78070370 0x7f4b7810abb0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f4b6800df90 tx=0x7f4b68006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:56:06.806 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:06.804+0000 7f4b66ffd700 1 -- 172.21.15.53:0/2339771793 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4b6801b410 con 0x7f4b78070370 2024-02-19T21:56:06.807 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:06.804+0000 7f4b66ffd700 1 -- 172.21.15.53:0/2339771793 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f4b6801ba00 con 0x7f4b78070370 2024-02-19T21:56:06.807 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:06.804+0000 7f4b806b5700 1 -- 172.21.15.53:0/2339771793 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b7810b2a0 con 0x7f4b78070370 2024-02-19T21:56:06.807 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:06.804+0000 7f4b66ffd700 1 -- 172.21.15.53:0/2339771793 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4b6801aa30 con 0x7f4b78070370 2024-02-19T21:56:06.807 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:06.804+0000 7f4b806b5700 1 -- 172.21.15.53:0/2339771793 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b7810b730 con 0x7f4b78070370 2024-02-19T21:56:06.807 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:06.805+0000 7f4b66ffd700 1 -- 172.21.15.53:0/2339771793 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f4b6800f070 con 0x7f4b78070370 2024-02-19T21:56:06.807 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:06.805+0000 7f4b66ffd700 1 --2- 172.21.15.53:0/2339771793 >> [v2:172.21.15.53:6800/1863695699,v1:172.21.15.53:6801/1863695699] conn(0x7f4b6002fe70 0x7f4b60032330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:56:06.807 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:06.805+0000 7f4b66ffd700 1 -- 172.21.15.53:0/2339771793 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f4b68041180 con 0x7f4b78070370 2024-02-19T21:56:06.807 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:06.805+0000 7f4b7dc50700 1 --2- 172.21.15.53:0/2339771793 >> [v2:172.21.15.53:6800/1863695699,v1:172.21.15.53:6801/1863695699] conn(0x7f4b6002fe70 0x7f4b60032330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:56:06.807 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:06.805+0000 7f4b7dc50700 1 --2- 172.21.15.53:0/2339771793 >> [v2:172.21.15.53:6800/1863695699,v1:172.21.15.53:6801/1863695699] conn(0x7f4b6002fe70 0x7f4b60032330 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f4b740096a0 tx=0x7f4b74006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:56:06.807 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:06.805+0000 7f4b806b5700 1 -- 172.21.15.53:0/2339771793 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b78045ad0 con 0x7f4b78070370 2024-02-19T21:56:06.810 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:06.810+0000 7f4b66ffd700 1 -- 172.21.15.53:0/2339771793 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4b68016070 con 0x7f4b78070370 2024-02-19T21:56:06.990 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:06.990+0000 7f4b806b5700 1 -- 172.21.15.53:0/2339771793 --> [v2:172.21.15.53:6800/1863695699,v1:172.21.15.53:6801/1863695699] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi053", "addr": "172.21.15.53", "target": ["mon-mgr", ""]}) v1 -- 0x7f4b78104bf0 con 0x7f4b6002fe70 2024-02-19T21:56:07.293 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:56:07 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:56:06.992357+0000 mgr.smithi053.fyhcdi (mgr.14114) 8 : audit [DBG] from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi053", "addr": "172.21.15.53", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T21:56:07.597 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: Added host 'smithi053' 2024-02-19T21:56:07.597 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:07.595+0000 7f4b66ffd700 1 -- 172.21.15.53:0/2339771793 <== mgr.14114 v2:172.21.15.53:6800/1863695699 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+23 (secure 0 0 0) 0x7f4b78104bf0 con 0x7f4b6002fe70 2024-02-19T21:56:07.597 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:07.597+0000 7f4b806b5700 1 -- 172.21.15.53:0/2339771793 >> [v2:172.21.15.53:6800/1863695699,v1:172.21.15.53:6801/1863695699] conn(0x7f4b6002fe70 msgr2=0x7f4b60032330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:56:07.598 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:07.597+0000 7f4b806b5700 1 --2- 172.21.15.53:0/2339771793 >> [v2:172.21.15.53:6800/1863695699,v1:172.21.15.53:6801/1863695699] conn(0x7f4b6002fe70 0x7f4b60032330 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f4b740096a0 tx=0x7f4b74006b40).stop 2024-02-19T21:56:07.598 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:07.598+0000 7f4b806b5700 1 -- 172.21.15.53:0/2339771793 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4b78070370 msgr2=0x7f4b7810abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:56:07.598 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:07.598+0000 7f4b806b5700 1 --2- 172.21.15.53:0/2339771793 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4b78070370 0x7f4b7810abb0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f4b6800df90 tx=0x7f4b68006ee0).stop 2024-02-19T21:56:07.599 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:07.598+0000 7f4b806b5700 1 -- 172.21.15.53:0/2339771793 shutdown_connections 2024-02-19T21:56:07.599 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:07.598+0000 7f4b806b5700 1 --2- 172.21.15.53:0/2339771793 >> [v2:172.21.15.53:6800/1863695699,v1:172.21.15.53:6801/1863695699] conn(0x7f4b6002fe70 0x7f4b60032330 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:56:07.599 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:07.598+0000 7f4b806b5700 1 --2- 172.21.15.53:0/2339771793 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4b78070370 0x7f4b7810abb0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:56:07.599 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:07.598+0000 7f4b806b5700 1 -- 172.21.15.53:0/2339771793 >> 172.21.15.53:0/2339771793 conn(0x7f4b780f6530 msgr2=0x7f4b780f7fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:56:07.599 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:07.598+0000 7f4b806b5700 1 -- 172.21.15.53:0/2339771793 shutdown_connections 2024-02-19T21:56:07.599 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:07.598+0000 7f4b806b5700 1 -- 172.21.15.53:0/2339771793 wait complete. 2024-02-19T21:56:08.212 INFO:teuthology.orchestra.run.smithi053.stderr:Deploying mon service with default placement... 2024-02-19T21:56:08.213 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi053 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/867b6b52-cf71-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpoyd_8g9j:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp0zy4ixsn:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply mon 2024-02-19T21:56:08.826 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:08.825+0000 7fb692fc7700 1 Processor -- start 2024-02-19T21:56:08.830 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:08.825+0000 7fb692fc7700 1 -- start start 2024-02-19T21:56:08.830 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:08.825+0000 7fb692fc7700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb68c0704a0 0x7fb68c0feb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T21:56:08.830 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:08.826+0000 7fb692fc7700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb68c0feff0 con 0x7fb68c0704a0 2024-02-19T21:56:08.830 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:08.826+0000 7fb690d63700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb68c0704a0 0x7fb68c0feb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:56:08.830 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:08.826+0000 7fb690d63700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb68c0704a0 0x7fb68c0feb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:36018/0 (socket says 172.21.15.53:36018) 2024-02-19T21:56:08.830 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:08.826+0000 7fb690d63700 1 -- 172.21.15.53:0/4203713725 learned_addr learned my addr 172.21.15.53:0/4203713725 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-02-19T21:56:08.830 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:08.826+0000 7fb690d63700 1 -- 172.21.15.53:0/4203713725 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb68c0ff130 con 0x7fb68c0704a0 2024-02-19T21:56:08.830 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:08.826+0000 7fb690d63700 1 --2- 172.21.15.53:0/4203713725 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb68c0704a0 0x7fb68c0feb00 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7fb68000b5f0 tx=0x7fb680006ee0).ready entity=mon.0 client_cookie=9b78898a9f3c4e33 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:56:08.830 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:08.826+0000 7fb68b7fe700 1 -- 172.21.15.53:0/4203713725 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb68000d600 con 0x7fb68c0704a0 2024-02-19T21:56:08.831 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:08.826+0000 7fb68b7fe700 1 -- 172.21.15.53:0/4203713725 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fb68000dbf0 con 0x7fb68c0704a0 2024-02-19T21:56:08.831 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:08.826+0000 7fb68b7fe700 1 -- 172.21.15.53:0/4203713725 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb680011cb0 con 0x7fb68c0704a0 2024-02-19T21:56:08.831 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:08.827+0000 7fb692fc7700 1 -- 172.21.15.53:0/4203713725 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb68c0704a0 msgr2=0x7fb68c0feb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:56:08.831 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:08.827+0000 7fb692fc7700 1 --2- 172.21.15.53:0/4203713725 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb68c0704a0 0x7fb68c0feb00 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7fb68000b5f0 tx=0x7fb680006ee0).stop 2024-02-19T21:56:08.831 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:08.827+0000 7fb692fc7700 1 -- 172.21.15.53:0/4203713725 shutdown_connections 2024-02-19T21:56:08.831 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:08.827+0000 7fb692fc7700 1 --2- 172.21.15.53:0/4203713725 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb68c0704a0 0x7fb68c0feb00 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:56:08.831 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:08.827+0000 7fb692fc7700 1 -- 172.21.15.53:0/4203713725 >> 172.21.15.53:0/4203713725 conn(0x7fb68c0f65c0 msgr2=0x7fb68c0f89e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:56:08.831 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:08.827+0000 7fb692fc7700 1 -- 172.21.15.53:0/4203713725 shutdown_connections 2024-02-19T21:56:08.831 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:08.827+0000 7fb692fc7700 1 -- 172.21.15.53:0/4203713725 wait complete. 2024-02-19T21:56:08.831 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:08.827+0000 7fb692fc7700 1 Processor -- start 2024-02-19T21:56:08.831 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:08.828+0000 7fb692fc7700 1 -- start start 2024-02-19T21:56:08.831 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:08.828+0000 7fb692fc7700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb68c0704a0 0x7fb68c10cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:56:08.831 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:08.828+0000 7fb692fc7700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb68c10d2b0 con 0x7fb68c0704a0 2024-02-19T21:56:08.831 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:08.828+0000 7fb690d63700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb68c0704a0 0x7fb68c10cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:56:08.832 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:08.828+0000 7fb690d63700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb68c0704a0 0x7fb68c10cdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:36032/0 (socket says 172.21.15.53:36032) 2024-02-19T21:56:08.832 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:08.828+0000 7fb690d63700 1 -- 172.21.15.53:0/703284440 learned_addr learned my addr 172.21.15.53:0/703284440 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-02-19T21:56:08.832 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:08.828+0000 7fb690d63700 1 -- 172.21.15.53:0/703284440 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb68000b200 con 0x7fb68c0704a0 2024-02-19T21:56:08.832 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:08.828+0000 7fb690d63700 1 --2- 172.21.15.53:0/703284440 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb68c0704a0 0x7fb68c10cdc0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7fb680006800 tx=0x7fb680006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:56:08.832 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:08.829+0000 7fb6897fa700 1 -- 172.21.15.53:0/703284440 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb68001b570 con 0x7fb68c0704a0 2024-02-19T21:56:08.832 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:08.829+0000 7fb692fc7700 1 -- 172.21.15.53:0/703284440 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb68c10d4b0 con 0x7fb68c0704a0 2024-02-19T21:56:08.832 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:08.829+0000 7fb6897fa700 1 -- 172.21.15.53:0/703284440 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fb68001bb60 con 0x7fb68c0704a0 2024-02-19T21:56:08.832 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:08.829+0000 7fb6897fa700 1 -- 172.21.15.53:0/703284440 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb68001a9f0 con 0x7fb68c0704a0 2024-02-19T21:56:08.832 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:08.829+0000 7fb692fc7700 1 -- 172.21.15.53:0/703284440 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb68c10e0f0 con 0x7fb68c0704a0 2024-02-19T21:56:08.832 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:08.830+0000 7fb692fc7700 1 -- 172.21.15.53:0/703284440 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb68c10d940 con 0x7fb68c0704a0 2024-02-19T21:56:08.833 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:08.830+0000 7fb6897fa700 1 -- 172.21.15.53:0/703284440 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fb68000f070 con 0x7fb68c0704a0 2024-02-19T21:56:08.833 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:08.830+0000 7fb6897fa700 1 --2- 172.21.15.53:0/703284440 >> [v2:172.21.15.53:6800/1863695699,v1:172.21.15.53:6801/1863695699] conn(0x7fb67002fe10 0x7fb6700322d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:56:08.833 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:08.830+0000 7fb6897fa700 1 -- 172.21.15.53:0/703284440 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7fb6800414c0 con 0x7fb68c0704a0 2024-02-19T21:56:08.833 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:08.830+0000 7fb68bfff700 1 --2- 172.21.15.53:0/703284440 >> [v2:172.21.15.53:6800/1863695699,v1:172.21.15.53:6801/1863695699] conn(0x7fb67002fe10 0x7fb6700322d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:56:08.833 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:08.830+0000 7fb68bfff700 1 --2- 172.21.15.53:0/703284440 >> [v2:172.21.15.53:6800/1863695699,v1:172.21.15.53:6801/1863695699] conn(0x7fb67002fe10 0x7fb6700322d0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7fb6780096a0 tx=0x7fb678006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:56:08.834 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:08.834+0000 7fb6897fa700 1 -- 172.21.15.53:0/703284440 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb680019c90 con 0x7fb68c0704a0 2024-02-19T21:56:09.021 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:09.021+0000 7fb692fc7700 1 -- 172.21.15.53:0/703284440 --> [v2:172.21.15.53:6800/1863695699,v1:172.21.15.53:6801/1863695699] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}) v1 -- 0x7fb68c10e3f0 con 0x7fb67002fe10 2024-02-19T21:56:09.025 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:09.025+0000 7fb6897fa700 1 -- 172.21.15.53:0/703284440 <== mgr.14114 v2:172.21.15.53:6800/1863695699 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fb68c10e3f0 con 0x7fb67002fe10 2024-02-19T21:56:09.025 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: Scheduled mon update... 2024-02-19T21:56:09.027 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:09.027+0000 7fb692fc7700 1 -- 172.21.15.53:0/703284440 >> [v2:172.21.15.53:6800/1863695699,v1:172.21.15.53:6801/1863695699] conn(0x7fb67002fe10 msgr2=0x7fb6700322d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:56:09.028 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:09.027+0000 7fb692fc7700 1 --2- 172.21.15.53:0/703284440 >> [v2:172.21.15.53:6800/1863695699,v1:172.21.15.53:6801/1863695699] conn(0x7fb67002fe10 0x7fb6700322d0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7fb6780096a0 tx=0x7fb678006b40).stop 2024-02-19T21:56:09.028 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:09.027+0000 7fb692fc7700 1 -- 172.21.15.53:0/703284440 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb68c0704a0 msgr2=0x7fb68c10cdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:56:09.028 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:09.027+0000 7fb692fc7700 1 --2- 172.21.15.53:0/703284440 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb68c0704a0 0x7fb68c10cdc0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7fb680006800 tx=0x7fb680006ee0).stop 2024-02-19T21:56:09.028 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:09.027+0000 7fb692fc7700 1 -- 172.21.15.53:0/703284440 shutdown_connections 2024-02-19T21:56:09.028 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:09.027+0000 7fb692fc7700 1 --2- 172.21.15.53:0/703284440 >> [v2:172.21.15.53:6800/1863695699,v1:172.21.15.53:6801/1863695699] conn(0x7fb67002fe10 0x7fb6700322d0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:56:09.028 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:09.027+0000 7fb692fc7700 1 --2- 172.21.15.53:0/703284440 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb68c0704a0 0x7fb68c10cdc0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:56:09.028 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:09.027+0000 7fb692fc7700 1 -- 172.21.15.53:0/703284440 >> 172.21.15.53:0/703284440 conn(0x7fb68c0f65c0 msgr2=0x7fb68c0f7f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:56:09.028 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:09.027+0000 7fb692fc7700 1 -- 172.21.15.53:0/703284440 shutdown_connections 2024-02-19T21:56:09.028 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:09.027+0000 7fb692fc7700 1 -- 172.21.15.53:0/703284440 wait complete. 2024-02-19T21:56:09.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:56:08 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:56:07.595792+0000 mon.smithi053 (mon.0) 54 : audit [INF] from='mgr.14114 172.21.15.53:0/1866081671' entity='mgr.smithi053.fyhcdi' 2024-02-19T21:56:09.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:56:08 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cephadm 2024-02-19T21:56:07.596107+0000 mgr.smithi053.fyhcdi (mgr.14114) 9 : cephadm [INF] Added host smithi053 2024-02-19T21:56:09.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:56:08 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:56:07.596383+0000 mon.smithi053 (mon.0) 55 : audit [DBG] from='mgr.14114 172.21.15.53:0/1866081671' entity='mgr.smithi053.fyhcdi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T21:56:09.645 INFO:teuthology.orchestra.run.smithi053.stderr:Deploying mgr service with default placement... 2024-02-19T21:56:09.645 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi053 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/867b6b52-cf71-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpoyd_8g9j:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp0zy4ixsn:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply mgr 2024-02-19T21:56:10.228 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:10.226+0000 7f2c88315700 1 Processor -- start 2024-02-19T21:56:10.228 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:10.226+0000 7f2c88315700 1 -- start start 2024-02-19T21:56:10.228 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:10.226+0000 7f2c88315700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2c800703b0 0x7f2c80100ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T21:56:10.228 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:10.226+0000 7f2c88315700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c800fc8c0 con 0x7f2c800703b0 2024-02-19T21:56:10.228 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:10.227+0000 7f2c860b1700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2c800703b0 0x7f2c80100ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:56:10.228 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:10.227+0000 7f2c860b1700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2c800703b0 0x7f2c80100ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:36048/0 (socket says 172.21.15.53:36048) 2024-02-19T21:56:10.228 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:10.227+0000 7f2c860b1700 1 -- 172.21.15.53:0/1369460517 learned_addr learned my addr 172.21.15.53:0/1369460517 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-02-19T21:56:10.228 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:10.227+0000 7f2c860b1700 1 -- 172.21.15.53:0/1369460517 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c801013c0 con 0x7f2c800703b0 2024-02-19T21:56:10.229 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:10.227+0000 7f2c860b1700 1 --2- 172.21.15.53:0/1369460517 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2c800703b0 0x7f2c80100ed0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f2c7000b5f0 tx=0x7f2c70006ee0).ready entity=mon.0 client_cookie=1d837a264f83628e server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:56:10.229 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:10.227+0000 7f2c850af700 1 -- 172.21.15.53:0/1369460517 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2c7000d600 con 0x7f2c800703b0 2024-02-19T21:56:10.229 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:10.227+0000 7f2c850af700 1 -- 172.21.15.53:0/1369460517 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f2c7000dbf0 con 0x7f2c800703b0 2024-02-19T21:56:10.229 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:10.227+0000 7f2c850af700 1 -- 172.21.15.53:0/1369460517 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2c70011cb0 con 0x7f2c800703b0 2024-02-19T21:56:10.229 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:10.228+0000 7f2c88315700 1 -- 172.21.15.53:0/1369460517 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2c800703b0 msgr2=0x7f2c80100ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:56:10.229 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:10.228+0000 7f2c88315700 1 --2- 172.21.15.53:0/1369460517 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2c800703b0 0x7f2c80100ed0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f2c7000b5f0 tx=0x7f2c70006ee0).stop 2024-02-19T21:56:10.229 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:10.228+0000 7f2c88315700 1 -- 172.21.15.53:0/1369460517 shutdown_connections 2024-02-19T21:56:10.229 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:10.228+0000 7f2c88315700 1 --2- 172.21.15.53:0/1369460517 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2c800703b0 0x7f2c80100ed0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:56:10.232 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:10.228+0000 7f2c88315700 1 -- 172.21.15.53:0/1369460517 >> 172.21.15.53:0/1369460517 conn(0x7f2c800f6530 msgr2=0x7f2c800f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:56:10.232 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:10.228+0000 7f2c88315700 1 -- 172.21.15.53:0/1369460517 shutdown_connections 2024-02-19T21:56:10.232 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:10.228+0000 7f2c88315700 1 -- 172.21.15.53:0/1369460517 wait complete. 2024-02-19T21:56:10.232 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:10.229+0000 7f2c88315700 1 Processor -- start 2024-02-19T21:56:10.232 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:10.229+0000 7f2c88315700 1 -- start start 2024-02-19T21:56:10.232 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:10.229+0000 7f2c88315700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2c800703b0 0x7f2c8010cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:56:10.232 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:10.229+0000 7f2c88315700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c8010d270 con 0x7f2c800703b0 2024-02-19T21:56:10.232 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:10.229+0000 7f2c860b1700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2c800703b0 0x7f2c8010cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:56:10.232 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:10.229+0000 7f2c860b1700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2c800703b0 0x7f2c8010cd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:36062/0 (socket says 172.21.15.53:36062) 2024-02-19T21:56:10.232 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:10.229+0000 7f2c860b1700 1 -- 172.21.15.53:0/3538043403 learned_addr learned my addr 172.21.15.53:0/3538043403 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-02-19T21:56:10.233 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:10.229+0000 7f2c860b1700 1 -- 172.21.15.53:0/3538043403 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c7000b200 con 0x7f2c800703b0 2024-02-19T21:56:10.233 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:10.229+0000 7f2c860b1700 1 --2- 172.21.15.53:0/3538043403 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2c800703b0 0x7f2c8010cd80 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f2c70006800 tx=0x7f2c70006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:56:10.233 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:10.230+0000 7f2c6effd700 1 -- 172.21.15.53:0/3538043403 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2c7001b570 con 0x7f2c800703b0 2024-02-19T21:56:10.233 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:10.230+0000 7f2c6effd700 1 -- 172.21.15.53:0/3538043403 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f2c7001bb60 con 0x7f2c800703b0 2024-02-19T21:56:10.233 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:10.230+0000 7f2c88315700 1 -- 172.21.15.53:0/3538043403 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c8010d470 con 0x7f2c800703b0 2024-02-19T21:56:10.233 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:10.230+0000 7f2c6effd700 1 -- 172.21.15.53:0/3538043403 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2c7001a9f0 con 0x7f2c800703b0 2024-02-19T21:56:10.233 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:10.230+0000 7f2c88315700 1 -- 172.21.15.53:0/3538043403 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c8010e080 con 0x7f2c800703b0 2024-02-19T21:56:10.233 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:10.230+0000 7f2c6effd700 1 -- 172.21.15.53:0/3538043403 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f2c7000f070 con 0x7f2c800703b0 2024-02-19T21:56:10.233 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:10.231+0000 7f2c6effd700 1 --2- 172.21.15.53:0/3538043403 >> [v2:172.21.15.53:6800/1863695699,v1:172.21.15.53:6801/1863695699] conn(0x7f2c6802fe70 0x7f2c68032330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:56:10.233 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:10.231+0000 7f2c6effd700 1 -- 172.21.15.53:0/3538043403 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f2c700413c0 con 0x7f2c800703b0 2024-02-19T21:56:10.233 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:10.231+0000 7f2c858b0700 1 --2- 172.21.15.53:0/3538043403 >> [v2:172.21.15.53:6800/1863695699,v1:172.21.15.53:6801/1863695699] conn(0x7f2c6802fe70 0x7f2c68032330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:56:10.233 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:10.231+0000 7f2c858b0700 1 --2- 172.21.15.53:0/3538043403 >> [v2:172.21.15.53:6800/1863695699,v1:172.21.15.53:6801/1863695699] conn(0x7f2c6802fe70 0x7f2c68032330 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f2c7c0096a0 tx=0x7f2c7c006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:56:10.233 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:10.231+0000 7f2c88315700 1 -- 172.21.15.53:0/3538043403 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c80059100 con 0x7f2c800703b0 2024-02-19T21:56:10.235 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:10.235+0000 7f2c6effd700 1 -- 172.21.15.53:0/3538043403 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2c70016020 con 0x7f2c800703b0 2024-02-19T21:56:10.293 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:56:10 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:56:09.023232+0000 mgr.smithi053.fyhcdi (mgr.14114) 10 : audit [DBG] from='client.14134 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T21:56:10.293 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:56:10 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cephadm 2024-02-19T21:56:09.023984+0000 mgr.smithi053.fyhcdi (mgr.14114) 11 : cephadm [INF] Saving service mon spec with placement count:5 2024-02-19T21:56:10.293 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:56:10 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:56:09.026192+0000 mon.smithi053 (mon.0) 56 : audit [INF] from='mgr.14114 172.21.15.53:0/1866081671' entity='mgr.smithi053.fyhcdi' 2024-02-19T21:56:10.425 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:10.425+0000 7f2c88315700 1 -- 172.21.15.53:0/3538043403 --> [v2:172.21.15.53:6800/1863695699,v1:172.21.15.53:6801/1863695699] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7f2c80102140 con 0x7f2c6802fe70 2024-02-19T21:56:10.428 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:10.428+0000 7f2c6effd700 1 -- 172.21.15.53:0/3538043403 <== mgr.14114 v2:172.21.15.53:6800/1863695699 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f2c80102140 con 0x7f2c6802fe70 2024-02-19T21:56:10.428 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: Scheduled mgr update... 2024-02-19T21:56:10.431 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:10.430+0000 7f2c88315700 1 -- 172.21.15.53:0/3538043403 >> [v2:172.21.15.53:6800/1863695699,v1:172.21.15.53:6801/1863695699] conn(0x7f2c6802fe70 msgr2=0x7f2c68032330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:56:10.431 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:10.430+0000 7f2c88315700 1 --2- 172.21.15.53:0/3538043403 >> [v2:172.21.15.53:6800/1863695699,v1:172.21.15.53:6801/1863695699] conn(0x7f2c6802fe70 0x7f2c68032330 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f2c7c0096a0 tx=0x7f2c7c006b40).stop 2024-02-19T21:56:10.431 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:10.430+0000 7f2c88315700 1 -- 172.21.15.53:0/3538043403 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2c800703b0 msgr2=0x7f2c8010cd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:56:10.431 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:10.430+0000 7f2c88315700 1 --2- 172.21.15.53:0/3538043403 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2c800703b0 0x7f2c8010cd80 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f2c70006800 tx=0x7f2c70006ee0).stop 2024-02-19T21:56:10.431 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:10.430+0000 7f2c88315700 1 -- 172.21.15.53:0/3538043403 shutdown_connections 2024-02-19T21:56:10.432 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:10.430+0000 7f2c88315700 1 --2- 172.21.15.53:0/3538043403 >> [v2:172.21.15.53:6800/1863695699,v1:172.21.15.53:6801/1863695699] conn(0x7f2c6802fe70 0x7f2c68032330 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:56:10.432 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:10.430+0000 7f2c88315700 1 --2- 172.21.15.53:0/3538043403 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2c800703b0 0x7f2c8010cd80 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:56:10.432 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:10.430+0000 7f2c88315700 1 -- 172.21.15.53:0/3538043403 >> 172.21.15.53:0/3538043403 conn(0x7f2c800f6530 msgr2=0x7f2c800f7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:56:10.432 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:10.431+0000 7f2c88315700 1 -- 172.21.15.53:0/3538043403 shutdown_connections 2024-02-19T21:56:10.432 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:10.431+0000 7f2c88315700 1 -- 172.21.15.53:0/3538043403 wait complete. 2024-02-19T21:56:11.001 INFO:teuthology.orchestra.run.smithi053.stderr:Deploying crash service with default placement... 2024-02-19T21:56:11.002 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi053 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/867b6b52-cf71-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpoyd_8g9j:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp0zy4ixsn:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply crash 2024-02-19T21:56:11.548 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:11.530+0000 7fc36b1cb700 1 Processor -- start 2024-02-19T21:56:11.548 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:11.530+0000 7fc36b1cb700 1 -- start start 2024-02-19T21:56:11.548 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:11.530+0000 7fc36b1cb700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc3640703b0 0x7fc364100ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T21:56:11.548 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:11.530+0000 7fc36b1cb700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3640fc8c0 con 0x7fc3640703b0 2024-02-19T21:56:11.548 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:11.531+0000 7fc368f67700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc3640703b0 0x7fc364100ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:56:11.549 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:11.531+0000 7fc368f67700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc3640703b0 0x7fc364100ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:36078/0 (socket says 172.21.15.53:36078) 2024-02-19T21:56:11.549 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:11.531+0000 7fc368f67700 1 -- 172.21.15.53:0/1542326826 learned_addr learned my addr 172.21.15.53:0/1542326826 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-02-19T21:56:11.549 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:11.531+0000 7fc368f67700 1 -- 172.21.15.53:0/1542326826 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3641013c0 con 0x7fc3640703b0 2024-02-19T21:56:11.549 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:11.531+0000 7fc368f67700 1 --2- 172.21.15.53:0/1542326826 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc3640703b0 0x7fc364100ed0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7fc35800b5f0 tx=0x7fc358006ee0).ready entity=mon.0 client_cookie=dace6bf3e0ec8176 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:56:11.549 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:11.531+0000 7fc3637fe700 1 -- 172.21.15.53:0/1542326826 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc35800d600 con 0x7fc3640703b0 2024-02-19T21:56:11.549 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:11.531+0000 7fc3637fe700 1 -- 172.21.15.53:0/1542326826 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fc35800dbf0 con 0x7fc3640703b0 2024-02-19T21:56:11.549 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:11.531+0000 7fc3637fe700 1 -- 172.21.15.53:0/1542326826 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc358011cb0 con 0x7fc3640703b0 2024-02-19T21:56:11.549 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:11.532+0000 7fc36b1cb700 1 -- 172.21.15.53:0/1542326826 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc3640703b0 msgr2=0x7fc364100ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:56:11.549 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:11.532+0000 7fc36b1cb700 1 --2- 172.21.15.53:0/1542326826 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc3640703b0 0x7fc364100ed0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7fc35800b5f0 tx=0x7fc358006ee0).stop 2024-02-19T21:56:11.549 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:11.532+0000 7fc36b1cb700 1 -- 172.21.15.53:0/1542326826 shutdown_connections 2024-02-19T21:56:11.550 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:11.532+0000 7fc36b1cb700 1 --2- 172.21.15.53:0/1542326826 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc3640703b0 0x7fc364100ed0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:56:11.550 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:11.532+0000 7fc36b1cb700 1 -- 172.21.15.53:0/1542326826 >> 172.21.15.53:0/1542326826 conn(0x7fc3640f6530 msgr2=0x7fc3640f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:56:11.550 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:11.532+0000 7fc36b1cb700 1 -- 172.21.15.53:0/1542326826 shutdown_connections 2024-02-19T21:56:11.550 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:11.532+0000 7fc36b1cb700 1 -- 172.21.15.53:0/1542326826 wait complete. 2024-02-19T21:56:11.550 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:11.532+0000 7fc36b1cb700 1 Processor -- start 2024-02-19T21:56:11.550 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:11.533+0000 7fc36b1cb700 1 -- start start 2024-02-19T21:56:11.550 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:11.533+0000 7fc36b1cb700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc3640703b0 0x7fc36410cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:56:11.550 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:11.533+0000 7fc36b1cb700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc36410d2c0 con 0x7fc3640703b0 2024-02-19T21:56:11.550 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:11.533+0000 7fc368f67700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc3640703b0 0x7fc36410cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:56:11.550 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:11.533+0000 7fc368f67700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc3640703b0 0x7fc36410cdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:36086/0 (socket says 172.21.15.53:36086) 2024-02-19T21:56:11.550 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:11.533+0000 7fc368f67700 1 -- 172.21.15.53:0/2315193504 learned_addr learned my addr 172.21.15.53:0/2315193504 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-02-19T21:56:11.551 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:11.533+0000 7fc368f67700 1 -- 172.21.15.53:0/2315193504 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc35800b200 con 0x7fc3640703b0 2024-02-19T21:56:11.551 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:11.533+0000 7fc368f67700 1 --2- 172.21.15.53:0/2315193504 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc3640703b0 0x7fc36410cdd0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7fc35800df90 tx=0x7fc358006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:56:11.551 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:11.534+0000 7fc3617fa700 1 -- 172.21.15.53:0/2315193504 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc35801b410 con 0x7fc3640703b0 2024-02-19T21:56:11.551 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:11.534+0000 7fc3617fa700 1 -- 172.21.15.53:0/2315193504 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fc35801ba00 con 0x7fc3640703b0 2024-02-19T21:56:11.551 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:11.534+0000 7fc36b1cb700 1 -- 172.21.15.53:0/2315193504 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc36410d4c0 con 0x7fc3640703b0 2024-02-19T21:56:11.551 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:11.534+0000 7fc3617fa700 1 -- 172.21.15.53:0/2315193504 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc35801aa30 con 0x7fc3640703b0 2024-02-19T21:56:11.551 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:11.534+0000 7fc36b1cb700 1 -- 172.21.15.53:0/2315193504 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc36410e100 con 0x7fc3640703b0 2024-02-19T21:56:11.551 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:11.535+0000 7fc3617fa700 1 -- 172.21.15.53:0/2315193504 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fc35800f070 con 0x7fc3640703b0 2024-02-19T21:56:11.551 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:11.535+0000 7fc36b1cb700 1 -- 172.21.15.53:0/2315193504 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc36410d950 con 0x7fc3640703b0 2024-02-19T21:56:11.551 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:11.535+0000 7fc3617fa700 1 --2- 172.21.15.53:0/2315193504 >> [v2:172.21.15.53:6800/1863695699,v1:172.21.15.53:6801/1863695699] conn(0x7fc34802fe70 0x7fc348032330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:56:11.551 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:11.535+0000 7fc3617fa700 1 -- 172.21.15.53:0/2315193504 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7fc358041190 con 0x7fc3640703b0 2024-02-19T21:56:11.551 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:11.535+0000 7fc363fff700 1 --2- 172.21.15.53:0/2315193504 >> [v2:172.21.15.53:6800/1863695699,v1:172.21.15.53:6801/1863695699] conn(0x7fc34802fe70 0x7fc348032330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:56:11.552 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:11.535+0000 7fc363fff700 1 --2- 172.21.15.53:0/2315193504 >> [v2:172.21.15.53:6800/1863695699,v1:172.21.15.53:6801/1863695699] conn(0x7fc34802fe70 0x7fc348032330 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7fc3500096a0 tx=0x7fc350006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:56:11.552 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:11.539+0000 7fc3617fa700 1 -- 172.21.15.53:0/2315193504 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc358022310 con 0x7fc3640703b0 2024-02-19T21:56:11.725 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:11.724+0000 7fc36b1cb700 1 -- 172.21.15.53:0/2315193504 --> [v2:172.21.15.53:6800/1863695699,v1:172.21.15.53:6801/1863695699] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}) v1 -- 0x7fc36410e400 con 0x7fc34802fe70 2024-02-19T21:56:11.728 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:11.728+0000 7fc3617fa700 1 -- 172.21.15.53:0/2315193504 <== mgr.14114 v2:172.21.15.53:6800/1863695699 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+26 (secure 0 0 0) 0x7fc36410e400 con 0x7fc34802fe70 2024-02-19T21:56:11.728 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: Scheduled crash update... 2024-02-19T21:56:11.731 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:11.730+0000 7fc36b1cb700 1 -- 172.21.15.53:0/2315193504 >> [v2:172.21.15.53:6800/1863695699,v1:172.21.15.53:6801/1863695699] conn(0x7fc34802fe70 msgr2=0x7fc348032330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:56:11.731 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:11.730+0000 7fc36b1cb700 1 --2- 172.21.15.53:0/2315193504 >> [v2:172.21.15.53:6800/1863695699,v1:172.21.15.53:6801/1863695699] conn(0x7fc34802fe70 0x7fc348032330 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7fc3500096a0 tx=0x7fc350006b40).stop 2024-02-19T21:56:11.731 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:11.730+0000 7fc36b1cb700 1 -- 172.21.15.53:0/2315193504 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc3640703b0 msgr2=0x7fc36410cdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:56:11.731 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:11.730+0000 7fc36b1cb700 1 --2- 172.21.15.53:0/2315193504 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc3640703b0 0x7fc36410cdd0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7fc35800df90 tx=0x7fc358006ee0).stop 2024-02-19T21:56:11.731 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:11.730+0000 7fc36b1cb700 1 -- 172.21.15.53:0/2315193504 shutdown_connections 2024-02-19T21:56:11.731 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:11.730+0000 7fc36b1cb700 1 --2- 172.21.15.53:0/2315193504 >> [v2:172.21.15.53:6800/1863695699,v1:172.21.15.53:6801/1863695699] conn(0x7fc34802fe70 0x7fc348032330 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:56:11.731 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:11.730+0000 7fc36b1cb700 1 --2- 172.21.15.53:0/2315193504 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc3640703b0 0x7fc36410cdd0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:56:11.732 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:11.730+0000 7fc36b1cb700 1 -- 172.21.15.53:0/2315193504 >> 172.21.15.53:0/2315193504 conn(0x7fc3640f6530 msgr2=0x7fc3640f7fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:56:11.732 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:11.730+0000 7fc36b1cb700 1 -- 172.21.15.53:0/2315193504 shutdown_connections 2024-02-19T21:56:11.732 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:11.730+0000 7fc36b1cb700 1 -- 172.21.15.53:0/2315193504 wait complete. 2024-02-19T21:56:11.750 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:56:11 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:56:10.426891+0000 mgr.smithi053.fyhcdi (mgr.14114) 12 : audit [DBG] from='client.14136 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T21:56:11.751 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:56:11 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cephadm 2024-02-19T21:56:10.427423+0000 mgr.smithi053.fyhcdi (mgr.14114) 13 : cephadm [INF] Saving service mgr spec with placement count:2 2024-02-19T21:56:11.751 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:56:11 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:56:10.429619+0000 mon.smithi053 (mon.0) 57 : audit [INF] from='mgr.14114 172.21.15.53:0/1866081671' entity='mgr.smithi053.fyhcdi' 2024-02-19T21:56:12.325 INFO:teuthology.orchestra.run.smithi053.stderr:Enabling mgr prometheus module... 2024-02-19T21:56:12.326 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi053 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/867b6b52-cf71-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpoyd_8g9j:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp0zy4ixsn:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable prometheus 2024-02-19T21:56:12.848 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:12.846+0000 7fc150236700 1 Processor -- start 2024-02-19T21:56:12.849 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:12.846+0000 7fc150236700 1 -- start start 2024-02-19T21:56:12.849 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:12.846+0000 7fc150236700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc1480fbea0 0x7fc1480fc2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T21:56:12.849 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:12.846+0000 7fc150236700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1480fc7b0 con 0x7fc1480fbea0 2024-02-19T21:56:12.849 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:12.846+0000 7fc14dfd2700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc1480fbea0 0x7fc1480fc2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:56:12.849 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:12.846+0000 7fc14dfd2700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc1480fbea0 0x7fc1480fc2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:36102/0 (socket says 172.21.15.53:36102) 2024-02-19T21:56:12.849 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:12.846+0000 7fc14dfd2700 1 -- 172.21.15.53:0/1994230757 learned_addr learned my addr 172.21.15.53:0/1994230757 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-02-19T21:56:12.849 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:12.847+0000 7fc14dfd2700 1 -- 172.21.15.53:0/1994230757 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1480fc8f0 con 0x7fc1480fbea0 2024-02-19T21:56:12.849 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:12.847+0000 7fc14dfd2700 1 --2- 172.21.15.53:0/1994230757 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc1480fbea0 0x7fc1480fc2c0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7fc14400b5f0 tx=0x7fc144006ee0).ready entity=mon.0 client_cookie=273b242e9a9d2e3 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:56:12.850 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:12.847+0000 7fc14cfd0700 1 -- 172.21.15.53:0/1994230757 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc14400d600 con 0x7fc1480fbea0 2024-02-19T21:56:12.850 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:12.847+0000 7fc14cfd0700 1 -- 172.21.15.53:0/1994230757 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fc14400dbf0 con 0x7fc1480fbea0 2024-02-19T21:56:12.850 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:12.847+0000 7fc14cfd0700 1 -- 172.21.15.53:0/1994230757 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc144011cb0 con 0x7fc1480fbea0 2024-02-19T21:56:12.850 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:12.847+0000 7fc150236700 1 -- 172.21.15.53:0/1994230757 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc1480fbea0 msgr2=0x7fc1480fc2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:56:12.850 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:12.847+0000 7fc150236700 1 --2- 172.21.15.53:0/1994230757 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc1480fbea0 0x7fc1480fc2c0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7fc14400b5f0 tx=0x7fc144006ee0).stop 2024-02-19T21:56:12.850 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:12.847+0000 7fc150236700 1 -- 172.21.15.53:0/1994230757 shutdown_connections 2024-02-19T21:56:12.850 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:12.847+0000 7fc150236700 1 --2- 172.21.15.53:0/1994230757 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc1480fbea0 0x7fc1480fc2c0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:56:12.850 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:12.848+0000 7fc150236700 1 -- 172.21.15.53:0/1994230757 >> 172.21.15.53:0/1994230757 conn(0x7fc1480f76a0 msgr2=0x7fc1480f9ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:56:12.850 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:12.848+0000 7fc150236700 1 -- 172.21.15.53:0/1994230757 shutdown_connections 2024-02-19T21:56:12.850 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:12.848+0000 7fc150236700 1 -- 172.21.15.53:0/1994230757 wait complete. 2024-02-19T21:56:12.850 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:12.848+0000 7fc150236700 1 Processor -- start 2024-02-19T21:56:12.850 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:12.848+0000 7fc150236700 1 -- start start 2024-02-19T21:56:12.851 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:12.849+0000 7fc150236700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc1480fbea0 0x7fc1480725e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:56:12.851 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:12.849+0000 7fc150236700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc148070a60 con 0x7fc1480fbea0 2024-02-19T21:56:12.853 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:12.849+0000 7fc14dfd2700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc1480fbea0 0x7fc1480725e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:56:12.853 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:12.849+0000 7fc14dfd2700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc1480fbea0 0x7fc1480725e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:36114/0 (socket says 172.21.15.53:36114) 2024-02-19T21:56:12.853 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:12.849+0000 7fc14dfd2700 1 -- 172.21.15.53:0/3954798525 learned_addr learned my addr 172.21.15.53:0/3954798525 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-02-19T21:56:12.853 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:12.849+0000 7fc14dfd2700 1 -- 172.21.15.53:0/3954798525 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc14400b200 con 0x7fc1480fbea0 2024-02-19T21:56:12.853 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:12.849+0000 7fc14dfd2700 1 --2- 172.21.15.53:0/3954798525 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc1480fbea0 0x7fc1480725e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7fc14400bf40 tx=0x7fc144006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:56:12.853 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:12.849+0000 7fc1367fc700 1 -- 172.21.15.53:0/3954798525 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc14401b570 con 0x7fc1480fbea0 2024-02-19T21:56:12.853 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:12.849+0000 7fc1367fc700 1 -- 172.21.15.53:0/3954798525 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fc14401bb60 con 0x7fc1480fbea0 2024-02-19T21:56:12.853 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:12.849+0000 7fc150236700 1 -- 172.21.15.53:0/3954798525 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc148070c00 con 0x7fc1480fbea0 2024-02-19T21:56:12.854 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:12.849+0000 7fc1367fc700 1 -- 172.21.15.53:0/3954798525 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc14401aa20 con 0x7fc1480fbea0 2024-02-19T21:56:12.854 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:12.849+0000 7fc150236700 1 -- 172.21.15.53:0/3954798525 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc148071090 con 0x7fc1480fbea0 2024-02-19T21:56:12.854 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:12.850+0000 7fc1367fc700 1 -- 172.21.15.53:0/3954798525 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fc14400f070 con 0x7fc1480fbea0 2024-02-19T21:56:12.854 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:12.850+0000 7fc1367fc700 1 --2- 172.21.15.53:0/3954798525 >> [v2:172.21.15.53:6800/1863695699,v1:172.21.15.53:6801/1863695699] conn(0x7fc13002fe10 0x7fc1300322d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:56:12.854 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:12.850+0000 7fc1367fc700 1 -- 172.21.15.53:0/3954798525 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7fc1440414d0 con 0x7fc1480fbea0 2024-02-19T21:56:12.854 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:12.850+0000 7fc150236700 1 -- 172.21.15.53:0/3954798525 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc148045ad0 con 0x7fc1480fbea0 2024-02-19T21:56:12.854 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:12.851+0000 7fc14d7d1700 1 --2- 172.21.15.53:0/3954798525 >> [v2:172.21.15.53:6800/1863695699,v1:172.21.15.53:6801/1863695699] conn(0x7fc13002fe10 0x7fc1300322d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:56:12.854 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:12.851+0000 7fc14d7d1700 1 --2- 172.21.15.53:0/3954798525 >> [v2:172.21.15.53:6800/1863695699,v1:172.21.15.53:6801/1863695699] conn(0x7fc13002fe10 0x7fc1300322d0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7fc1380096a0 tx=0x7fc138006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:56:12.855 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:12.855+0000 7fc1367fc700 1 -- 172.21.15.53:0/3954798525 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc144016070 con 0x7fc1480fbea0 2024-02-19T21:56:13.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:56:12 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:56:11.726515+0000 mgr.smithi053.fyhcdi (mgr.14114) 14 : audit [DBG] from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T21:56:13.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:56:12 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cephadm 2024-02-19T21:56:11.727156+0000 mgr.smithi053.fyhcdi (mgr.14114) 15 : cephadm [INF] Saving service crash spec with placement * 2024-02-19T21:56:13.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:56:12 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:56:11.729390+0000 mon.smithi053 (mon.0) 58 : audit [INF] from='mgr.14114 172.21.15.53:0/1866081671' entity='mgr.smithi053.fyhcdi' 2024-02-19T21:56:13.052 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:13.052+0000 7fc150236700 1 -- 172.21.15.53:0/3954798525 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "prometheus"} v 0) v1 -- 0x7fc1480008d0 con 0x7fc1480fbea0 2024-02-19T21:56:13.733 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:13.732+0000 7fc1367fc700 1 -- 172.21.15.53:0/3954798525 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mgrmap(e 9) v1 ==== 36134+0+0 (secure 0 0 0) 0x7fc14401bcd0 con 0x7fc1480fbea0 2024-02-19T21:56:13.733 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:13.733+0000 7fc1367fc700 1 -- 172.21.15.53:0/3954798525 <== mon.0 v2:172.21.15.53:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "prometheus"}]=0 v9) v1 ==== 89+0+0 (secure 0 0 0) 0x7fc1440058e0 con 0x7fc1480fbea0 2024-02-19T21:56:13.735 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:13.735+0000 7fc150236700 1 -- 172.21.15.53:0/3954798525 >> [v2:172.21.15.53:6800/1863695699,v1:172.21.15.53:6801/1863695699] conn(0x7fc13002fe10 msgr2=0x7fc1300322d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:56:13.736 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:13.735+0000 7fc150236700 1 --2- 172.21.15.53:0/3954798525 >> [v2:172.21.15.53:6800/1863695699,v1:172.21.15.53:6801/1863695699] conn(0x7fc13002fe10 0x7fc1300322d0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7fc1380096a0 tx=0x7fc138006b40).stop 2024-02-19T21:56:13.736 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:13.735+0000 7fc150236700 1 -- 172.21.15.53:0/3954798525 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc1480fbea0 msgr2=0x7fc1480725e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:56:13.736 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:13.735+0000 7fc150236700 1 --2- 172.21.15.53:0/3954798525 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc1480fbea0 0x7fc1480725e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7fc14400bf40 tx=0x7fc144006ee0).stop 2024-02-19T21:56:13.737 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:13.736+0000 7fc150236700 1 -- 172.21.15.53:0/3954798525 shutdown_connections 2024-02-19T21:56:13.737 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:13.736+0000 7fc150236700 1 --2- 172.21.15.53:0/3954798525 >> [v2:172.21.15.53:6800/1863695699,v1:172.21.15.53:6801/1863695699] conn(0x7fc13002fe10 0x7fc1300322d0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:56:13.737 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:13.736+0000 7fc150236700 1 --2- 172.21.15.53:0/3954798525 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc1480fbea0 0x7fc1480725e0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:56:13.737 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:13.736+0000 7fc150236700 1 -- 172.21.15.53:0/3954798525 >> 172.21.15.53:0/3954798525 conn(0x7fc1480f76a0 msgr2=0x7fc1480f8fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:56:13.737 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:13.736+0000 7fc150236700 1 -- 172.21.15.53:0/3954798525 shutdown_connections 2024-02-19T21:56:13.738 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:13.736+0000 7fc150236700 1 -- 172.21.15.53:0/3954798525 wait complete. 2024-02-19T21:56:14.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:56:13 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:56:13.054141+0000 mon.smithi053 (mon.0) 59 : audit [INF] from='client.? 172.21.15.53:0/3954798525' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-02-19T21:56:14.456 INFO:teuthology.orchestra.run.smithi053.stderr:Deploying prometheus service with default placement... 2024-02-19T21:56:14.457 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi053 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/867b6b52-cf71-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpoyd_8g9j:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp0zy4ixsn:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply prometheus 2024-02-19T21:56:15.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:56:14 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:56:13.734382+0000 mon.smithi053 (mon.0) 60 : audit [INF] from='client.? 172.21.15.53:0/3954798525' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-02-19T21:56:15.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:56:14 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T21:56:13.734470+0000 mon.smithi053 (mon.0) 61 : cluster [DBG] mgrmap e9: smithi053.fyhcdi(active, since 14s) 2024-02-19T21:56:15.158 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:15.157+0000 7f104f59e700 1 Processor -- start 2024-02-19T21:56:15.158 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:15.157+0000 7f104f59e700 1 -- start start 2024-02-19T21:56:15.158 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:15.157+0000 7f104f59e700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f10500681e0 0x7f1050068600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T21:56:15.158 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:15.157+0000 7f104f59e700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1050069950 con 0x7f10500681e0 2024-02-19T21:56:15.158 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:15.157+0000 7f104e59c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f10500681e0 0x7f1050068600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:56:15.158 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:15.157+0000 7f104e59c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f10500681e0 0x7f1050068600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:36146/0 (socket says 172.21.15.53:36146) 2024-02-19T21:56:15.158 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:15.158+0000 7f104e59c700 1 -- 172.21.15.53:0/1096766569 learned_addr learned my addr 172.21.15.53:0/1096766569 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-02-19T21:56:15.159 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:15.158+0000 7f104e59c700 1 -- 172.21.15.53:0/1096766569 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1050068af0 con 0x7f10500681e0 2024-02-19T21:56:15.159 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:15.158+0000 7f104e59c700 1 --2- 172.21.15.53:0/1096766569 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f10500681e0 0x7f1050068600 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f104000b5f0 tx=0x7f1040006ee0).ready entity=mon.0 client_cookie=52650f1e87f4556e server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:56:15.160 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:15.158+0000 7f104d59a700 1 -- 172.21.15.53:0/1096766569 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f104000d600 con 0x7f10500681e0 2024-02-19T21:56:15.160 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:15.158+0000 7f104d59a700 1 -- 172.21.15.53:0/1096766569 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f104000dbf0 con 0x7f10500681e0 2024-02-19T21:56:15.160 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:15.158+0000 7f104d59a700 1 -- 172.21.15.53:0/1096766569 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1040011cb0 con 0x7f10500681e0 2024-02-19T21:56:15.160 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:15.159+0000 7f104f59e700 1 -- 172.21.15.53:0/1096766569 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f10500681e0 msgr2=0x7f1050068600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:56:15.160 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:15.159+0000 7f104f59e700 1 --2- 172.21.15.53:0/1096766569 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f10500681e0 0x7f1050068600 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f104000b5f0 tx=0x7f1040006ee0).stop 2024-02-19T21:56:15.161 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:15.159+0000 7f104f59e700 1 -- 172.21.15.53:0/1096766569 shutdown_connections 2024-02-19T21:56:15.161 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:15.159+0000 7f104f59e700 1 --2- 172.21.15.53:0/1096766569 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f10500681e0 0x7f1050068600 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:56:15.161 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:15.159+0000 7f104f59e700 1 -- 172.21.15.53:0/1096766569 >> 172.21.15.53:0/1096766569 conn(0x7f1050063ba0 msgr2=0x7f1050065fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:56:15.161 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:15.159+0000 7f104f59e700 1 -- 172.21.15.53:0/1096766569 shutdown_connections 2024-02-19T21:56:15.161 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:15.159+0000 7f104f59e700 1 -- 172.21.15.53:0/1096766569 wait complete. 2024-02-19T21:56:15.161 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:15.160+0000 7f104f59e700 1 Processor -- start 2024-02-19T21:56:15.162 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:15.160+0000 7f104f59e700 1 -- start start 2024-02-19T21:56:15.162 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:15.160+0000 7f104f59e700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f10500681e0 0x7f1050115380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:56:15.162 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:15.160+0000 7f104f59e700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1050115870 con 0x7f10500681e0 2024-02-19T21:56:15.162 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:15.160+0000 7f104e59c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f10500681e0 0x7f1050115380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:56:15.162 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:15.160+0000 7f104e59c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f10500681e0 0x7f1050115380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:36156/0 (socket says 172.21.15.53:36156) 2024-02-19T21:56:15.162 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:15.160+0000 7f104e59c700 1 -- 172.21.15.53:0/875972132 learned_addr learned my addr 172.21.15.53:0/875972132 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-02-19T21:56:15.163 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:15.160+0000 7f104e59c700 1 -- 172.21.15.53:0/875972132 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f104000b200 con 0x7f10500681e0 2024-02-19T21:56:15.163 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:15.161+0000 7f104e59c700 1 --2- 172.21.15.53:0/875972132 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f10500681e0 0x7f1050115380 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f104000b190 tx=0x7f1040006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:56:15.163 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:15.161+0000 7f103effd700 1 -- 172.21.15.53:0/875972132 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f104001b410 con 0x7f10500681e0 2024-02-19T21:56:15.163 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:15.161+0000 7f103effd700 1 -- 172.21.15.53:0/875972132 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f104001ba00 con 0x7f10500681e0 2024-02-19T21:56:15.164 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:15.161+0000 7f103effd700 1 -- 172.21.15.53:0/875972132 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f104001a920 con 0x7f10500681e0 2024-02-19T21:56:15.164 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:15.161+0000 7f104f59e700 1 -- 172.21.15.53:0/875972132 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1050115a70 con 0x7f10500681e0 2024-02-19T21:56:15.164 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:15.161+0000 7f104f59e700 1 -- 172.21.15.53:0/875972132 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1050115f00 con 0x7f10500681e0 2024-02-19T21:56:15.164 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:15.162+0000 7f103effd700 1 -- 172.21.15.53:0/875972132 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 9) v1 ==== 36134+0+0 (secure 0 0 0) 0x7f104000f070 con 0x7f10500681e0 2024-02-19T21:56:15.164 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:15.162+0000 7f104f59e700 1 -- 172.21.15.53:0/875972132 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1050045ad0 con 0x7f10500681e0 2024-02-19T21:56:15.164 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:15.162+0000 7f103effd700 1 --2- 172.21.15.53:0/875972132 >> [v2:172.21.15.53:6800/1863695699,v1:172.21.15.53:6801/1863695699] conn(0x7f1034038720 0x7f103403abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:56:15.164 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:15.162+0000 7f103effd700 1 -- 172.21.15.53:0/875972132 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f1040041310 con 0x7f10500681e0 2024-02-19T21:56:15.164 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:15.164+0000 7f104dd9b700 1 -- 172.21.15.53:0/875972132 >> [v2:172.21.15.53:6800/1863695699,v1:172.21.15.53:6801/1863695699] conn(0x7f1034038720 msgr2=0x7f103403abe0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.53:6800/1863695699 2024-02-19T21:56:15.165 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:15.164+0000 7f104dd9b700 1 --2- 172.21.15.53:0/875972132 >> [v2:172.21.15.53:6800/1863695699,v1:172.21.15.53:6801/1863695699] conn(0x7f1034038720 0x7f103403abe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-02-19T21:56:15.166 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:15.167+0000 7f103effd700 1 -- 172.21.15.53:0/875972132 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1040010360 con 0x7f10500681e0 2024-02-19T21:56:15.356 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:15.356+0000 7f104f59e700 1 -- 172.21.15.53:0/875972132 --> [v2:172.21.15.53:6800/1863695699,v1:172.21.15.53:6801/1863695699] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7f1050116760 con 0x7f1034038720 2024-02-19T21:56:15.365 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:15.365+0000 7f104dd9b700 1 -- 172.21.15.53:0/875972132 >> [v2:172.21.15.53:6800/1863695699,v1:172.21.15.53:6801/1863695699] conn(0x7f1034038720 msgr2=0x7f103403abe0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.53:6800/1863695699 2024-02-19T21:56:15.365 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:15.365+0000 7f104dd9b700 1 --2- 172.21.15.53:0/875972132 >> [v2:172.21.15.53:6800/1863695699,v1:172.21.15.53:6801/1863695699] conn(0x7f1034038720 0x7f103403abe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2024-02-19T21:56:15.766 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:15.766+0000 7f104dd9b700 1 -- 172.21.15.53:0/875972132 >> [v2:172.21.15.53:6800/1863695699,v1:172.21.15.53:6801/1863695699] conn(0x7f1034038720 msgr2=0x7f103403abe0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.53:6800/1863695699 2024-02-19T21:56:15.766 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:15.766+0000 7f104dd9b700 1 --2- 172.21.15.53:0/875972132 >> [v2:172.21.15.53:6800/1863695699,v1:172.21.15.53:6801/1863695699] conn(0x7f1034038720 0x7f103403abe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2024-02-19T21:56:16.567 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:16.567+0000 7f104dd9b700 1 -- 172.21.15.53:0/875972132 >> [v2:172.21.15.53:6800/1863695699,v1:172.21.15.53:6801/1863695699] conn(0x7f1034038720 msgr2=0x7f103403abe0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.53:6800/1863695699 2024-02-19T21:56:16.567 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:16.567+0000 7f104dd9b700 1 --2- 172.21.15.53:0/875972132 >> [v2:172.21.15.53:6800/1863695699,v1:172.21.15.53:6801/1863695699] conn(0x7f1034038720 0x7f103403abe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2024-02-19T21:56:18.168 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:18.168+0000 7f104dd9b700 1 -- 172.21.15.53:0/875972132 >> [v2:172.21.15.53:6800/1863695699,v1:172.21.15.53:6801/1863695699] conn(0x7f1034038720 msgr2=0x7f103403abe0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.53:6800/1863695699 2024-02-19T21:56:18.168 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:18.168+0000 7f104dd9b700 1 --2- 172.21.15.53:0/875972132 >> [v2:172.21.15.53:6800/1863695699,v1:172.21.15.53:6801/1863695699] conn(0x7f1034038720 0x7f103403abe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 3.200000 2024-02-19T21:56:19.029 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:19.028+0000 7f103effd700 1 -- 172.21.15.53:0/875972132 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mgrmap(e 10) v1 ==== 35984+0+0 (secure 0 0 0) 0x7f1040025980 con 0x7f10500681e0 2024-02-19T21:56:19.029 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:19.028+0000 7f103effd700 1 -- 172.21.15.53:0/875972132 >> [v2:172.21.15.53:6800/1863695699,v1:172.21.15.53:6801/1863695699] conn(0x7f1034038720 msgr2=0x7f103403abe0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-19T21:56:19.029 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:19.028+0000 7f103effd700 1 --2- 172.21.15.53:0/875972132 >> [v2:172.21.15.53:6800/1863695699,v1:172.21.15.53:6801/1863695699] conn(0x7f1034038720 0x7f103403abe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-19T21:56:20.031 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:20.030+0000 7f103effd700 1 -- 172.21.15.53:0/875972132 <== mon.0 v2:172.21.15.53:3300/0 8 ==== mgrmap(e 11) v1 ==== 36054+0+0 (secure 0 0 0) 0x7f104003d590 con 0x7f10500681e0 2024-02-19T21:56:20.031 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:20.030+0000 7f103effd700 1 --2- 172.21.15.53:0/875972132 >> [v2:172.21.15.53:6800/2447819309,v1:172.21.15.53:6801/2447819309] conn(0x7f103403c540 0x7f103403e930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:56:20.031 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:20.030+0000 7f103effd700 1 -- 172.21.15.53:0/875972132 --> [v2:172.21.15.53:6800/2447819309,v1:172.21.15.53:6801/2447819309] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7f1050116760 con 0x7f103403c540 2024-02-19T21:56:20.031 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:20.030+0000 7f104dd9b700 1 --2- 172.21.15.53:0/875972132 >> [v2:172.21.15.53:6800/2447819309,v1:172.21.15.53:6801/2447819309] conn(0x7f103403c540 0x7f103403e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:56:20.031 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:20.031+0000 7f104dd9b700 1 --2- 172.21.15.53:0/875972132 >> [v2:172.21.15.53:6800/2447819309,v1:172.21.15.53:6801/2447819309] conn(0x7f103403c540 0x7f103403e930 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f1044006fd0 tx=0x7f1044006db0).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:56:20.035 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:20.035+0000 7f103effd700 1 -- 172.21.15.53:0/875972132 <== mgr.14142 v2:172.21.15.53:6800/2447819309 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+31 (secure 0 0 0) 0x7f1050116760 con 0x7f103403c540 2024-02-19T21:56:20.035 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: Scheduled prometheus update... 2024-02-19T21:56:20.038 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:20.037+0000 7f104f59e700 1 -- 172.21.15.53:0/875972132 >> [v2:172.21.15.53:6800/2447819309,v1:172.21.15.53:6801/2447819309] conn(0x7f103403c540 msgr2=0x7f103403e930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:56:20.038 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:20.037+0000 7f104f59e700 1 --2- 172.21.15.53:0/875972132 >> [v2:172.21.15.53:6800/2447819309,v1:172.21.15.53:6801/2447819309] conn(0x7f103403c540 0x7f103403e930 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f1044006fd0 tx=0x7f1044006db0).stop 2024-02-19T21:56:20.039 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:20.038+0000 7f104f59e700 1 -- 172.21.15.53:0/875972132 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f10500681e0 msgr2=0x7f1050115380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:56:20.039 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:20.038+0000 7f104f59e700 1 --2- 172.21.15.53:0/875972132 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f10500681e0 0x7f1050115380 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f104000b190 tx=0x7f1040006ee0).stop 2024-02-19T21:56:20.039 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:20.038+0000 7f104f59e700 1 -- 172.21.15.53:0/875972132 shutdown_connections 2024-02-19T21:56:20.039 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:20.038+0000 7f104f59e700 1 --2- 172.21.15.53:0/875972132 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f10500681e0 0x7f1050115380 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:56:20.039 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:20.038+0000 7f104f59e700 1 --2- 172.21.15.53:0/875972132 >> [v2:172.21.15.53:6800/2447819309,v1:172.21.15.53:6801/2447819309] conn(0x7f103403c540 0x7f103403e930 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:56:20.039 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:20.038+0000 7f104f59e700 1 --2- 172.21.15.53:0/875972132 >> [v2:172.21.15.53:6800/1863695699,v1:172.21.15.53:6801/1863695699] conn(0x7f1034038720 0x7f103403abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-19T21:56:20.039 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:20.038+0000 7f104f59e700 1 -- 172.21.15.53:0/875972132 >> 172.21.15.53:0/875972132 conn(0x7f1050063ba0 msgr2=0x7f1050066450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:56:20.039 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:20.038+0000 7f104f59e700 1 -- 172.21.15.53:0/875972132 shutdown_connections 2024-02-19T21:56:20.039 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:20.038+0000 7f104f59e700 1 -- 172.21.15.53:0/875972132 wait complete. 2024-02-19T21:56:20.293 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:56:19 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T21:56:18.977851+0000 mon.smithi053 (mon.0) 62 : cluster [INF] Active manager daemon smithi053.fyhcdi restarted 2024-02-19T21:56:20.294 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:56:19 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T21:56:18.978250+0000 mon.smithi053 (mon.0) 63 : cluster [INF] Activating manager daemon smithi053.fyhcdi 2024-02-19T21:56:20.294 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:56:19 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T21:56:18.979904+0000 mon.smithi053 (mon.0) 64 : cluster [DBG] osdmap e3: 0 total, 0 up, 0 in 2024-02-19T21:56:20.294 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:56:19 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T21:56:19.030764+0000 mon.smithi053 (mon.0) 65 : cluster [DBG] mgrmap e10: smithi053.fyhcdi(active, starting, since 0.0526013s) 2024-02-19T21:56:20.294 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:56:19 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:56:19.031862+0000 mon.smithi053 (mon.0) 66 : audit [DBG] from='mgr.14142 172.21.15.53:0/2487765561' entity='mgr.smithi053.fyhcdi' cmd=[{"prefix": "mon metadata", "id": "smithi053"}]: dispatch 2024-02-19T21:56:20.294 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:56:19 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:56:19.032622+0000 mon.smithi053 (mon.0) 67 : audit [DBG] from='mgr.14142 172.21.15.53:0/2487765561' entity='mgr.smithi053.fyhcdi' cmd=[{"prefix": "mgr metadata", "who": "smithi053.fyhcdi", "id": "smithi053.fyhcdi"}]: dispatch 2024-02-19T21:56:20.294 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:56:19 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:56:19.033417+0000 mon.smithi053 (mon.0) 68 : audit [DBG] from='mgr.14142 172.21.15.53:0/2487765561' entity='mgr.smithi053.fyhcdi' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-19T21:56:20.294 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:56:19 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:56:19.033656+0000 mon.smithi053 (mon.0) 69 : audit [DBG] from='mgr.14142 172.21.15.53:0/2487765561' entity='mgr.smithi053.fyhcdi' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-19T21:56:20.294 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:56:19 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:56:19.033865+0000 mon.smithi053 (mon.0) 70 : audit [DBG] from='mgr.14142 172.21.15.53:0/2487765561' entity='mgr.smithi053.fyhcdi' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-19T21:56:20.294 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:56:19 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T21:56:19.035933+0000 mon.smithi053 (mon.0) 71 : cluster [INF] Manager daemon smithi053.fyhcdi is now available 2024-02-19T21:56:20.294 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:56:19 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:56:19.139973+0000 mon.smithi053 (mon.0) 72 : audit [DBG] from='mgr.14142 172.21.15.53:0/2487765561' entity='mgr.smithi053.fyhcdi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T21:56:20.294 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:56:19 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:56:19.141509+0000 mon.smithi053 (mon.0) 73 : audit [DBG] from='mgr.14142 172.21.15.53:0/2487765561' entity='mgr.smithi053.fyhcdi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T21:56:20.294 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:56:19 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:56:19.157399+0000 mon.smithi053 (mon.0) 74 : audit [INF] from='mgr.14142 172.21.15.53:0/2487765561' entity='mgr.smithi053.fyhcdi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi053.fyhcdi/mirror_snapshot_schedule"}]: dispatch 2024-02-19T21:56:20.295 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:56:19 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:56:19.159309+0000 mon.smithi053 (mon.0) 75 : audit [INF] from='mgr.14142 172.21.15.53:0/2487765561' entity='mgr.smithi053.fyhcdi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi053.fyhcdi/trash_purge_schedule"}]: dispatch 2024-02-19T21:56:20.567 INFO:teuthology.orchestra.run.smithi053.stderr:Deploying grafana service with default placement... 2024-02-19T21:56:20.567 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi053 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/867b6b52-cf71-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpoyd_8g9j:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp0zy4ixsn:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply grafana 2024-02-19T21:56:21.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:56:21 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T21:56:20.033355+0000 mon.smithi053 (mon.0) 76 : cluster [DBG] mgrmap e11: smithi053.fyhcdi(active, since 1.05519s) 2024-02-19T21:56:21.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:56:21 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cephadm 2024-02-19T21:56:20.034350+0000 mgr.smithi053.fyhcdi (mgr.14142) 2 : cephadm [INF] Saving service prometheus spec with placement count:1 2024-02-19T21:56:21.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:56:21 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:56:20.036564+0000 mon.smithi053 (mon.0) 77 : audit [INF] from='mgr.14142 172.21.15.53:0/2487765561' entity='mgr.smithi053.fyhcdi' 2024-02-19T21:56:21.114 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:21.113+0000 7f7cb0be4700 1 Processor -- start 2024-02-19T21:56:21.114 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:21.113+0000 7f7cb0be4700 1 -- start start 2024-02-19T21:56:21.115 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:21.113+0000 7f7cb0be4700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7cac0703b0 0x7f7cac100d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T21:56:21.115 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:21.113+0000 7f7cb0be4700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cac0fc700 con 0x7f7cac0703b0 2024-02-19T21:56:21.115 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:21.113+0000 7f7cab7fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7cac0703b0 0x7f7cac100d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:56:21.115 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:21.113+0000 7f7cab7fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7cac0703b0 0x7f7cac100d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:58294/0 (socket says 172.21.15.53:58294) 2024-02-19T21:56:21.115 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:21.113+0000 7f7cab7fe700 1 -- 172.21.15.53:0/177411909 learned_addr learned my addr 172.21.15.53:0/177411909 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-02-19T21:56:21.115 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:21.114+0000 7f7cab7fe700 1 -- 172.21.15.53:0/177411909 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cac101200 con 0x7f7cac0703b0 2024-02-19T21:56:21.115 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:21.114+0000 7f7cab7fe700 1 --2- 172.21.15.53:0/177411909 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7cac0703b0 0x7f7cac100d10 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f7c9c00b5f0 tx=0x7f7c9c006ee0).ready entity=mon.0 client_cookie=1a1b8b582444ed84 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:56:21.115 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:21.114+0000 7f7caa7fc700 1 -- 172.21.15.53:0/177411909 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7c9c00d600 con 0x7f7cac0703b0 2024-02-19T21:56:21.115 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:21.114+0000 7f7caa7fc700 1 -- 172.21.15.53:0/177411909 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f7c9c00dbf0 con 0x7f7cac0703b0 2024-02-19T21:56:21.115 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:21.114+0000 7f7caa7fc700 1 -- 172.21.15.53:0/177411909 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7c9c011cb0 con 0x7f7cac0703b0 2024-02-19T21:56:21.118 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:21.115+0000 7f7cb0be4700 1 -- 172.21.15.53:0/177411909 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7cac0703b0 msgr2=0x7f7cac100d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:56:21.118 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:21.115+0000 7f7cb0be4700 1 --2- 172.21.15.53:0/177411909 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7cac0703b0 0x7f7cac100d10 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f7c9c00b5f0 tx=0x7f7c9c006ee0).stop 2024-02-19T21:56:21.118 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:21.115+0000 7f7cb0be4700 1 -- 172.21.15.53:0/177411909 shutdown_connections 2024-02-19T21:56:21.118 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:21.115+0000 7f7cb0be4700 1 --2- 172.21.15.53:0/177411909 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7cac0703b0 0x7f7cac100d10 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:56:21.118 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:21.115+0000 7f7cb0be4700 1 -- 172.21.15.53:0/177411909 >> 172.21.15.53:0/177411909 conn(0x7f7cac0f6390 msgr2=0x7f7cac0f87b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:56:21.118 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:21.115+0000 7f7cb0be4700 1 -- 172.21.15.53:0/177411909 shutdown_connections 2024-02-19T21:56:21.118 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:21.115+0000 7f7cb0be4700 1 -- 172.21.15.53:0/177411909 wait complete. 2024-02-19T21:56:21.118 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:21.115+0000 7f7cb0be4700 1 Processor -- start 2024-02-19T21:56:21.118 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:21.116+0000 7f7cb0be4700 1 -- start start 2024-02-19T21:56:21.118 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:21.116+0000 7f7cb0be4700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7cac0703b0 0x7f7cac10cb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:56:21.118 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:21.116+0000 7f7cb0be4700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cac10d070 con 0x7f7cac0703b0 2024-02-19T21:56:21.118 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:21.116+0000 7f7cab7fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7cac0703b0 0x7f7cac10cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:56:21.118 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:21.116+0000 7f7cab7fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7cac0703b0 0x7f7cac10cb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:58298/0 (socket says 172.21.15.53:58298) 2024-02-19T21:56:21.119 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:21.116+0000 7f7cab7fe700 1 -- 172.21.15.53:0/1237841115 learned_addr learned my addr 172.21.15.53:0/1237841115 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-02-19T21:56:21.119 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:21.116+0000 7f7cab7fe700 1 -- 172.21.15.53:0/1237841115 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c9c00b200 con 0x7f7cac0703b0 2024-02-19T21:56:21.119 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:21.116+0000 7f7cab7fe700 1 --2- 172.21.15.53:0/1237841115 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7cac0703b0 0x7f7cac10cb80 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f7c9c006800 tx=0x7f7c9c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:56:21.119 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:21.116+0000 7f7ca3fff700 1 -- 172.21.15.53:0/1237841115 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7c9c01b570 con 0x7f7cac0703b0 2024-02-19T21:56:21.119 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:21.116+0000 7f7cb0be4700 1 -- 172.21.15.53:0/1237841115 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cac10d270 con 0x7f7cac0703b0 2024-02-19T21:56:21.119 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:21.117+0000 7f7ca3fff700 1 -- 172.21.15.53:0/1237841115 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f7c9c01bb60 con 0x7f7cac0703b0 2024-02-19T21:56:21.119 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:21.117+0000 7f7ca3fff700 1 -- 172.21.15.53:0/1237841115 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7c9c01a9f0 con 0x7f7cac0703b0 2024-02-19T21:56:21.119 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:21.117+0000 7f7cb0be4700 1 -- 172.21.15.53:0/1237841115 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cac10d700 con 0x7f7cac0703b0 2024-02-19T21:56:21.119 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:21.118+0000 7f7cb0be4700 1 -- 172.21.15.53:0/1237841115 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7cac045ad0 con 0x7f7cac0703b0 2024-02-19T21:56:21.120 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:21.118+0000 7f7ca3fff700 1 -- 172.21.15.53:0/1237841115 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f7c9c00f070 con 0x7f7cac0703b0 2024-02-19T21:56:21.120 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:21.118+0000 7f7ca3fff700 1 --2- 172.21.15.53:0/1237841115 >> [v2:172.21.15.53:6800/2447819309,v1:172.21.15.53:6801/2447819309] conn(0x7f7c8c02ff20 0x7f7c8c0323e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:56:21.120 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:21.118+0000 7f7ca3fff700 1 -- 172.21.15.53:0/1237841115 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f7c9c041d90 con 0x7f7cac0703b0 2024-02-19T21:56:21.120 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:21.119+0000 7f7caaffd700 1 --2- 172.21.15.53:0/1237841115 >> [v2:172.21.15.53:6800/2447819309,v1:172.21.15.53:6801/2447819309] conn(0x7f7c8c02ff20 0x7f7c8c0323e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:56:21.120 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:21.119+0000 7f7caaffd700 1 --2- 172.21.15.53:0/1237841115 >> [v2:172.21.15.53:6800/2447819309,v1:172.21.15.53:6801/2447819309] conn(0x7f7c8c02ff20 0x7f7c8c0323e0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f7c940096a0 tx=0x7f7c94006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:56:21.122 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:21.122+0000 7f7ca3fff700 1 -- 172.21.15.53:0/1237841115 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7c9c023b50 con 0x7f7cac0703b0 2024-02-19T21:56:21.311 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:21.309+0000 7f7cb0be4700 1 -- 172.21.15.53:0/1237841115 --> [v2:172.21.15.53:6800/2447819309,v1:172.21.15.53:6801/2447819309] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}) v1 -- 0x7f7cac10e020 con 0x7f7c8c02ff20 2024-02-19T21:56:21.313 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:21.313+0000 7f7ca3fff700 1 -- 172.21.15.53:0/1237841115 <== mgr.14142 v2:172.21.15.53:6800/2447819309 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7f7cac10e020 con 0x7f7c8c02ff20 2024-02-19T21:56:21.313 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: Scheduled grafana update... 2024-02-19T21:56:21.316 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:21.315+0000 7f7cb0be4700 1 -- 172.21.15.53:0/1237841115 >> [v2:172.21.15.53:6800/2447819309,v1:172.21.15.53:6801/2447819309] conn(0x7f7c8c02ff20 msgr2=0x7f7c8c0323e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:56:21.316 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:21.315+0000 7f7cb0be4700 1 --2- 172.21.15.53:0/1237841115 >> [v2:172.21.15.53:6800/2447819309,v1:172.21.15.53:6801/2447819309] conn(0x7f7c8c02ff20 0x7f7c8c0323e0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f7c940096a0 tx=0x7f7c94006b40).stop 2024-02-19T21:56:21.316 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:21.315+0000 7f7cb0be4700 1 -- 172.21.15.53:0/1237841115 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7cac0703b0 msgr2=0x7f7cac10cb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:56:21.316 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:21.315+0000 7f7cb0be4700 1 --2- 172.21.15.53:0/1237841115 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7cac0703b0 0x7f7cac10cb80 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f7c9c006800 tx=0x7f7c9c006ee0).stop 2024-02-19T21:56:21.316 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:21.315+0000 7f7cb0be4700 1 -- 172.21.15.53:0/1237841115 shutdown_connections 2024-02-19T21:56:21.316 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:21.315+0000 7f7cb0be4700 1 --2- 172.21.15.53:0/1237841115 >> [v2:172.21.15.53:6800/2447819309,v1:172.21.15.53:6801/2447819309] conn(0x7f7c8c02ff20 0x7f7c8c0323e0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:56:21.316 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:21.315+0000 7f7cb0be4700 1 --2- 172.21.15.53:0/1237841115 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7cac0703b0 0x7f7cac10cb80 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:56:21.316 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:21.315+0000 7f7cb0be4700 1 -- 172.21.15.53:0/1237841115 >> 172.21.15.53:0/1237841115 conn(0x7f7cac0f6390 msgr2=0x7f7cac0f7db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:56:21.316 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:21.315+0000 7f7cb0be4700 1 -- 172.21.15.53:0/1237841115 shutdown_connections 2024-02-19T21:56:21.316 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:21.315+0000 7f7cb0be4700 1 -- 172.21.15.53:0/1237841115 wait complete. 2024-02-19T21:56:21.857 INFO:teuthology.orchestra.run.smithi053.stderr:Deploying node-exporter service with default placement... 2024-02-19T21:56:21.857 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi053 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/867b6b52-cf71-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpoyd_8g9j:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp0zy4ixsn:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply node-exporter 2024-02-19T21:56:22.115 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:56:22 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T21:56:21.037701+0000 mon.smithi053 (mon.0) 78 : cluster [DBG] mgrmap e12: smithi053.fyhcdi(active, since 2s) 2024-02-19T21:56:22.115 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:56:22 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:56:21.311621+0000 mgr.smithi053.fyhcdi (mgr.14142) 3 : audit [DBG] from='client.14150 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T21:56:22.115 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:56:22 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cephadm 2024-02-19T21:56:21.312137+0000 mgr.smithi053.fyhcdi (mgr.14142) 4 : cephadm [INF] Saving service grafana spec with placement count:1 2024-02-19T21:56:22.115 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:56:22 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:56:21.314235+0000 mon.smithi053 (mon.0) 79 : audit [INF] from='mgr.14142 172.21.15.53:0/2487765561' entity='mgr.smithi053.fyhcdi' 2024-02-19T21:56:22.407 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:22.399+0000 7fecda573700 1 Processor -- start 2024-02-19T21:56:22.407 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:22.399+0000 7fecda573700 1 -- start start 2024-02-19T21:56:22.407 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:22.400+0000 7fecda573700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fecd40703f0 0x7fecd40fea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T21:56:22.407 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:22.400+0000 7fecda573700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecd40fef80 con 0x7fecd40703f0 2024-02-19T21:56:22.407 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:22.400+0000 7fecd3fff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fecd40703f0 0x7fecd40fea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:56:22.407 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:22.400+0000 7fecd3fff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fecd40703f0 0x7fecd40fea90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:58308/0 (socket says 172.21.15.53:58308) 2024-02-19T21:56:22.407 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:22.400+0000 7fecd3fff700 1 -- 172.21.15.53:0/3895452442 learned_addr learned my addr 172.21.15.53:0/3895452442 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-02-19T21:56:22.407 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:22.400+0000 7fecd3fff700 1 -- 172.21.15.53:0/3895452442 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecd40ff0c0 con 0x7fecd40703f0 2024-02-19T21:56:22.407 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:22.400+0000 7fecd3fff700 1 --2- 172.21.15.53:0/3895452442 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fecd40703f0 0x7fecd40fea90 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7fecc400b5f0 tx=0x7fecc4006ee0).ready entity=mon.0 client_cookie=8e5a63dc673d10cd server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:56:22.408 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:22.400+0000 7fecd2ffd700 1 -- 172.21.15.53:0/3895452442 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fecc400d600 con 0x7fecd40703f0 2024-02-19T21:56:22.408 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:22.400+0000 7fecd2ffd700 1 -- 172.21.15.53:0/3895452442 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fecc400dbf0 con 0x7fecd40703f0 2024-02-19T21:56:22.408 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:22.400+0000 7fecd2ffd700 1 -- 172.21.15.53:0/3895452442 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fecc4011ce0 con 0x7fecd40703f0 2024-02-19T21:56:22.408 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:22.401+0000 7fecda573700 1 -- 172.21.15.53:0/3895452442 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fecd40703f0 msgr2=0x7fecd40fea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:56:22.408 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:22.401+0000 7fecda573700 1 --2- 172.21.15.53:0/3895452442 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fecd40703f0 0x7fecd40fea90 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7fecc400b5f0 tx=0x7fecc4006ee0).stop 2024-02-19T21:56:22.408 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:22.401+0000 7fecda573700 1 -- 172.21.15.53:0/3895452442 shutdown_connections 2024-02-19T21:56:22.408 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:22.401+0000 7fecda573700 1 --2- 172.21.15.53:0/3895452442 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fecd40703f0 0x7fecd40fea90 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:56:22.408 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:22.401+0000 7fecda573700 1 -- 172.21.15.53:0/3895452442 >> 172.21.15.53:0/3895452442 conn(0x7fecd40f6530 msgr2=0x7fecd40f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:56:22.408 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:22.401+0000 7fecda573700 1 -- 172.21.15.53:0/3895452442 shutdown_connections 2024-02-19T21:56:22.408 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:22.401+0000 7fecda573700 1 -- 172.21.15.53:0/3895452442 wait complete. 2024-02-19T21:56:22.408 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:22.402+0000 7fecda573700 1 Processor -- start 2024-02-19T21:56:22.408 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:22.402+0000 7fecda573700 1 -- start start 2024-02-19T21:56:22.408 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:22.402+0000 7fecda573700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fecd40703f0 0x7fecd410ab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:56:22.408 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:22.402+0000 7fecda573700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecd410b040 con 0x7fecd40703f0 2024-02-19T21:56:22.409 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:22.402+0000 7fecd3fff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fecd40703f0 0x7fecd410ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:56:22.409 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:22.402+0000 7fecd3fff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fecd40703f0 0x7fecd410ab50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:58312/0 (socket says 172.21.15.53:58312) 2024-02-19T21:56:22.409 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:22.403+0000 7fecd3fff700 1 -- 172.21.15.53:0/3243597306 learned_addr learned my addr 172.21.15.53:0/3243597306 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-02-19T21:56:22.409 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:22.403+0000 7fecd3fff700 1 -- 172.21.15.53:0/3243597306 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecc400b200 con 0x7fecd40703f0 2024-02-19T21:56:22.409 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:22.403+0000 7fecd3fff700 1 --2- 172.21.15.53:0/3243597306 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fecd40703f0 0x7fecd410ab50 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7fecc4011ed0 tx=0x7fecc4006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:56:22.409 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:22.403+0000 7fecd0ff9700 1 -- 172.21.15.53:0/3243597306 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fecc401b570 con 0x7fecd40703f0 2024-02-19T21:56:22.409 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:22.403+0000 7fecd0ff9700 1 -- 172.21.15.53:0/3243597306 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fecc401bb60 con 0x7fecd40703f0 2024-02-19T21:56:22.409 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:22.403+0000 7fecd0ff9700 1 -- 172.21.15.53:0/3243597306 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fecc401aad0 con 0x7fecd40703f0 2024-02-19T21:56:22.409 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:22.403+0000 7fecda573700 1 -- 172.21.15.53:0/3243597306 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fecd410b240 con 0x7fecd40703f0 2024-02-19T21:56:22.409 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:22.403+0000 7fecda573700 1 -- 172.21.15.53:0/3243597306 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fecd410b6d0 con 0x7fecd40703f0 2024-02-19T21:56:22.409 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:22.404+0000 7fecd0ff9700 1 -- 172.21.15.53:0/3243597306 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7fecc400f070 con 0x7fecd40703f0 2024-02-19T21:56:22.409 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:22.404+0000 7fecd0ff9700 1 --2- 172.21.15.53:0/3243597306 >> [v2:172.21.15.53:6800/2447819309,v1:172.21.15.53:6801/2447819309] conn(0x7fecc002ff20 0x7fecc00323e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:56:22.409 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:22.404+0000 7fecd0ff9700 1 -- 172.21.15.53:0/3243597306 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7fecc40415c0 con 0x7fecd40703f0 2024-02-19T21:56:22.409 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:22.404+0000 7fecd37fe700 1 --2- 172.21.15.53:0/3243597306 >> [v2:172.21.15.53:6800/2447819309,v1:172.21.15.53:6801/2447819309] conn(0x7fecc002ff20 0x7fecc00323e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:56:22.409 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:22.404+0000 7fecda573700 1 -- 172.21.15.53:0/3243597306 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fecd4045ad0 con 0x7fecd40703f0 2024-02-19T21:56:22.410 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:22.405+0000 7fecd37fe700 1 --2- 172.21.15.53:0/3243597306 >> [v2:172.21.15.53:6800/2447819309,v1:172.21.15.53:6801/2447819309] conn(0x7fecc002ff20 0x7fecc00323e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7fecc80096a0 tx=0x7fecc8006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:56:22.410 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:22.409+0000 7fecd0ff9700 1 -- 172.21.15.53:0/3243597306 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fecc4016070 con 0x7fecd40703f0 2024-02-19T21:56:22.593 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:22.593+0000 7fecda573700 1 -- 172.21.15.53:0/3243597306 --> [v2:172.21.15.53:6800/2447819309,v1:172.21.15.53:6801/2447819309] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}) v1 -- 0x7fecd4104c00 con 0x7fecc002ff20 2024-02-19T21:56:22.596 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:22.596+0000 7fecd0ff9700 1 -- 172.21.15.53:0/3243597306 <== mgr.14142 v2:172.21.15.53:6800/2447819309 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7fecd4104c00 con 0x7fecc002ff20 2024-02-19T21:56:22.596 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: Scheduled node-exporter update... 2024-02-19T21:56:22.599 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:22.598+0000 7fecda573700 1 -- 172.21.15.53:0/3243597306 >> [v2:172.21.15.53:6800/2447819309,v1:172.21.15.53:6801/2447819309] conn(0x7fecc002ff20 msgr2=0x7fecc00323e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:56:22.599 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:22.598+0000 7fecda573700 1 --2- 172.21.15.53:0/3243597306 >> [v2:172.21.15.53:6800/2447819309,v1:172.21.15.53:6801/2447819309] conn(0x7fecc002ff20 0x7fecc00323e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7fecc80096a0 tx=0x7fecc8006b40).stop 2024-02-19T21:56:22.599 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:22.598+0000 7fecda573700 1 -- 172.21.15.53:0/3243597306 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fecd40703f0 msgr2=0x7fecd410ab50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:56:22.599 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:22.598+0000 7fecda573700 1 --2- 172.21.15.53:0/3243597306 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fecd40703f0 0x7fecd410ab50 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7fecc4011ed0 tx=0x7fecc4006ee0).stop 2024-02-19T21:56:22.599 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:22.598+0000 7fecda573700 1 -- 172.21.15.53:0/3243597306 shutdown_connections 2024-02-19T21:56:22.600 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:22.598+0000 7fecda573700 1 --2- 172.21.15.53:0/3243597306 >> [v2:172.21.15.53:6800/2447819309,v1:172.21.15.53:6801/2447819309] conn(0x7fecc002ff20 0x7fecc00323e0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:56:22.600 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:22.598+0000 7fecda573700 1 --2- 172.21.15.53:0/3243597306 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fecd40703f0 0x7fecd410ab50 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:56:22.600 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:22.598+0000 7fecda573700 1 -- 172.21.15.53:0/3243597306 >> 172.21.15.53:0/3243597306 conn(0x7fecd40f6530 msgr2=0x7fecd40f7ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:56:22.600 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:22.598+0000 7fecda573700 1 -- 172.21.15.53:0/3243597306 shutdown_connections 2024-02-19T21:56:22.600 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:22.598+0000 7fecda573700 1 -- 172.21.15.53:0/3243597306 wait complete. 2024-02-19T21:56:23.147 INFO:teuthology.orchestra.run.smithi053.stderr:Deploying alertmanager service with default placement... 2024-02-19T21:56:23.147 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi053 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/867b6b52-cf71-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpoyd_8g9j:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp0zy4ixsn:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply alertmanager 2024-02-19T21:56:23.644 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:23.642+0000 7f83aac27700 1 Processor -- start 2024-02-19T21:56:23.644 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:23.642+0000 7f83aac27700 1 -- start start 2024-02-19T21:56:23.644 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:23.642+0000 7f83aac27700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f83a40703d0 0x7f83a40feab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T21:56:23.644 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:23.642+0000 7f83aac27700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83a40fefa0 con 0x7f83a40703d0 2024-02-19T21:56:23.644 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:23.643+0000 7f83a89c3700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f83a40703d0 0x7f83a40feab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:56:23.644 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:23.643+0000 7f83a89c3700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f83a40703d0 0x7f83a40feab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:58314/0 (socket says 172.21.15.53:58314) 2024-02-19T21:56:23.644 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:23.643+0000 7f83a89c3700 1 -- 172.21.15.53:0/826071378 learned_addr learned my addr 172.21.15.53:0/826071378 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-02-19T21:56:23.644 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:23.643+0000 7f83a89c3700 1 -- 172.21.15.53:0/826071378 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83a40ff0e0 con 0x7f83a40703d0 2024-02-19T21:56:23.644 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:23.643+0000 7f83a89c3700 1 --2- 172.21.15.53:0/826071378 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f83a40703d0 0x7f83a40feab0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f839800b5f0 tx=0x7f8398006ee0).ready entity=mon.0 client_cookie=9f4eabafce9b1fd4 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:56:23.644 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:23.643+0000 7f83a37fe700 1 -- 172.21.15.53:0/826071378 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f839800d600 con 0x7f83a40703d0 2024-02-19T21:56:23.644 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:23.643+0000 7f83a37fe700 1 -- 172.21.15.53:0/826071378 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f839800dbf0 con 0x7f83a40703d0 2024-02-19T21:56:23.647 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:23.643+0000 7f83a37fe700 1 -- 172.21.15.53:0/826071378 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8398011cb0 con 0x7f83a40703d0 2024-02-19T21:56:23.647 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:23.644+0000 7f83aac27700 1 -- 172.21.15.53:0/826071378 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f83a40703d0 msgr2=0x7f83a40feab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:56:23.647 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:23.644+0000 7f83aac27700 1 --2- 172.21.15.53:0/826071378 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f83a40703d0 0x7f83a40feab0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f839800b5f0 tx=0x7f8398006ee0).stop 2024-02-19T21:56:23.647 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:23.644+0000 7f83aac27700 1 -- 172.21.15.53:0/826071378 shutdown_connections 2024-02-19T21:56:23.647 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:23.644+0000 7f83aac27700 1 --2- 172.21.15.53:0/826071378 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f83a40703d0 0x7f83a40feab0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:56:23.647 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:23.644+0000 7f83aac27700 1 -- 172.21.15.53:0/826071378 >> 172.21.15.53:0/826071378 conn(0x7f83a40f6530 msgr2=0x7f83a40f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:56:23.647 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:23.644+0000 7f83aac27700 1 -- 172.21.15.53:0/826071378 shutdown_connections 2024-02-19T21:56:23.647 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:23.644+0000 7f83aac27700 1 -- 172.21.15.53:0/826071378 wait complete. 2024-02-19T21:56:23.647 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:23.645+0000 7f83aac27700 1 Processor -- start 2024-02-19T21:56:23.648 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:23.645+0000 7f83aac27700 1 -- start start 2024-02-19T21:56:23.648 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:23.645+0000 7f83aac27700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f83a40703d0 0x7f83a410abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:56:23.648 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:23.645+0000 7f83aac27700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83a410b0b0 con 0x7f83a40703d0 2024-02-19T21:56:23.648 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:23.645+0000 7f83a89c3700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f83a40703d0 0x7f83a410abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:56:23.648 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:23.645+0000 7f83a89c3700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f83a40703d0 0x7f83a410abc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:58330/0 (socket says 172.21.15.53:58330) 2024-02-19T21:56:23.648 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:23.645+0000 7f83a89c3700 1 -- 172.21.15.53:0/3141702257 learned_addr learned my addr 172.21.15.53:0/3141702257 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-02-19T21:56:23.648 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:23.646+0000 7f83a89c3700 1 -- 172.21.15.53:0/3141702257 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f839800b200 con 0x7f83a40703d0 2024-02-19T21:56:23.648 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:23.646+0000 7f83a89c3700 1 --2- 172.21.15.53:0/3141702257 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f83a40703d0 0x7f83a410abc0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f8398006c70 tx=0x7f8398006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:56:23.648 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:23.646+0000 7f83a17fa700 1 -- 172.21.15.53:0/3141702257 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8398011400 con 0x7f83a40703d0 2024-02-19T21:56:23.648 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:23.646+0000 7f83a17fa700 1 -- 172.21.15.53:0/3141702257 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f839801b8a0 con 0x7f83a40703d0 2024-02-19T21:56:23.648 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:23.646+0000 7f83aac27700 1 -- 172.21.15.53:0/3141702257 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83a410b2b0 con 0x7f83a40703d0 2024-02-19T21:56:23.648 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:23.646+0000 7f83a17fa700 1 -- 172.21.15.53:0/3141702257 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f839801a7c0 con 0x7f83a40703d0 2024-02-19T21:56:23.648 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:23.646+0000 7f83aac27700 1 -- 172.21.15.53:0/3141702257 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83a410b740 con 0x7f83a40703d0 2024-02-19T21:56:23.648 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:23.647+0000 7f83a17fa700 1 -- 172.21.15.53:0/3141702257 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f839800f070 con 0x7f83a40703d0 2024-02-19T21:56:23.649 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:23.647+0000 7f83a17fa700 1 --2- 172.21.15.53:0/3141702257 >> [v2:172.21.15.53:6800/2447819309,v1:172.21.15.53:6801/2447819309] conn(0x7f838802ff20 0x7f83880323e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:56:23.649 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:23.647+0000 7f83aac27700 1 -- 172.21.15.53:0/3141702257 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83a4045ad0 con 0x7f83a40703d0 2024-02-19T21:56:23.651 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:23.647+0000 7f83a17fa700 1 -- 172.21.15.53:0/3141702257 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f83980413d0 con 0x7f83a40703d0 2024-02-19T21:56:23.651 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:23.647+0000 7f83a3fff700 1 --2- 172.21.15.53:0/3141702257 >> [v2:172.21.15.53:6800/2447819309,v1:172.21.15.53:6801/2447819309] conn(0x7f838802ff20 0x7f83880323e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:56:23.651 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:23.648+0000 7f83a3fff700 1 --2- 172.21.15.53:0/3141702257 >> [v2:172.21.15.53:6800/2447819309,v1:172.21.15.53:6801/2447819309] conn(0x7f838802ff20 0x7f83880323e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f83900096a0 tx=0x7f8390006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:56:23.651 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:23.651+0000 7f83a17fa700 1 -- 172.21.15.53:0/3141702257 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8398019af0 con 0x7f83a40703d0 2024-02-19T21:56:23.839 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:23.839+0000 7f83aac27700 1 -- 172.21.15.53:0/3141702257 --> [v2:172.21.15.53:6800/2447819309,v1:172.21.15.53:6801/2447819309] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}) v1 -- 0x7f83a410bfa0 con 0x7f838802ff20 2024-02-19T21:56:23.843 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:23.843+0000 7f83a17fa700 1 -- 172.21.15.53:0/3141702257 <== mgr.14142 v2:172.21.15.53:6800/2447819309 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+33 (secure 0 0 0) 0x7f83a410bfa0 con 0x7f838802ff20 2024-02-19T21:56:23.843 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: Scheduled alertmanager update... 2024-02-19T21:56:23.845 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:23.844+0000 7f83aac27700 1 -- 172.21.15.53:0/3141702257 >> [v2:172.21.15.53:6800/2447819309,v1:172.21.15.53:6801/2447819309] conn(0x7f838802ff20 msgr2=0x7f83880323e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:56:23.845 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:23.844+0000 7f83aac27700 1 --2- 172.21.15.53:0/3141702257 >> [v2:172.21.15.53:6800/2447819309,v1:172.21.15.53:6801/2447819309] conn(0x7f838802ff20 0x7f83880323e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f83900096a0 tx=0x7f8390006b40).stop 2024-02-19T21:56:23.845 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:23.844+0000 7f83aac27700 1 -- 172.21.15.53:0/3141702257 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f83a40703d0 msgr2=0x7f83a410abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:56:23.845 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:23.844+0000 7f83aac27700 1 --2- 172.21.15.53:0/3141702257 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f83a40703d0 0x7f83a410abc0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f8398006c70 tx=0x7f8398006ee0).stop 2024-02-19T21:56:23.846 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:23.845+0000 7f83aac27700 1 -- 172.21.15.53:0/3141702257 shutdown_connections 2024-02-19T21:56:23.846 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:23.845+0000 7f83aac27700 1 --2- 172.21.15.53:0/3141702257 >> [v2:172.21.15.53:6800/2447819309,v1:172.21.15.53:6801/2447819309] conn(0x7f838802ff20 0x7f83880323e0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:56:23.846 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:23.845+0000 7f83aac27700 1 --2- 172.21.15.53:0/3141702257 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f83a40703d0 0x7f83a410abc0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:56:23.846 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:23.845+0000 7f83aac27700 1 -- 172.21.15.53:0/3141702257 >> 172.21.15.53:0/3141702257 conn(0x7f83a40f6530 msgr2=0x7f83a40f7f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:56:23.846 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:23.845+0000 7f83aac27700 1 -- 172.21.15.53:0/3141702257 shutdown_connections 2024-02-19T21:56:23.846 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:23.845+0000 7f83aac27700 1 -- 172.21.15.53:0/3141702257 wait complete. 2024-02-19T21:56:23.865 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:56:23 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:56:22.594885+0000 mgr.smithi053.fyhcdi (mgr.14142) 5 : audit [DBG] from='client.14152 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T21:56:23.865 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:56:23 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cephadm 2024-02-19T21:56:22.595536+0000 mgr.smithi053.fyhcdi (mgr.14142) 6 : cephadm [INF] Saving service node-exporter spec with placement * 2024-02-19T21:56:23.865 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:56:23 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:56:22.597640+0000 mon.smithi053 (mon.0) 80 : audit [INF] from='mgr.14142 172.21.15.53:0/2487765561' entity='mgr.smithi053.fyhcdi' 2024-02-19T21:56:24.530 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi053 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/867b6b52-cf71-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpoyd_8g9j:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp0zy4ixsn:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mgr mgr/cephadm/container_init True --force 2024-02-19T21:56:25.068 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:25.034+0000 7f049144e700 1 Processor -- start 2024-02-19T21:56:25.068 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:25.034+0000 7f049144e700 1 -- start start 2024-02-19T21:56:25.069 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:25.035+0000 7f049144e700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f048c0fbea0 0x7f048c0fc2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T21:56:25.069 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:25.035+0000 7f049144e700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f048c0fc7b0 con 0x7f048c0fbea0 2024-02-19T21:56:25.069 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:25.035+0000 7f048affd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f048c0fbea0 0x7f048c0fc2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:56:25.069 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:25.035+0000 7f048affd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f048c0fbea0 0x7f048c0fc2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:58338/0 (socket says 172.21.15.53:58338) 2024-02-19T21:56:25.069 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:25.035+0000 7f048affd700 1 -- 172.21.15.53:0/1119252802 learned_addr learned my addr 172.21.15.53:0/1119252802 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-02-19T21:56:25.069 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:25.035+0000 7f048affd700 1 -- 172.21.15.53:0/1119252802 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f048c0fc8f0 con 0x7f048c0fbea0 2024-02-19T21:56:25.069 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:25.035+0000 7f048affd700 1 --2- 172.21.15.53:0/1119252802 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f048c0fbea0 0x7f048c0fc2c0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f048000b5f0 tx=0x7f0480006ee0).ready entity=mon.0 client_cookie=efb66f9025d6b7a5 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:56:25.069 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:25.035+0000 7f0489ffb700 1 -- 172.21.15.53:0/1119252802 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f048000d600 con 0x7f048c0fbea0 2024-02-19T21:56:25.070 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:25.035+0000 7f0489ffb700 1 -- 172.21.15.53:0/1119252802 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f048000dbf0 con 0x7f048c0fbea0 2024-02-19T21:56:25.070 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:25.035+0000 7f0489ffb700 1 -- 172.21.15.53:0/1119252802 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0480011cb0 con 0x7f048c0fbea0 2024-02-19T21:56:25.070 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:25.036+0000 7f049144e700 1 -- 172.21.15.53:0/1119252802 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f048c0fbea0 msgr2=0x7f048c0fc2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:56:25.070 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:25.036+0000 7f049144e700 1 --2- 172.21.15.53:0/1119252802 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f048c0fbea0 0x7f048c0fc2c0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f048000b5f0 tx=0x7f0480006ee0).stop 2024-02-19T21:56:25.070 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:25.036+0000 7f049144e700 1 -- 172.21.15.53:0/1119252802 shutdown_connections 2024-02-19T21:56:25.070 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:25.036+0000 7f049144e700 1 --2- 172.21.15.53:0/1119252802 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f048c0fbea0 0x7f048c0fc2c0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:56:25.070 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:25.036+0000 7f049144e700 1 -- 172.21.15.53:0/1119252802 >> 172.21.15.53:0/1119252802 conn(0x7f048c0f76a0 msgr2=0x7f048c0f9ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:56:25.070 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:25.036+0000 7f049144e700 1 -- 172.21.15.53:0/1119252802 shutdown_connections 2024-02-19T21:56:25.070 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:25.036+0000 7f049144e700 1 -- 172.21.15.53:0/1119252802 wait complete. 2024-02-19T21:56:25.070 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:25.037+0000 7f049144e700 1 Processor -- start 2024-02-19T21:56:25.070 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:25.037+0000 7f049144e700 1 -- start start 2024-02-19T21:56:25.070 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:25.037+0000 7f049144e700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f048c0fbea0 0x7f048c10abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:56:25.070 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:25.037+0000 7f049144e700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f048c10b0c0 con 0x7f048c0fbea0 2024-02-19T21:56:25.070 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:25.037+0000 7f048affd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f048c0fbea0 0x7f048c10abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:56:25.071 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:25.037+0000 7f048affd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f048c0fbea0 0x7f048c10abd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:58352/0 (socket says 172.21.15.53:58352) 2024-02-19T21:56:25.071 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:25.037+0000 7f048affd700 1 -- 172.21.15.53:0/3627245745 learned_addr learned my addr 172.21.15.53:0/3627245745 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-02-19T21:56:25.071 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:25.037+0000 7f048affd700 1 -- 172.21.15.53:0/3627245745 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f048000b200 con 0x7f048c0fbea0 2024-02-19T21:56:25.071 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:25.038+0000 7f048affd700 1 --2- 172.21.15.53:0/3627245745 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f048c0fbea0 0x7f048c10abd0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f0480002f40 tx=0x7f0480006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:56:25.071 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:25.038+0000 7f04737fe700 1 -- 172.21.15.53:0/3627245745 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f048001b410 con 0x7f048c0fbea0 2024-02-19T21:56:25.071 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:25.038+0000 7f04737fe700 1 -- 172.21.15.53:0/3627245745 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f048001ba00 con 0x7f048c0fbea0 2024-02-19T21:56:25.071 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:25.038+0000 7f04737fe700 1 -- 172.21.15.53:0/3627245745 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f048001a980 con 0x7f048c0fbea0 2024-02-19T21:56:25.071 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:25.038+0000 7f049144e700 1 -- 172.21.15.53:0/3627245745 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f048c10b2c0 con 0x7f048c0fbea0 2024-02-19T21:56:25.071 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:25.038+0000 7f049144e700 1 -- 172.21.15.53:0/3627245745 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f048c10b750 con 0x7f048c0fbea0 2024-02-19T21:56:25.071 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:25.039+0000 7f049144e700 1 -- 172.21.15.53:0/3627245745 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f048c045ad0 con 0x7f048c0fbea0 2024-02-19T21:56:25.071 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:25.039+0000 7f04737fe700 1 -- 172.21.15.53:0/3627245745 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f048000f070 con 0x7f048c0fbea0 2024-02-19T21:56:25.071 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:25.039+0000 7f04737fe700 1 --2- 172.21.15.53:0/3627245745 >> [v2:172.21.15.53:6800/2447819309,v1:172.21.15.53:6801/2447819309] conn(0x7f047802ff20 0x7f04780323e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:56:25.071 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:25.039+0000 7f04737fe700 1 -- 172.21.15.53:0/3627245745 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f0480041d60 con 0x7f048c0fbea0 2024-02-19T21:56:25.071 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:25.040+0000 7f048a7fc700 1 --2- 172.21.15.53:0/3627245745 >> [v2:172.21.15.53:6800/2447819309,v1:172.21.15.53:6801/2447819309] conn(0x7f047802ff20 0x7f04780323e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:56:25.072 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:25.040+0000 7f048a7fc700 1 --2- 172.21.15.53:0/3627245745 >> [v2:172.21.15.53:6800/2447819309,v1:172.21.15.53:6801/2447819309] conn(0x7f047802ff20 0x7f04780323e0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f047c0096a0 tx=0x7f047c006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:56:25.072 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:25.043+0000 7f04737fe700 1 -- 172.21.15.53:0/3627245745 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0480019300 con 0x7f048c0fbea0 2024-02-19T21:56:25.201 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:25.201+0000 7f049144e700 1 -- 172.21.15.53:0/3627245745 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7f048c0fdc00 con 0x7f048c0fbea0 2024-02-19T21:56:25.204 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:25.204+0000 7f04737fe700 1 -- 172.21.15.53:0/3627245745 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v8)=0 v8) v1 ==== 142+0+0 (secure 0 0 0) 0x7f0480016030 con 0x7f048c0fbea0 2024-02-19T21:56:25.206 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:25.205+0000 7f049144e700 1 -- 172.21.15.53:0/3627245745 >> [v2:172.21.15.53:6800/2447819309,v1:172.21.15.53:6801/2447819309] conn(0x7f047802ff20 msgr2=0x7f04780323e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:56:25.206 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:25.205+0000 7f049144e700 1 --2- 172.21.15.53:0/3627245745 >> [v2:172.21.15.53:6800/2447819309,v1:172.21.15.53:6801/2447819309] conn(0x7f047802ff20 0x7f04780323e0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f047c0096a0 tx=0x7f047c006b40).stop 2024-02-19T21:56:25.207 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:25.205+0000 7f049144e700 1 -- 172.21.15.53:0/3627245745 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f048c0fbea0 msgr2=0x7f048c10abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:56:25.207 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:25.205+0000 7f049144e700 1 --2- 172.21.15.53:0/3627245745 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f048c0fbea0 0x7f048c10abd0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f0480002f40 tx=0x7f0480006ee0).stop 2024-02-19T21:56:25.207 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:25.206+0000 7f049144e700 1 -- 172.21.15.53:0/3627245745 shutdown_connections 2024-02-19T21:56:25.207 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:25.206+0000 7f049144e700 1 --2- 172.21.15.53:0/3627245745 >> [v2:172.21.15.53:6800/2447819309,v1:172.21.15.53:6801/2447819309] conn(0x7f047802ff20 0x7f04780323e0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:56:25.207 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:25.206+0000 7f049144e700 1 --2- 172.21.15.53:0/3627245745 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f048c0fbea0 0x7f048c10abd0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:56:25.207 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:25.206+0000 7f049144e700 1 -- 172.21.15.53:0/3627245745 >> 172.21.15.53:0/3627245745 conn(0x7f048c0f76a0 msgr2=0x7f048c0f9070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:56:25.207 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:25.206+0000 7f049144e700 1 -- 172.21.15.53:0/3627245745 shutdown_connections 2024-02-19T21:56:25.207 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:25.206+0000 7f049144e700 1 -- 172.21.15.53:0/3627245745 wait complete. 2024-02-19T21:56:25.228 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:56:24 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:56:23.841388+0000 mgr.smithi053.fyhcdi (mgr.14142) 7 : audit [DBG] from='client.14154 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T21:56:25.228 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:56:24 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cephadm 2024-02-19T21:56:23.841891+0000 mgr.smithi053.fyhcdi (mgr.14142) 8 : cephadm [INF] Saving service alertmanager spec with placement count:1 2024-02-19T21:56:25.229 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:56:24 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:56:23.844003+0000 mon.smithi053 (mon.0) 81 : audit [INF] from='mgr.14142 172.21.15.53:0/2487765561' entity='mgr.smithi053.fyhcdi' 2024-02-19T21:56:25.778 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi053 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/867b6b52-cf71-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpoyd_8g9j:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp0zy4ixsn:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mgr mgr/dashboard/ssl_server_port 8443 2024-02-19T21:56:26.293 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:56:26 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:56:25.205266+0000 mon.smithi053 (mon.0) 82 : audit [INF] from='client.? 172.21.15.53:0/3627245745' entity='client.admin' 2024-02-19T21:56:26.366 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:26.364+0000 7fe23e666700 1 Processor -- start 2024-02-19T21:56:26.366 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:26.365+0000 7fe23e666700 1 -- start start 2024-02-19T21:56:26.366 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:26.365+0000 7fe23e666700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe2380703f0 0x7fe238100e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T21:56:26.366 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:26.365+0000 7fe23e666700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2380fc880 con 0x7fe2380703f0 2024-02-19T21:56:26.366 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:26.365+0000 7fe237fff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe2380703f0 0x7fe238100e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:56:26.366 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:26.365+0000 7fe237fff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe2380703f0 0x7fe238100e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:58356/0 (socket says 172.21.15.53:58356) 2024-02-19T21:56:26.366 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:26.365+0000 7fe237fff700 1 -- 172.21.15.53:0/1601267499 learned_addr learned my addr 172.21.15.53:0/1601267499 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-02-19T21:56:26.366 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:26.365+0000 7fe237fff700 1 -- 172.21.15.53:0/1601267499 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe238101380 con 0x7fe2380703f0 2024-02-19T21:56:26.367 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:26.366+0000 7fe237fff700 1 --2- 172.21.15.53:0/1601267499 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe2380703f0 0x7fe238100e90 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7fe22800b5f0 tx=0x7fe228006ee0).ready entity=mon.0 client_cookie=ab97f001d9bf23c8 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:56:26.367 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:26.366+0000 7fe236ffd700 1 -- 172.21.15.53:0/1601267499 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe22800d600 con 0x7fe2380703f0 2024-02-19T21:56:26.367 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:26.366+0000 7fe236ffd700 1 -- 172.21.15.53:0/1601267499 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fe22800dbf0 con 0x7fe2380703f0 2024-02-19T21:56:26.367 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:26.366+0000 7fe236ffd700 1 -- 172.21.15.53:0/1601267499 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe228011ce0 con 0x7fe2380703f0 2024-02-19T21:56:26.370 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:26.366+0000 7fe23e666700 1 -- 172.21.15.53:0/1601267499 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe2380703f0 msgr2=0x7fe238100e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:56:26.370 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:26.366+0000 7fe23e666700 1 --2- 172.21.15.53:0/1601267499 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe2380703f0 0x7fe238100e90 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7fe22800b5f0 tx=0x7fe228006ee0).stop 2024-02-19T21:56:26.370 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:26.366+0000 7fe23e666700 1 -- 172.21.15.53:0/1601267499 shutdown_connections 2024-02-19T21:56:26.370 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:26.366+0000 7fe23e666700 1 --2- 172.21.15.53:0/1601267499 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe2380703f0 0x7fe238100e90 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:56:26.370 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:26.366+0000 7fe23e666700 1 -- 172.21.15.53:0/1601267499 >> 172.21.15.53:0/1601267499 conn(0x7fe2380f6530 msgr2=0x7fe2380f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:56:26.370 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:26.366+0000 7fe23e666700 1 -- 172.21.15.53:0/1601267499 shutdown_connections 2024-02-19T21:56:26.370 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:26.366+0000 7fe23e666700 1 -- 172.21.15.53:0/1601267499 wait complete. 2024-02-19T21:56:26.370 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:26.367+0000 7fe23e666700 1 Processor -- start 2024-02-19T21:56:26.370 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:26.367+0000 7fe23e666700 1 -- start start 2024-02-19T21:56:26.370 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:26.367+0000 7fe23e666700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe2380703f0 0x7fe23810cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:56:26.370 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:26.367+0000 7fe23e666700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe23810d280 con 0x7fe2380703f0 2024-02-19T21:56:26.370 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:26.367+0000 7fe237fff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe2380703f0 0x7fe23810cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:56:26.370 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:26.367+0000 7fe237fff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe2380703f0 0x7fe23810cd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:58358/0 (socket says 172.21.15.53:58358) 2024-02-19T21:56:26.371 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:26.367+0000 7fe237fff700 1 -- 172.21.15.53:0/4275896369 learned_addr learned my addr 172.21.15.53:0/4275896369 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-02-19T21:56:26.371 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:26.368+0000 7fe237fff700 1 -- 172.21.15.53:0/4275896369 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe22800b200 con 0x7fe2380703f0 2024-02-19T21:56:26.371 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:26.368+0000 7fe237fff700 1 --2- 172.21.15.53:0/4275896369 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe2380703f0 0x7fe23810cd90 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7fe228011ed0 tx=0x7fe228006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:56:26.371 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:26.368+0000 7fe234ff9700 1 -- 172.21.15.53:0/4275896369 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe22801b570 con 0x7fe2380703f0 2024-02-19T21:56:26.371 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:26.368+0000 7fe23e666700 1 -- 172.21.15.53:0/4275896369 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe23810d480 con 0x7fe2380703f0 2024-02-19T21:56:26.371 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:26.368+0000 7fe234ff9700 1 -- 172.21.15.53:0/4275896369 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fe22801bb60 con 0x7fe2380703f0 2024-02-19T21:56:26.371 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:26.368+0000 7fe234ff9700 1 -- 172.21.15.53:0/4275896369 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe22801aad0 con 0x7fe2380703f0 2024-02-19T21:56:26.371 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:26.368+0000 7fe23e666700 1 -- 172.21.15.53:0/4275896369 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe23810e090 con 0x7fe2380703f0 2024-02-19T21:56:26.371 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:26.369+0000 7fe234ff9700 1 -- 172.21.15.53:0/4275896369 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7fe22800f070 con 0x7fe2380703f0 2024-02-19T21:56:26.371 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:26.369+0000 7fe234ff9700 1 --2- 172.21.15.53:0/4275896369 >> [v2:172.21.15.53:6800/2447819309,v1:172.21.15.53:6801/2447819309] conn(0x7fe22402ff70 0x7fe224032430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:56:26.371 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:26.369+0000 7fe23e666700 1 -- 172.21.15.53:0/4275896369 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe238059100 con 0x7fe2380703f0 2024-02-19T21:56:26.371 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:26.369+0000 7fe234ff9700 1 -- 172.21.15.53:0/4275896369 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7fe2280414d0 con 0x7fe2380703f0 2024-02-19T21:56:26.371 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:26.369+0000 7fe2377fe700 1 --2- 172.21.15.53:0/4275896369 >> [v2:172.21.15.53:6800/2447819309,v1:172.21.15.53:6801/2447819309] conn(0x7fe22402ff70 0x7fe224032430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:56:26.371 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:26.370+0000 7fe2377fe700 1 --2- 172.21.15.53:0/4275896369 >> [v2:172.21.15.53:6800/2447819309,v1:172.21.15.53:6801/2447819309] conn(0x7fe22402ff70 0x7fe224032430 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7fe22c0096a0 tx=0x7fe22c006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:56:26.373 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:26.373+0000 7fe234ff9700 1 -- 172.21.15.53:0/4275896369 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe228016070 con 0x7fe2380703f0 2024-02-19T21:56:26.551 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:26.551+0000 7fe23e666700 1 -- 172.21.15.53:0/4275896369 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7fe23810eec0 con 0x7fe2380703f0 2024-02-19T21:56:26.553 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:26.554+0000 7fe234ff9700 1 -- 172.21.15.53:0/4275896369 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v9)=0 v9) v1 ==== 130+0+0 (secure 0 0 0) 0x7fe228010360 con 0x7fe2380703f0 2024-02-19T21:56:26.556 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:26.555+0000 7fe23e666700 1 -- 172.21.15.53:0/4275896369 >> [v2:172.21.15.53:6800/2447819309,v1:172.21.15.53:6801/2447819309] conn(0x7fe22402ff70 msgr2=0x7fe224032430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:56:26.556 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:26.555+0000 7fe23e666700 1 --2- 172.21.15.53:0/4275896369 >> [v2:172.21.15.53:6800/2447819309,v1:172.21.15.53:6801/2447819309] conn(0x7fe22402ff70 0x7fe224032430 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7fe22c0096a0 tx=0x7fe22c006b40).stop 2024-02-19T21:56:26.556 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:26.555+0000 7fe23e666700 1 -- 172.21.15.53:0/4275896369 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe2380703f0 msgr2=0x7fe23810cd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:56:26.556 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:26.555+0000 7fe23e666700 1 --2- 172.21.15.53:0/4275896369 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe2380703f0 0x7fe23810cd90 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7fe228011ed0 tx=0x7fe228006ee0).stop 2024-02-19T21:56:26.557 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:26.555+0000 7fe23e666700 1 -- 172.21.15.53:0/4275896369 shutdown_connections 2024-02-19T21:56:26.557 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:26.555+0000 7fe23e666700 1 --2- 172.21.15.53:0/4275896369 >> [v2:172.21.15.53:6800/2447819309,v1:172.21.15.53:6801/2447819309] conn(0x7fe22402ff70 0x7fe224032430 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:56:26.557 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:26.555+0000 7fe23e666700 1 --2- 172.21.15.53:0/4275896369 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe2380703f0 0x7fe23810cd90 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:56:26.557 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:26.555+0000 7fe23e666700 1 -- 172.21.15.53:0/4275896369 >> 172.21.15.53:0/4275896369 conn(0x7fe2380f6530 msgr2=0x7fe2380f7ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:56:26.557 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:26.556+0000 7fe23e666700 1 -- 172.21.15.53:0/4275896369 shutdown_connections 2024-02-19T21:56:26.557 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:26.556+0000 7fe23e666700 1 -- 172.21.15.53:0/4275896369 wait complete. 2024-02-19T21:56:27.299 INFO:teuthology.orchestra.run.smithi053.stderr:Enabling the dashboard module... 2024-02-19T21:56:27.299 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi053 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/867b6b52-cf71-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpoyd_8g9j:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp0zy4ixsn:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable dashboard 2024-02-19T21:56:27.869 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:56:27 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:56:26.555059+0000 mon.smithi053 (mon.0) 83 : audit [INF] from='client.? 172.21.15.53:0/4275896369' entity='client.admin' 2024-02-19T21:56:28.422 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:28.421+0000 7f62ea7fc700 1 Processor -- start 2024-02-19T21:56:28.422 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:28.421+0000 7f62ea7fc700 1 -- start start 2024-02-19T21:56:28.422 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:28.422+0000 7f62ea7fc700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f62e4071cb0 0x7f62e4070250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T21:56:28.423 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:28.422+0000 7f62ea7fc700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62e40720d0 con 0x7f62e4071cb0 2024-02-19T21:56:28.423 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:28.422+0000 7f62e97fa700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f62e4071cb0 0x7f62e4070250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:56:28.423 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:28.422+0000 7f62e97fa700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f62e4071cb0 0x7f62e4070250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:38276/0 (socket says 172.21.15.53:38276) 2024-02-19T21:56:28.423 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:28.422+0000 7f62e97fa700 1 -- 172.21.15.53:0/1118099937 learned_addr learned my addr 172.21.15.53:0/1118099937 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-02-19T21:56:28.424 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:28.422+0000 7f62e97fa700 1 -- 172.21.15.53:0/1118099937 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62e4070740 con 0x7f62e4071cb0 2024-02-19T21:56:28.424 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:28.422+0000 7f62e97fa700 1 --2- 172.21.15.53:0/1118099937 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f62e4071cb0 0x7f62e4070250 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f62e000b5f0 tx=0x7f62e0006ee0).ready entity=mon.0 client_cookie=72174aa565f0019d server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:56:28.425 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:28.423+0000 7f62d3fff700 1 -- 172.21.15.53:0/1118099937 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f62e000d600 con 0x7f62e4071cb0 2024-02-19T21:56:28.425 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:28.423+0000 7f62d3fff700 1 -- 172.21.15.53:0/1118099937 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f62e000dc30 con 0x7f62e4071cb0 2024-02-19T21:56:28.425 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:28.423+0000 7f62d3fff700 1 -- 172.21.15.53:0/1118099937 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f62e0011d30 con 0x7f62e4071cb0 2024-02-19T21:56:28.425 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:28.423+0000 7f62ea7fc700 1 -- 172.21.15.53:0/1118099937 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f62e4071cb0 msgr2=0x7f62e4070250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:56:28.425 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:28.423+0000 7f62ea7fc700 1 --2- 172.21.15.53:0/1118099937 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f62e4071cb0 0x7f62e4070250 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f62e000b5f0 tx=0x7f62e0006ee0).stop 2024-02-19T21:56:28.425 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:28.423+0000 7f62ea7fc700 1 -- 172.21.15.53:0/1118099937 shutdown_connections 2024-02-19T21:56:28.425 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:28.423+0000 7f62ea7fc700 1 --2- 172.21.15.53:0/1118099937 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f62e4071cb0 0x7f62e4070250 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:56:28.425 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:28.423+0000 7f62ea7fc700 1 -- 172.21.15.53:0/1118099937 >> 172.21.15.53:0/1118099937 conn(0x7f62e40f7480 msgr2=0x7f62e40f98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:56:28.425 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:28.424+0000 7f62ea7fc700 1 -- 172.21.15.53:0/1118099937 shutdown_connections 2024-02-19T21:56:28.425 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:28.424+0000 7f62ea7fc700 1 -- 172.21.15.53:0/1118099937 wait complete. 2024-02-19T21:56:28.425 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:28.424+0000 7f62ea7fc700 1 Processor -- start 2024-02-19T21:56:28.425 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:28.424+0000 7f62ea7fc700 1 -- start start 2024-02-19T21:56:28.425 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:28.424+0000 7f62ea7fc700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f62e4071cb0 0x7f62e410cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:56:28.425 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:28.424+0000 7f62ea7fc700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62e410d200 con 0x7f62e4071cb0 2024-02-19T21:56:28.427 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:28.425+0000 7f62e97fa700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f62e4071cb0 0x7f62e410cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:56:28.427 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:28.425+0000 7f62e97fa700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f62e4071cb0 0x7f62e410cd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:38292/0 (socket says 172.21.15.53:38292) 2024-02-19T21:56:28.427 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:28.425+0000 7f62e97fa700 1 -- 172.21.15.53:0/2354273020 learned_addr learned my addr 172.21.15.53:0/2354273020 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-02-19T21:56:28.427 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:28.425+0000 7f62e97fa700 1 -- 172.21.15.53:0/2354273020 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62e000b200 con 0x7f62e4071cb0 2024-02-19T21:56:28.427 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:28.425+0000 7f62e97fa700 1 --2- 172.21.15.53:0/2354273020 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f62e4071cb0 0x7f62e410cd10 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f62e0006800 tx=0x7f62e0006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:56:28.427 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:28.425+0000 7f62d1ffb700 1 -- 172.21.15.53:0/2354273020 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f62e000d960 con 0x7f62e4071cb0 2024-02-19T21:56:28.427 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:28.425+0000 7f62d1ffb700 1 -- 172.21.15.53:0/2354273020 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f62e000dac0 con 0x7f62e4071cb0 2024-02-19T21:56:28.427 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:28.425+0000 7f62ea7fc700 1 -- 172.21.15.53:0/2354273020 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62e410d400 con 0x7f62e4071cb0 2024-02-19T21:56:28.427 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:28.425+0000 7f62d1ffb700 1 -- 172.21.15.53:0/2354273020 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f62e001a820 con 0x7f62e4071cb0 2024-02-19T21:56:28.428 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:28.425+0000 7f62ea7fc700 1 -- 172.21.15.53:0/2354273020 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62e410dfb0 con 0x7f62e4071cb0 2024-02-19T21:56:28.428 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:28.426+0000 7f62d1ffb700 1 -- 172.21.15.53:0/2354273020 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f62e000f070 con 0x7f62e4071cb0 2024-02-19T21:56:28.428 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:28.426+0000 7f62ea7fc700 1 -- 172.21.15.53:0/2354273020 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62e4059100 con 0x7f62e4071cb0 2024-02-19T21:56:28.428 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:28.426+0000 7f62d1ffb700 1 --2- 172.21.15.53:0/2354273020 >> [v2:172.21.15.53:6800/2447819309,v1:172.21.15.53:6801/2447819309] conn(0x7f62cc02ff20 0x7f62cc0323e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:56:28.428 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:28.426+0000 7f62d1ffb700 1 -- 172.21.15.53:0/2354273020 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f62e0041370 con 0x7f62e4071cb0 2024-02-19T21:56:28.428 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:28.427+0000 7f62e8ff9700 1 --2- 172.21.15.53:0/2354273020 >> [v2:172.21.15.53:6800/2447819309,v1:172.21.15.53:6801/2447819309] conn(0x7f62cc02ff20 0x7f62cc0323e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:56:28.428 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:28.427+0000 7f62e8ff9700 1 --2- 172.21.15.53:0/2354273020 >> [v2:172.21.15.53:6800/2447819309,v1:172.21.15.53:6801/2447819309] conn(0x7f62cc02ff20 0x7f62cc0323e0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f62d40096a0 tx=0x7f62d4006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:56:28.432 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:28.432+0000 7f62d1ffb700 1 -- 172.21.15.53:0/2354273020 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f62e0022d20 con 0x7f62e4071cb0 2024-02-19T21:56:28.634 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:28.634+0000 7f62ea7fc700 1 -- 172.21.15.53:0/2354273020 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7f62e4045ad0 con 0x7f62e4071cb0 2024-02-19T21:56:29.293 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:56:29 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:56:28.156883+0000 mon.smithi053 (mon.0) 84 : audit [INF] from='mgr.14142 172.21.15.53:0/2487765561' entity='mgr.smithi053.fyhcdi' 2024-02-19T21:56:29.294 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:56:29 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:56:28.473308+0000 mon.smithi053 (mon.0) 85 : audit [INF] from='mgr.14142 172.21.15.53:0/2487765561' entity='mgr.smithi053.fyhcdi' 2024-02-19T21:56:29.294 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:56:29 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:56:28.635452+0000 mon.smithi053 (mon.0) 86 : audit [INF] from='client.? 172.21.15.53:0/2354273020' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-02-19T21:56:29.473 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:29.472+0000 7f62d1ffb700 1 -- 172.21.15.53:0/2354273020 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mgrmap(e 13) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f62e001bce0 con 0x7f62e4071cb0 2024-02-19T21:56:29.473 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:29.474+0000 7f62d1ffb700 1 -- 172.21.15.53:0/2354273020 <== mon.0 v2:172.21.15.53:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v13) v1 ==== 88+0+0 (secure 0 0 0) 0x7f62e0041760 con 0x7f62e4071cb0 2024-02-19T21:56:29.477 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:29.476+0000 7f62ea7fc700 1 -- 172.21.15.53:0/2354273020 >> [v2:172.21.15.53:6800/2447819309,v1:172.21.15.53:6801/2447819309] conn(0x7f62cc02ff20 msgr2=0x7f62cc0323e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:56:29.477 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:29.476+0000 7f62ea7fc700 1 --2- 172.21.15.53:0/2354273020 >> [v2:172.21.15.53:6800/2447819309,v1:172.21.15.53:6801/2447819309] conn(0x7f62cc02ff20 0x7f62cc0323e0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f62d40096a0 tx=0x7f62d4006b40).stop 2024-02-19T21:56:29.477 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:29.476+0000 7f62ea7fc700 1 -- 172.21.15.53:0/2354273020 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f62e4071cb0 msgr2=0x7f62e410cd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:56:29.477 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:29.476+0000 7f62ea7fc700 1 --2- 172.21.15.53:0/2354273020 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f62e4071cb0 0x7f62e410cd10 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f62e0006800 tx=0x7f62e0006ee0).stop 2024-02-19T21:56:29.477 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:29.476+0000 7f62ea7fc700 1 -- 172.21.15.53:0/2354273020 shutdown_connections 2024-02-19T21:56:29.477 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:29.476+0000 7f62ea7fc700 1 --2- 172.21.15.53:0/2354273020 >> [v2:172.21.15.53:6800/2447819309,v1:172.21.15.53:6801/2447819309] conn(0x7f62cc02ff20 0x7f62cc0323e0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:56:29.477 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:29.476+0000 7f62ea7fc700 1 --2- 172.21.15.53:0/2354273020 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f62e4071cb0 0x7f62e410cd10 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:56:29.477 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:29.476+0000 7f62ea7fc700 1 -- 172.21.15.53:0/2354273020 >> 172.21.15.53:0/2354273020 conn(0x7f62e40f7480 msgr2=0x7f62e40f8e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:56:29.478 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:29.478+0000 7f62ea7fc700 1 -- 172.21.15.53:0/2354273020 shutdown_connections 2024-02-19T21:56:29.478 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:29.478+0000 7f62ea7fc700 1 -- 172.21.15.53:0/2354273020 wait complete. 2024-02-19T21:56:30.114 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi053 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/867b6b52-cf71-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpoyd_8g9j:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp0zy4ixsn:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr stat 2024-02-19T21:56:30.389 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:56:30 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:56:29.341070+0000 mon.smithi053 (mon.0) 87 : audit [INF] from='mgr.14142 172.21.15.53:0/2487765561' entity='mgr.smithi053.fyhcdi' 2024-02-19T21:56:30.389 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:56:30 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:56:29.475072+0000 mon.smithi053 (mon.0) 88 : audit [INF] from='client.? 172.21.15.53:0/2354273020' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-02-19T21:56:30.389 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:56:30 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T21:56:29.475132+0000 mon.smithi053 (mon.0) 89 : cluster [DBG] mgrmap e13: smithi053.fyhcdi(active, since 10s) 2024-02-19T21:56:31.477 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:56:31 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:56:31.090804+0000 mon.smithi053 (mon.0) 90 : audit [DBG] from='client.? 172.21.15.53:0/2471657102' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-02-19T21:56:31.595 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:30.880+0000 7f05cb9bb700 1 Processor -- start 2024-02-19T21:56:31.595 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:30.880+0000 7f05cb9bb700 1 -- start start 2024-02-19T21:56:31.595 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:30.880+0000 7f05cb9bb700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f05c40703f0 0x7f05c40feac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T21:56:31.595 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:30.880+0000 7f05cb9bb700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05c40fefb0 con 0x7f05c40703f0 2024-02-19T21:56:31.596 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:30.880+0000 7f05c9757700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f05c40703f0 0x7f05c40feac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:56:31.596 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:30.880+0000 7f05c9757700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f05c40703f0 0x7f05c40feac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:38324/0 (socket says 172.21.15.53:38324) 2024-02-19T21:56:31.596 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:30.880+0000 7f05c9757700 1 -- 172.21.15.53:0/1595021098 learned_addr learned my addr 172.21.15.53:0/1595021098 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-02-19T21:56:31.596 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:30.881+0000 7f05c9757700 1 -- 172.21.15.53:0/1595021098 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05c40ff0f0 con 0x7f05c40703f0 2024-02-19T21:56:31.596 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:30.881+0000 7f05c9757700 1 --2- 172.21.15.53:0/1595021098 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f05c40703f0 0x7f05c40feac0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f05c000b800 tx=0x7f05c0006ee0).ready entity=mon.0 client_cookie=2a33dab03be5f576 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:56:31.596 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:30.881+0000 7f05b3fff700 1 -- 172.21.15.53:0/1595021098 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f05c00097d0 con 0x7f05c40703f0 2024-02-19T21:56:31.596 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:30.881+0000 7f05b3fff700 1 -- 172.21.15.53:0/1595021098 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f05c0009e00 con 0x7f05c40703f0 2024-02-19T21:56:31.596 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:30.881+0000 7f05b3fff700 1 -- 172.21.15.53:0/1595021098 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f05c0010360 con 0x7f05c40703f0 2024-02-19T21:56:31.596 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:30.882+0000 7f05cb9bb700 1 -- 172.21.15.53:0/1595021098 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f05c40703f0 msgr2=0x7f05c40feac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:56:31.596 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:30.882+0000 7f05cb9bb700 1 --2- 172.21.15.53:0/1595021098 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f05c40703f0 0x7f05c40feac0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f05c000b800 tx=0x7f05c0006ee0).stop 2024-02-19T21:56:31.596 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:30.882+0000 7f05cb9bb700 1 -- 172.21.15.53:0/1595021098 shutdown_connections 2024-02-19T21:56:31.596 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:30.882+0000 7f05cb9bb700 1 --2- 172.21.15.53:0/1595021098 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f05c40703f0 0x7f05c40feac0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:56:31.597 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:30.882+0000 7f05cb9bb700 1 -- 172.21.15.53:0/1595021098 >> 172.21.15.53:0/1595021098 conn(0x7f05c40f6520 msgr2=0x7f05c40f8940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:56:31.597 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:30.882+0000 7f05cb9bb700 1 -- 172.21.15.53:0/1595021098 shutdown_connections 2024-02-19T21:56:31.597 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:30.882+0000 7f05cb9bb700 1 -- 172.21.15.53:0/1595021098 wait complete. 2024-02-19T21:56:31.597 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:30.883+0000 7f05cb9bb700 1 Processor -- start 2024-02-19T21:56:31.597 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:30.883+0000 7f05cb9bb700 1 -- start start 2024-02-19T21:56:31.597 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:30.883+0000 7f05cb9bb700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f05c40703f0 0x7f05c410ad00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:56:31.597 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:30.883+0000 7f05cb9bb700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05c410b1f0 con 0x7f05c40703f0 2024-02-19T21:56:31.597 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:30.883+0000 7f05c9757700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f05c40703f0 0x7f05c410ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:56:31.597 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:30.883+0000 7f05c9757700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f05c40703f0 0x7f05c410ad00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:38326/0 (socket says 172.21.15.53:38326) 2024-02-19T21:56:31.597 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:30.883+0000 7f05c9757700 1 -- 172.21.15.53:0/2471657102 learned_addr learned my addr 172.21.15.53:0/2471657102 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-02-19T21:56:31.597 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:30.883+0000 7f05c9757700 1 -- 172.21.15.53:0/2471657102 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05c000b200 con 0x7f05c40703f0 2024-02-19T21:56:31.597 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:30.883+0000 7f05c9757700 1 --2- 172.21.15.53:0/2471657102 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f05c40703f0 0x7f05c410ad00 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f05c00098a0 tx=0x7f05c0006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:56:31.597 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:30.884+0000 7f05b1ffb700 1 -- 172.21.15.53:0/2471657102 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f05c0011d50 con 0x7f05c40703f0 2024-02-19T21:56:31.597 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:30.884+0000 7f05cb9bb700 1 -- 172.21.15.53:0/2471657102 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05c410b3f0 con 0x7f05c40703f0 2024-02-19T21:56:31.598 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:30.884+0000 7f05cb9bb700 1 -- 172.21.15.53:0/2471657102 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05c410bfa0 con 0x7f05c40703f0 2024-02-19T21:56:31.598 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:30.884+0000 7f05b1ffb700 1 -- 172.21.15.53:0/2471657102 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f05c0010aa0 con 0x7f05c40703f0 2024-02-19T21:56:31.598 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:30.884+0000 7f05b1ffb700 1 -- 172.21.15.53:0/2471657102 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f05c001bad0 con 0x7f05c40703f0 2024-02-19T21:56:31.598 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:30.885+0000 7f05b1ffb700 1 -- 172.21.15.53:0/2471657102 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 13) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f05c000f070 con 0x7f05c40703f0 2024-02-19T21:56:31.598 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:30.885+0000 7f05cb9bb700 1 -- 172.21.15.53:0/2471657102 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05a4004fa0 con 0x7f05c40703f0 2024-02-19T21:56:31.598 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:30.885+0000 7f05b1ffb700 1 --2- 172.21.15.53:0/2471657102 >> [v2:172.21.15.53:6800/2447819309,v1:172.21.15.53:6801/2447819309] conn(0x7f05ac0343d0 0x7f05ac036890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:56:31.598 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:30.885+0000 7f05b1ffb700 1 -- 172.21.15.53:0/2471657102 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f05c0042650 con 0x7f05c40703f0 2024-02-19T21:56:31.598 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:30.885+0000 7f05c8f56700 1 -- 172.21.15.53:0/2471657102 >> [v2:172.21.15.53:6800/2447819309,v1:172.21.15.53:6801/2447819309] conn(0x7f05ac0343d0 msgr2=0x7f05ac036890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.53:6800/2447819309 2024-02-19T21:56:31.598 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:30.885+0000 7f05c8f56700 1 --2- 172.21.15.53:0/2471657102 >> [v2:172.21.15.53:6800/2447819309,v1:172.21.15.53:6801/2447819309] conn(0x7f05ac0343d0 0x7f05ac036890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-02-19T21:56:31.598 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:30.890+0000 7f05b1ffb700 1 -- 172.21.15.53:0/2471657102 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f05c0019950 con 0x7f05c40703f0 2024-02-19T21:56:31.598 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:31.086+0000 7f05c8f56700 1 -- 172.21.15.53:0/2471657102 >> [v2:172.21.15.53:6800/2447819309,v1:172.21.15.53:6801/2447819309] conn(0x7f05ac0343d0 msgr2=0x7f05ac036890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.53:6800/2447819309 2024-02-19T21:56:31.598 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:31.086+0000 7f05c8f56700 1 --2- 172.21.15.53:0/2471657102 >> [v2:172.21.15.53:6800/2447819309,v1:172.21.15.53:6801/2447819309] conn(0x7f05ac0343d0 0x7f05ac036890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2024-02-19T21:56:31.599 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:31.089+0000 7f05cb9bb700 1 -- 172.21.15.53:0/2471657102 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f05a4005e80 con 0x7f05c40703f0 2024-02-19T21:56:31.599 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:31.089+0000 7f05b1ffb700 1 -- 172.21.15.53:0/2471657102 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v13) v1 ==== 56+0+104 (secure 0 0 0) 0x7f05c0016070 con 0x7f05c40703f0 2024-02-19T21:56:31.599 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:31.091+0000 7f05cb9bb700 1 -- 172.21.15.53:0/2471657102 >> [v2:172.21.15.53:6800/2447819309,v1:172.21.15.53:6801/2447819309] conn(0x7f05ac0343d0 msgr2=0x7f05ac036890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-19T21:56:31.599 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:31.091+0000 7f05cb9bb700 1 --2- 172.21.15.53:0/2471657102 >> [v2:172.21.15.53:6800/2447819309,v1:172.21.15.53:6801/2447819309] conn(0x7f05ac0343d0 0x7f05ac036890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-19T21:56:31.599 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:31.091+0000 7f05cb9bb700 1 -- 172.21.15.53:0/2471657102 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f05c40703f0 msgr2=0x7f05c410ad00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:56:31.599 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:31.091+0000 7f05cb9bb700 1 --2- 172.21.15.53:0/2471657102 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f05c40703f0 0x7f05c410ad00 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f05c00098a0 tx=0x7f05c0006ee0).stop 2024-02-19T21:56:31.599 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:31.091+0000 7f05cb9bb700 1 -- 172.21.15.53:0/2471657102 shutdown_connections 2024-02-19T21:56:31.599 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:31.091+0000 7f05cb9bb700 1 --2- 172.21.15.53:0/2471657102 >> [v2:172.21.15.53:6800/2447819309,v1:172.21.15.53:6801/2447819309] conn(0x7f05ac0343d0 0x7f05ac036890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-19T21:56:31.599 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:31.091+0000 7f05cb9bb700 1 --2- 172.21.15.53:0/2471657102 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f05c40703f0 0x7f05c410ad00 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:56:31.599 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:31.091+0000 7f05cb9bb700 1 -- 172.21.15.53:0/2471657102 >> 172.21.15.53:0/2471657102 conn(0x7f05c40f6520 msgr2=0x7f05c40f7f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:56:31.600 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:31.091+0000 7f05cb9bb700 1 -- 172.21.15.53:0/2471657102 shutdown_connections 2024-02-19T21:56:31.600 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:31.091+0000 7f05cb9bb700 1 -- 172.21.15.53:0/2471657102 wait complete. 2024-02-19T21:56:31.600 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: { 2024-02-19T21:56:31.600 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "epoch": 13, 2024-02-19T21:56:31.600 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "available": true, 2024-02-19T21:56:31.600 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "active_name": "smithi053.fyhcdi", 2024-02-19T21:56:31.600 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "num_standby": 0 2024-02-19T21:56:31.600 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: } 2024-02-19T21:56:32.220 INFO:teuthology.orchestra.run.smithi053.stderr:Waiting for the mgr to restart... 2024-02-19T21:56:32.220 INFO:teuthology.orchestra.run.smithi053.stderr:Waiting for mgr epoch 13... 2024-02-19T21:56:32.220 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi053 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/867b6b52-cf71-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpoyd_8g9j:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp0zy4ixsn:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 tell mgr mgr_status 2024-02-19T21:56:32.774 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:32.765+0000 7f37812ba700 1 Processor -- start 2024-02-19T21:56:32.775 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:32.765+0000 7f37812ba700 1 -- start start 2024-02-19T21:56:32.775 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:32.766+0000 7f37812ba700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f377c0703d0 0x7f377c100eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T21:56:32.775 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:32.766+0000 7f37812ba700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f377c0fc8a0 con 0x7f377c0703d0 2024-02-19T21:56:32.775 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:32.766+0000 7f377affd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f377c0703d0 0x7f377c100eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:56:32.775 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:32.766+0000 7f377affd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f377c0703d0 0x7f377c100eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:38340/0 (socket says 172.21.15.53:38340) 2024-02-19T21:56:32.775 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:32.766+0000 7f377affd700 1 -- 172.21.15.53:0/2766757105 learned_addr learned my addr 172.21.15.53:0/2766757105 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-02-19T21:56:32.775 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:32.766+0000 7f377affd700 1 -- 172.21.15.53:0/2766757105 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f377c1013a0 con 0x7f377c0703d0 2024-02-19T21:56:32.775 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:32.766+0000 7f377affd700 1 --2- 172.21.15.53:0/2766757105 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f377c0703d0 0x7f377c100eb0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f376c00b5f0 tx=0x7f376c006ee0).ready entity=mon.0 client_cookie=cc53eec183736a83 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:56:32.775 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:32.766+0000 7f3779ffb700 1 -- 172.21.15.53:0/2766757105 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f376c00d600 con 0x7f377c0703d0 2024-02-19T21:56:32.775 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:32.766+0000 7f3779ffb700 1 -- 172.21.15.53:0/2766757105 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f376c00dc30 con 0x7f377c0703d0 2024-02-19T21:56:32.775 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:32.766+0000 7f3779ffb700 1 -- 172.21.15.53:0/2766757105 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f376c011d30 con 0x7f377c0703d0 2024-02-19T21:56:32.775 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:32.767+0000 7f37812ba700 1 -- 172.21.15.53:0/2766757105 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f377c0703d0 msgr2=0x7f377c100eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:56:32.776 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:32.767+0000 7f37812ba700 1 --2- 172.21.15.53:0/2766757105 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f377c0703d0 0x7f377c100eb0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f376c00b5f0 tx=0x7f376c006ee0).stop 2024-02-19T21:56:32.776 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:32.767+0000 7f37812ba700 1 -- 172.21.15.53:0/2766757105 shutdown_connections 2024-02-19T21:56:32.776 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:32.767+0000 7f37812ba700 1 --2- 172.21.15.53:0/2766757105 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f377c0703d0 0x7f377c100eb0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:56:32.776 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:32.767+0000 7f37812ba700 1 -- 172.21.15.53:0/2766757105 >> 172.21.15.53:0/2766757105 conn(0x7f377c0f6530 msgr2=0x7f377c0f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:56:32.776 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:32.767+0000 7f37812ba700 1 -- 172.21.15.53:0/2766757105 shutdown_connections 2024-02-19T21:56:32.776 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:32.767+0000 7f37812ba700 1 -- 172.21.15.53:0/2766757105 wait complete. 2024-02-19T21:56:32.776 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:32.768+0000 7f37812ba700 1 Processor -- start 2024-02-19T21:56:32.776 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:32.768+0000 7f37812ba700 1 -- start start 2024-02-19T21:56:32.776 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:32.768+0000 7f37812ba700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f377c0703d0 0x7f377c10cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:56:32.777 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:32.768+0000 7f37812ba700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f377c10d3b0 con 0x7f377c0703d0 2024-02-19T21:56:32.777 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:32.768+0000 7f377affd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f377c0703d0 0x7f377c10cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:56:32.777 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:32.768+0000 7f377affd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f377c0703d0 0x7f377c10cec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:38348/0 (socket says 172.21.15.53:38348) 2024-02-19T21:56:32.777 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:32.768+0000 7f377affd700 1 -- 172.21.15.53:0/3680958354 learned_addr learned my addr 172.21.15.53:0/3680958354 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-02-19T21:56:32.777 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:32.768+0000 7f377affd700 1 -- 172.21.15.53:0/3680958354 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f376c00b200 con 0x7f377c0703d0 2024-02-19T21:56:32.777 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:32.768+0000 7f377affd700 1 --2- 172.21.15.53:0/3680958354 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f377c0703d0 0x7f377c10cec0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f376c006c70 tx=0x7f376c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:56:32.777 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:32.768+0000 7f37637fe700 1 -- 172.21.15.53:0/3680958354 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f376c00d850 con 0x7f377c0703d0 2024-02-19T21:56:32.777 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:32.769+0000 7f37637fe700 1 -- 172.21.15.53:0/3680958354 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f376c00d9b0 con 0x7f377c0703d0 2024-02-19T21:56:32.777 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:32.769+0000 7f37812ba700 1 -- 172.21.15.53:0/3680958354 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f377c10d5b0 con 0x7f377c0703d0 2024-02-19T21:56:32.777 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:32.769+0000 7f37637fe700 1 -- 172.21.15.53:0/3680958354 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f376c01a750 con 0x7f377c0703d0 2024-02-19T21:56:32.777 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:32.769+0000 7f37812ba700 1 -- 172.21.15.53:0/3680958354 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f377c10e160 con 0x7f377c0703d0 2024-02-19T21:56:32.777 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:32.769+0000 7f37637fe700 1 -- 172.21.15.53:0/3680958354 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 13) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f376c00f070 con 0x7f377c0703d0 2024-02-19T21:56:32.777 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:32.770+0000 7f37637fe700 1 --2- 172.21.15.53:0/3680958354 >> [v2:172.21.15.53:6800/2447819309,v1:172.21.15.53:6801/2447819309] conn(0x7f376802ffc0 0x7f3768032480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:56:32.777 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:32.770+0000 7f37637fe700 1 -- 172.21.15.53:0/3680958354 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f376c040de0 con 0x7f377c0703d0 2024-02-19T21:56:32.777 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:32.770+0000 7f377a7fc700 1 -- 172.21.15.53:0/3680958354 >> [v2:172.21.15.53:6800/2447819309,v1:172.21.15.53:6801/2447819309] conn(0x7f376802ffc0 msgr2=0x7f3768032480 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.53:6800/2447819309 2024-02-19T21:56:32.778 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:32.770+0000 7f377a7fc700 1 --2- 172.21.15.53:0/3680958354 >> [v2:172.21.15.53:6800/2447819309,v1:172.21.15.53:6801/2447819309] conn(0x7f376802ffc0 0x7f3768032480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-02-19T21:56:32.778 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:32.770+0000 7f37812ba700 1 -- 172.21.15.53:0/3680958354 --> [v2:172.21.15.53:6800/2447819309,v1:172.21.15.53:6801/2447819309] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f377c059100 con 0x7f376802ffc0 2024-02-19T21:56:32.971 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:32.971+0000 7f377a7fc700 1 -- 172.21.15.53:0/3680958354 >> [v2:172.21.15.53:6800/2447819309,v1:172.21.15.53:6801/2447819309] conn(0x7f376802ffc0 msgr2=0x7f3768032480 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.53:6800/2447819309 2024-02-19T21:56:32.971 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:32.971+0000 7f377a7fc700 1 --2- 172.21.15.53:0/3680958354 >> [v2:172.21.15.53:6800/2447819309,v1:172.21.15.53:6801/2447819309] conn(0x7f376802ffc0 0x7f3768032480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2024-02-19T21:56:33.372 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:33.372+0000 7f377a7fc700 1 -- 172.21.15.53:0/3680958354 >> [v2:172.21.15.53:6800/2447819309,v1:172.21.15.53:6801/2447819309] conn(0x7f376802ffc0 msgr2=0x7f3768032480 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.53:6800/2447819309 2024-02-19T21:56:33.372 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:33.372+0000 7f377a7fc700 1 --2- 172.21.15.53:0/3680958354 >> [v2:172.21.15.53:6800/2447819309,v1:172.21.15.53:6801/2447819309] conn(0x7f376802ffc0 0x7f3768032480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2024-02-19T21:56:34.173 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:34.173+0000 7f377a7fc700 1 -- 172.21.15.53:0/3680958354 >> [v2:172.21.15.53:6800/2447819309,v1:172.21.15.53:6801/2447819309] conn(0x7f376802ffc0 msgr2=0x7f3768032480 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.53:6800/2447819309 2024-02-19T21:56:34.173 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:34.173+0000 7f377a7fc700 1 --2- 172.21.15.53:0/3680958354 >> [v2:172.21.15.53:6800/2447819309,v1:172.21.15.53:6801/2447819309] conn(0x7f376802ffc0 0x7f3768032480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2024-02-19T21:56:35.432 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:35.431+0000 7f37637fe700 1 -- 172.21.15.53:0/3680958354 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mgrmap(e 14) v1 ==== 35997+0+0 (secure 0 0 0) 0x7f376c01add0 con 0x7f377c0703d0 2024-02-19T21:56:35.432 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:35.432+0000 7f37637fe700 1 -- 172.21.15.53:0/3680958354 >> [v2:172.21.15.53:6800/2447819309,v1:172.21.15.53:6801/2447819309] conn(0x7f376802ffc0 msgr2=0x7f3768032480 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-19T21:56:35.433 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:35.432+0000 7f37637fe700 1 --2- 172.21.15.53:0/3680958354 >> [v2:172.21.15.53:6800/2447819309,v1:172.21.15.53:6801/2447819309] conn(0x7f376802ffc0 0x7f3768032480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-19T21:56:36.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:56:35 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T21:56:34.715073+0000 mon.smithi053 (mon.0) 91 : cluster [INF] Active manager daemon smithi053.fyhcdi restarted 2024-02-19T21:56:36.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:56:35 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T21:56:34.715506+0000 mon.smithi053 (mon.0) 92 : cluster [INF] Activating manager daemon smithi053.fyhcdi 2024-02-19T21:56:36.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:56:35 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T21:56:34.717155+0000 mon.smithi053 (mon.0) 93 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-02-19T21:56:36.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:56:35 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T21:56:35.433961+0000 mon.smithi053 (mon.0) 94 : cluster [DBG] mgrmap e14: smithi053.fyhcdi(active, starting, since 0.718567s) 2024-02-19T21:56:36.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:56:35 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:56:35.435378+0000 mon.smithi053 (mon.0) 95 : audit [DBG] from='mgr.14162 172.21.15.53:0/2860358250' entity='mgr.smithi053.fyhcdi' cmd=[{"prefix": "mon metadata", "id": "smithi053"}]: dispatch 2024-02-19T21:56:36.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:56:35 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:56:35.436112+0000 mon.smithi053 (mon.0) 96 : audit [DBG] from='mgr.14162 172.21.15.53:0/2860358250' entity='mgr.smithi053.fyhcdi' cmd=[{"prefix": "mgr metadata", "who": "smithi053.fyhcdi", "id": "smithi053.fyhcdi"}]: dispatch 2024-02-19T21:56:36.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:56:35 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:56:35.437461+0000 mon.smithi053 (mon.0) 97 : audit [DBG] from='mgr.14162 172.21.15.53:0/2860358250' entity='mgr.smithi053.fyhcdi' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-19T21:56:36.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:56:35 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:56:35.437789+0000 mon.smithi053 (mon.0) 98 : audit [DBG] from='mgr.14162 172.21.15.53:0/2860358250' entity='mgr.smithi053.fyhcdi' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-19T21:56:36.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:56:35 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:56:35.438089+0000 mon.smithi053 (mon.0) 99 : audit [DBG] from='mgr.14162 172.21.15.53:0/2860358250' entity='mgr.smithi053.fyhcdi' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-19T21:56:36.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:56:35 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T21:56:35.452041+0000 mon.smithi053 (mon.0) 100 : cluster [INF] Manager daemon smithi053.fyhcdi is now available 2024-02-19T21:56:36.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:56:35 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:56:35.568759+0000 mon.smithi053 (mon.0) 101 : audit [DBG] from='mgr.14162 172.21.15.53:0/2860358250' entity='mgr.smithi053.fyhcdi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T21:56:36.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:56:35 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:56:35.571250+0000 mon.smithi053 (mon.0) 102 : audit [DBG] from='mgr.14162 172.21.15.53:0/2860358250' entity='mgr.smithi053.fyhcdi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T21:56:36.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:56:35 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:56:35.589948+0000 mon.smithi053 (mon.0) 103 : audit [INF] from='mgr.14162 172.21.15.53:0/2860358250' entity='mgr.smithi053.fyhcdi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi053.fyhcdi/mirror_snapshot_schedule"}]: dispatch 2024-02-19T21:56:36.045 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:56:35 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:56:35.598473+0000 mon.smithi053 (mon.0) 104 : audit [INF] from='mgr.14162 172.21.15.53:0/2860358250' entity='mgr.smithi053.fyhcdi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi053.fyhcdi/trash_purge_schedule"}]: dispatch 2024-02-19T21:56:36.433 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:36.432+0000 7f37637fe700 1 -- 172.21.15.53:0/3680958354 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mgrmap(e 15) v1 ==== 36067+0+0 (secure 0 0 0) 0x7f376c01bc00 con 0x7f377c0703d0 2024-02-19T21:56:36.433 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:36.432+0000 7f37637fe700 1 --2- 172.21.15.53:0/3680958354 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f3768033aa0 0x7f3768035e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:56:36.433 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:36.432+0000 7f37637fe700 1 -- 172.21.15.53:0/3680958354 --> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f377c059100 con 0x7f3768033aa0 2024-02-19T21:56:36.434 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:36.433+0000 7f377a7fc700 1 --2- 172.21.15.53:0/3680958354 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f3768033aa0 0x7f3768035e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:56:36.434 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:36.434+0000 7f377a7fc700 1 --2- 172.21.15.53:0/3680958354 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f3768033aa0 0x7f3768035e90 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f37700036d0 tx=0x7f3770006d20).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:56:36.434 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:36.434+0000 7f37637fe700 1 -- 172.21.15.53:0/3680958354 <== mgr.14162 v2:172.21.15.53:6800/1543499597 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6438 (secure 0 0 0) 0x7f377c059100 con 0x7f3768033aa0 2024-02-19T21:56:36.445 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:36.445+0000 7f37812ba700 1 -- 172.21.15.53:0/3680958354 --> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f377c10da40 con 0x7f3768033aa0 2024-02-19T21:56:36.445 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:36.445+0000 7f37637fe700 1 -- 172.21.15.53:0/3680958354 <== mgr.14162 v2:172.21.15.53:6800/1543499597 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7f377c10da40 con 0x7f3768033aa0 2024-02-19T21:56:36.445 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: { 2024-02-19T21:56:36.446 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "mgrmap_epoch": 15, 2024-02-19T21:56:36.446 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "initialized": true 2024-02-19T21:56:36.446 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: } 2024-02-19T21:56:36.447 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:36.446+0000 7f37812ba700 1 -- 172.21.15.53:0/3680958354 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f3768033aa0 msgr2=0x7f3768035e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:56:36.447 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:36.446+0000 7f37812ba700 1 --2- 172.21.15.53:0/3680958354 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f3768033aa0 0x7f3768035e90 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f37700036d0 tx=0x7f3770006d20).stop 2024-02-19T21:56:36.447 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:36.446+0000 7f37812ba700 1 -- 172.21.15.53:0/3680958354 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f377c0703d0 msgr2=0x7f377c10cec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:56:36.447 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:36.446+0000 7f37812ba700 1 --2- 172.21.15.53:0/3680958354 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f377c0703d0 0x7f377c10cec0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f376c006c70 tx=0x7f376c006ee0).stop 2024-02-19T21:56:36.447 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:36.446+0000 7f37812ba700 1 -- 172.21.15.53:0/3680958354 shutdown_connections 2024-02-19T21:56:36.447 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:36.446+0000 7f37812ba700 1 --2- 172.21.15.53:0/3680958354 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f3768033aa0 0x7f3768035e90 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:56:36.447 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:36.446+0000 7f37812ba700 1 --2- 172.21.15.53:0/3680958354 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f377c0703d0 0x7f377c10cec0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:56:36.448 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:36.446+0000 7f37812ba700 1 --2- 172.21.15.53:0/3680958354 >> [v2:172.21.15.53:6800/2447819309,v1:172.21.15.53:6801/2447819309] conn(0x7f376802ffc0 0x7f3768032480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-19T21:56:36.448 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:36.446+0000 7f37812ba700 1 -- 172.21.15.53:0/3680958354 >> 172.21.15.53:0/3680958354 conn(0x7f377c0f6530 msgr2=0x7f377c0f7f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:56:36.448 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:36.446+0000 7f37812ba700 1 -- 172.21.15.53:0/3680958354 shutdown_connections 2024-02-19T21:56:36.448 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:36.446+0000 7f37812ba700 1 -- 172.21.15.53:0/3680958354 wait complete. 2024-02-19T21:56:37.324 INFO:teuthology.orchestra.run.smithi053.stderr:mgr epoch 13 is available 2024-02-19T21:56:37.324 INFO:teuthology.orchestra.run.smithi053.stderr:Generating a dashboard self-signed certificate... 2024-02-19T21:56:37.324 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi053 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/867b6b52-cf71-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpoyd_8g9j:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp0zy4ixsn:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 dashboard create-self-signed-cert 2024-02-19T21:56:37.577 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:56:37 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:56:36.435537+0000 mgr.smithi053.fyhcdi (mgr.14162) 1 : audit [DBG] from='client.14166 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-02-19T21:56:37.577 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:56:37 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T21:56:36.436050+0000 mon.smithi053 (mon.0) 105 : cluster [DBG] mgrmap e15: smithi053.fyhcdi(active, since 1.72066s) 2024-02-19T21:56:37.577 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:56:37 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:56:36.446631+0000 mgr.smithi053.fyhcdi (mgr.14162) 2 : audit [DBG] from='client.14166 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-02-19T21:56:37.577 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:56:37 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:56:36.961862+0000 mon.smithi053 (mon.0) 106 : audit [INF] from='mgr.14162 172.21.15.53:0/2860358250' entity='mgr.smithi053.fyhcdi' 2024-02-19T21:56:38.729 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:56:38 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T21:56:37.437391+0000 mon.smithi053 (mon.0) 107 : cluster [DBG] mgrmap e16: smithi053.fyhcdi(active, since 2s) 2024-02-19T21:56:38.729 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:56:38 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:56:38.362501+0000 mon.smithi053 (mon.0) 108 : audit [INF] from='mgr.14162 172.21.15.53:0/2860358250' entity='mgr.smithi053.fyhcdi' 2024-02-19T21:56:38.729 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:56:38 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:56:38.364240+0000 mon.smithi053 (mon.0) 109 : audit [INF] from='mgr.14162 172.21.15.53:0/2860358250' entity='mgr.smithi053.fyhcdi' 2024-02-19T21:56:38.847 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:38.095+0000 7f0fc8690700 1 Processor -- start 2024-02-19T21:56:38.847 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:38.095+0000 7f0fc8690700 1 -- start start 2024-02-19T21:56:38.847 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:38.096+0000 7f0fc8690700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0fc00703b0 0x7f0fc0100ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T21:56:38.848 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:38.096+0000 7f0fc8690700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fc00fc8c0 con 0x7f0fc00703b0 2024-02-19T21:56:38.848 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:38.096+0000 7f0fc642c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0fc00703b0 0x7f0fc0100ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:56:38.848 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:38.096+0000 7f0fc642c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0fc00703b0 0x7f0fc0100ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:33884/0 (socket says 172.21.15.53:33884) 2024-02-19T21:56:38.848 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:38.096+0000 7f0fc642c700 1 -- 172.21.15.53:0/2631012786 learned_addr learned my addr 172.21.15.53:0/2631012786 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-02-19T21:56:38.848 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:38.096+0000 7f0fc642c700 1 -- 172.21.15.53:0/2631012786 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fc01013c0 con 0x7f0fc00703b0 2024-02-19T21:56:38.848 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:38.096+0000 7f0fc642c700 1 --2- 172.21.15.53:0/2631012786 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0fc00703b0 0x7f0fc0100ed0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f0fb000b5f0 tx=0x7f0fb0006ee0).ready entity=mon.0 client_cookie=ca663decd3414bf6 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:56:38.848 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:38.097+0000 7f0fc542a700 1 -- 172.21.15.53:0/2631012786 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0fb000d600 con 0x7f0fc00703b0 2024-02-19T21:56:38.848 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:38.097+0000 7f0fc542a700 1 -- 172.21.15.53:0/2631012786 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0fb000dc30 con 0x7f0fc00703b0 2024-02-19T21:56:38.848 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:38.097+0000 7f0fc542a700 1 -- 172.21.15.53:0/2631012786 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0fb0011d30 con 0x7f0fc00703b0 2024-02-19T21:56:38.848 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:38.097+0000 7f0fc8690700 1 -- 172.21.15.53:0/2631012786 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0fc00703b0 msgr2=0x7f0fc0100ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:56:38.848 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:38.097+0000 7f0fc8690700 1 --2- 172.21.15.53:0/2631012786 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0fc00703b0 0x7f0fc0100ed0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f0fb000b5f0 tx=0x7f0fb0006ee0).stop 2024-02-19T21:56:38.848 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:38.097+0000 7f0fc8690700 1 -- 172.21.15.53:0/2631012786 shutdown_connections 2024-02-19T21:56:38.848 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:38.097+0000 7f0fc8690700 1 --2- 172.21.15.53:0/2631012786 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0fc00703b0 0x7f0fc0100ed0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:56:38.848 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:38.097+0000 7f0fc8690700 1 -- 172.21.15.53:0/2631012786 >> 172.21.15.53:0/2631012786 conn(0x7f0fc00f6530 msgr2=0x7f0fc00f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:56:38.849 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:38.097+0000 7f0fc8690700 1 -- 172.21.15.53:0/2631012786 shutdown_connections 2024-02-19T21:56:38.849 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:38.097+0000 7f0fc8690700 1 -- 172.21.15.53:0/2631012786 wait complete. 2024-02-19T21:56:38.849 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:38.098+0000 7f0fc8690700 1 Processor -- start 2024-02-19T21:56:38.849 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:38.098+0000 7f0fc8690700 1 -- start start 2024-02-19T21:56:38.849 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:38.098+0000 7f0fc8690700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0fc00703b0 0x7f0fc010cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:56:38.849 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:38.098+0000 7f0fc8690700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fc010d290 con 0x7f0fc00703b0 2024-02-19T21:56:38.849 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:38.098+0000 7f0fc642c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0fc00703b0 0x7f0fc010cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:56:38.849 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:38.098+0000 7f0fc642c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0fc00703b0 0x7f0fc010cda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:33890/0 (socket says 172.21.15.53:33890) 2024-02-19T21:56:38.849 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:38.098+0000 7f0fc642c700 1 -- 172.21.15.53:0/1284122303 learned_addr learned my addr 172.21.15.53:0/1284122303 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-02-19T21:56:38.849 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:38.098+0000 7f0fc642c700 1 -- 172.21.15.53:0/1284122303 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fb000b200 con 0x7f0fc00703b0 2024-02-19T21:56:38.849 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:38.098+0000 7f0fc642c700 1 --2- 172.21.15.53:0/1284122303 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0fc00703b0 0x7f0fc010cda0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f0fb000e040 tx=0x7f0fb0006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:56:38.849 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:38.098+0000 7f0faeffd700 1 -- 172.21.15.53:0/1284122303 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0fb000da60 con 0x7f0fc00703b0 2024-02-19T21:56:38.849 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:38.098+0000 7f0faeffd700 1 -- 172.21.15.53:0/1284122303 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0fb001b8e0 con 0x7f0fc00703b0 2024-02-19T21:56:38.849 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:38.099+0000 7f0fc8690700 1 -- 172.21.15.53:0/1284122303 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fc010d490 con 0x7f0fc00703b0 2024-02-19T21:56:38.850 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:38.099+0000 7f0faeffd700 1 -- 172.21.15.53:0/1284122303 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0fb001a900 con 0x7f0fc00703b0 2024-02-19T21:56:38.850 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:38.099+0000 7f0fc8690700 1 -- 172.21.15.53:0/1284122303 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fc010e0d0 con 0x7f0fc00703b0 2024-02-19T21:56:38.850 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:38.099+0000 7f0faeffd700 1 -- 172.21.15.53:0/1284122303 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 16) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f0fb000f070 con 0x7f0fc00703b0 2024-02-19T21:56:38.850 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:38.099+0000 7f0fc8690700 1 -- 172.21.15.53:0/1284122303 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fc010d920 con 0x7f0fc00703b0 2024-02-19T21:56:38.850 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:38.099+0000 7f0faeffd700 1 --2- 172.21.15.53:0/1284122303 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f0fa802ffc0 0x7f0fa8032480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:56:38.850 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:38.100+0000 7f0faeffd700 1 -- 172.21.15.53:0/1284122303 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0fb0041480 con 0x7f0fc00703b0 2024-02-19T21:56:38.850 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:38.100+0000 7f0fc5c2b700 1 --2- 172.21.15.53:0/1284122303 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f0fa802ffc0 0x7f0fa8032480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:56:38.850 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:38.100+0000 7f0fc5c2b700 1 --2- 172.21.15.53:0/1284122303 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f0fa802ffc0 0x7f0fa8032480 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f0fbc0096a0 tx=0x7f0fbc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:56:38.850 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:38.104+0000 7f0faeffd700 1 -- 172.21.15.53:0/1284122303 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0fb00224f0 con 0x7f0fc00703b0 2024-02-19T21:56:38.850 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:38.268+0000 7f0fc8690700 1 -- 172.21.15.53:0/1284122303 --> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7f0fc010e310 con 0x7f0fa802ffc0 2024-02-19T21:56:38.851 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:38.363+0000 7f0faeffd700 1 -- 172.21.15.53:0/1284122303 <== mgr.14162 v2:172.21.15.53:6800/1543499597 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7f0fc010e310 con 0x7f0fa802ffc0 2024-02-19T21:56:38.851 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:38.365+0000 7f0fc8690700 1 -- 172.21.15.53:0/1284122303 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f0fa802ffc0 msgr2=0x7f0fa8032480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:56:38.851 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:38.365+0000 7f0fc8690700 1 --2- 172.21.15.53:0/1284122303 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f0fa802ffc0 0x7f0fa8032480 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f0fbc0096a0 tx=0x7f0fbc006b40).stop 2024-02-19T21:56:38.851 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:38.365+0000 7f0fc8690700 1 -- 172.21.15.53:0/1284122303 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0fc00703b0 msgr2=0x7f0fc010cda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:56:38.851 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:38.365+0000 7f0fc8690700 1 --2- 172.21.15.53:0/1284122303 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0fc00703b0 0x7f0fc010cda0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f0fb000e040 tx=0x7f0fb0006ee0).stop 2024-02-19T21:56:38.851 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:38.365+0000 7f0fc8690700 1 -- 172.21.15.53:0/1284122303 shutdown_connections 2024-02-19T21:56:38.851 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:38.365+0000 7f0fc8690700 1 --2- 172.21.15.53:0/1284122303 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f0fa802ffc0 0x7f0fa8032480 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:56:38.851 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:38.365+0000 7f0fc8690700 1 --2- 172.21.15.53:0/1284122303 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0fc00703b0 0x7f0fc010cda0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:56:38.851 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:38.365+0000 7f0fc8690700 1 -- 172.21.15.53:0/1284122303 >> 172.21.15.53:0/1284122303 conn(0x7f0fc00f6530 msgr2=0x7f0fc00f7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:56:38.851 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:38.365+0000 7f0fc8690700 1 -- 172.21.15.53:0/1284122303 shutdown_connections 2024-02-19T21:56:38.851 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:38.365+0000 7f0fc8690700 1 -- 172.21.15.53:0/1284122303 wait complete. 2024-02-19T21:56:38.851 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: Self-signed certificate created 2024-02-19T21:56:39.514 INFO:teuthology.orchestra.run.smithi053.stderr:Creating initial admin user... 2024-02-19T21:56:39.516 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi053 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/867b6b52-cf71-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpoyd_8g9j:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp0zy4ixsn:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpdc6gd7ls:/tmp/dashboard.pw:z docker.io/ceph/ceph:v16.2.4 dashboard ac-user-create admin -i /tmp/dashboard.pw administrator --force-password --pwd-update-required 2024-02-19T21:56:39.517 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:56:39 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:56:38.269629+0000 mgr.smithi053.fyhcdi (mgr.14162) 3 : audit [DBG] from='client.14172 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T21:56:40.051 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:40.050+0000 7fef9cba2700 1 Processor -- start 2024-02-19T21:56:40.051 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:40.050+0000 7fef9cba2700 1 -- start start 2024-02-19T21:56:40.053 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:40.050+0000 7fef9cba2700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fef98070430 0x7fef980fe890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T21:56:40.053 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:40.050+0000 7fef9cba2700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef980fed80 con 0x7fef98070430 2024-02-19T21:56:40.053 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:40.051+0000 7fef977fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fef98070430 0x7fef980fe890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:56:40.053 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:40.051+0000 7fef977fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fef98070430 0x7fef980fe890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:33894/0 (socket says 172.21.15.53:33894) 2024-02-19T21:56:40.053 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:40.051+0000 7fef977fe700 1 -- 172.21.15.53:0/4043723324 learned_addr learned my addr 172.21.15.53:0/4043723324 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-02-19T21:56:40.054 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:40.051+0000 7fef977fe700 1 -- 172.21.15.53:0/4043723324 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef980feec0 con 0x7fef98070430 2024-02-19T21:56:40.054 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:40.051+0000 7fef977fe700 1 --2- 172.21.15.53:0/4043723324 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fef98070430 0x7fef980fe890 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7fef8c00b5f0 tx=0x7fef8c006ee0).ready entity=mon.0 client_cookie=1c13062cd9a92e75 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:56:40.054 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:40.051+0000 7fef967fc700 1 -- 172.21.15.53:0/4043723324 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fef8c00d600 con 0x7fef98070430 2024-02-19T21:56:40.054 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:40.051+0000 7fef967fc700 1 -- 172.21.15.53:0/4043723324 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fef8c00dc30 con 0x7fef98070430 2024-02-19T21:56:40.054 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:40.051+0000 7fef967fc700 1 -- 172.21.15.53:0/4043723324 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fef8c011d30 con 0x7fef98070430 2024-02-19T21:56:40.054 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:40.052+0000 7fef9cba2700 1 -- 172.21.15.53:0/4043723324 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fef98070430 msgr2=0x7fef980fe890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:56:40.054 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:40.052+0000 7fef9cba2700 1 --2- 172.21.15.53:0/4043723324 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fef98070430 0x7fef980fe890 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7fef8c00b5f0 tx=0x7fef8c006ee0).stop 2024-02-19T21:56:40.054 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:40.052+0000 7fef9cba2700 1 -- 172.21.15.53:0/4043723324 shutdown_connections 2024-02-19T21:56:40.054 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:40.052+0000 7fef9cba2700 1 --2- 172.21.15.53:0/4043723324 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fef98070430 0x7fef980fe890 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:56:40.054 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:40.052+0000 7fef9cba2700 1 -- 172.21.15.53:0/4043723324 >> 172.21.15.53:0/4043723324 conn(0x7fef980f6390 msgr2=0x7fef980f87b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:56:40.055 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:40.052+0000 7fef9cba2700 1 -- 172.21.15.53:0/4043723324 shutdown_connections 2024-02-19T21:56:40.055 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:40.052+0000 7fef9cba2700 1 -- 172.21.15.53:0/4043723324 wait complete. 2024-02-19T21:56:40.055 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:40.052+0000 7fef9cba2700 1 Processor -- start 2024-02-19T21:56:40.055 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:40.053+0000 7fef9cba2700 1 -- start start 2024-02-19T21:56:40.055 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:40.053+0000 7fef9cba2700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fef98070430 0x7fef980fc310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:56:40.055 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:40.053+0000 7fef9cba2700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef980fa790 con 0x7fef98070430 2024-02-19T21:56:40.056 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:40.053+0000 7fef977fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fef98070430 0x7fef980fc310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:56:40.056 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:40.053+0000 7fef977fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fef98070430 0x7fef980fc310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:33906/0 (socket says 172.21.15.53:33906) 2024-02-19T21:56:40.056 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:40.053+0000 7fef977fe700 1 -- 172.21.15.53:0/3264424754 learned_addr learned my addr 172.21.15.53:0/3264424754 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-02-19T21:56:40.057 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:40.053+0000 7fef977fe700 1 -- 172.21.15.53:0/3264424754 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef8c00b200 con 0x7fef98070430 2024-02-19T21:56:40.057 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:40.053+0000 7fef977fe700 1 --2- 172.21.15.53:0/3264424754 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fef98070430 0x7fef980fc310 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7fef8c00e040 tx=0x7fef8c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:56:40.057 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:40.053+0000 7fef7ffff700 1 -- 172.21.15.53:0/3264424754 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fef8c00da60 con 0x7fef98070430 2024-02-19T21:56:40.057 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:40.053+0000 7fef7ffff700 1 -- 172.21.15.53:0/3264424754 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fef8c01b8e0 con 0x7fef98070430 2024-02-19T21:56:40.057 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:40.053+0000 7fef9cba2700 1 -- 172.21.15.53:0/3264424754 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef980fa930 con 0x7fef98070430 2024-02-19T21:56:40.057 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:40.053+0000 7fef7ffff700 1 -- 172.21.15.53:0/3264424754 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fef8c01a900 con 0x7fef98070430 2024-02-19T21:56:40.058 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:40.054+0000 7fef9cba2700 1 -- 172.21.15.53:0/3264424754 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef980fadc0 con 0x7fef98070430 2024-02-19T21:56:40.058 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:40.054+0000 7fef7ffff700 1 -- 172.21.15.53:0/3264424754 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 16) v1 ==== 36208+0+0 (secure 0 0 0) 0x7fef8c00f070 con 0x7fef98070430 2024-02-19T21:56:40.058 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:40.054+0000 7fef7ffff700 1 --2- 172.21.15.53:0/3264424754 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fef8402ff70 0x7fef84032430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:56:40.058 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:40.055+0000 7fef7ffff700 1 -- 172.21.15.53:0/3264424754 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fef8c042610 con 0x7fef98070430 2024-02-19T21:56:40.058 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:40.055+0000 7fef96ffd700 1 --2- 172.21.15.53:0/3264424754 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fef8402ff70 0x7fef84032430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:56:40.058 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:40.055+0000 7fef9cba2700 1 -- 172.21.15.53:0/3264424754 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef98045ad0 con 0x7fef98070430 2024-02-19T21:56:40.059 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:40.055+0000 7fef96ffd700 1 --2- 172.21.15.53:0/3264424754 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fef8402ff70 0x7fef84032430 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7fef880096a0 tx=0x7fef88006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:56:40.059 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:40.059+0000 7fef7ffff700 1 -- 172.21.15.53:0/3264424754 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fef8c016070 con 0x7fef98070430 2024-02-19T21:56:40.236 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:40.235+0000 7fef9cba2700 1 -- 172.21.15.53:0/3264424754 --> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7fef98058f20 con 0x7fef8402ff70 2024-02-19T21:56:40.481 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:40.481+0000 7fef7ffff700 1 -- 172.21.15.53:0/3264424754 <== mgr.14162 v2:172.21.15.53:6800/1543499597 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7fef98058f20 con 0x7fef8402ff70 2024-02-19T21:56:40.482 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: {"username": "admin", "password": "$2b$12$K2vQevvLpElb4mFVrJXyse9JDRwztNCJT2vaJSRnPK3VykXmDEZjS", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1708379800, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-02-19T21:56:40.483 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:40.483+0000 7fef9cba2700 1 -- 172.21.15.53:0/3264424754 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fef8402ff70 msgr2=0x7fef84032430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:56:40.484 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:40.483+0000 7fef9cba2700 1 --2- 172.21.15.53:0/3264424754 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fef8402ff70 0x7fef84032430 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7fef880096a0 tx=0x7fef88006b40).stop 2024-02-19T21:56:40.484 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:40.483+0000 7fef9cba2700 1 -- 172.21.15.53:0/3264424754 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fef98070430 msgr2=0x7fef980fc310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:56:40.484 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:40.483+0000 7fef9cba2700 1 --2- 172.21.15.53:0/3264424754 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fef98070430 0x7fef980fc310 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7fef8c00e040 tx=0x7fef8c006ee0).stop 2024-02-19T21:56:40.484 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:40.483+0000 7fef9cba2700 1 -- 172.21.15.53:0/3264424754 shutdown_connections 2024-02-19T21:56:40.484 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:40.483+0000 7fef9cba2700 1 --2- 172.21.15.53:0/3264424754 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fef8402ff70 0x7fef84032430 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:56:40.484 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:40.483+0000 7fef9cba2700 1 --2- 172.21.15.53:0/3264424754 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fef98070430 0x7fef980fc310 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:56:40.484 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:40.483+0000 7fef9cba2700 1 -- 172.21.15.53:0/3264424754 >> 172.21.15.53:0/3264424754 conn(0x7fef980f6390 msgr2=0x7fef980f7e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:56:40.484 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:40.483+0000 7fef9cba2700 1 -- 172.21.15.53:0/3264424754 shutdown_connections 2024-02-19T21:56:40.484 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:40.483+0000 7fef9cba2700 1 -- 172.21.15.53:0/3264424754 wait complete. 2024-02-19T21:56:41.188 INFO:teuthology.orchestra.run.smithi053.stderr:Fetching dashboard port number... 2024-02-19T21:56:41.188 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi053 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/867b6b52-cf71-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpoyd_8g9j:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp0zy4ixsn:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config get mgr mgr/dashboard/ssl_server_port 2024-02-19T21:56:41.742 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:41.733+0000 7f15a8b60700 1 Processor -- start 2024-02-19T21:56:41.742 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:41.733+0000 7f15a8b60700 1 -- start start 2024-02-19T21:56:41.742 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:41.733+0000 7f15a8b60700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f15a40703d0 0x7f15a40feab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T21:56:41.742 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:41.733+0000 7f15a8b60700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15a40fefa0 con 0x7f15a40703d0 2024-02-19T21:56:41.742 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:41.734+0000 7f15a259c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f15a40703d0 0x7f15a40feab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:56:41.742 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:41.734+0000 7f15a259c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f15a40703d0 0x7f15a40feab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:33920/0 (socket says 172.21.15.53:33920) 2024-02-19T21:56:41.742 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:41.734+0000 7f15a259c700 1 -- 172.21.15.53:0/1143664003 learned_addr learned my addr 172.21.15.53:0/1143664003 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-02-19T21:56:41.742 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:41.734+0000 7f15a259c700 1 -- 172.21.15.53:0/1143664003 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15a40ff0e0 con 0x7f15a40703d0 2024-02-19T21:56:41.743 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:41.734+0000 7f15a259c700 1 --2- 172.21.15.53:0/1143664003 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f15a40703d0 0x7f15a40feab0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f159400b5f0 tx=0x7f1594006ee0).ready entity=mon.0 client_cookie=60a5d6adcccc8208 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:56:41.743 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:41.734+0000 7f15a159a700 1 -- 172.21.15.53:0/1143664003 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f159400d600 con 0x7f15a40703d0 2024-02-19T21:56:41.743 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:41.734+0000 7f15a159a700 1 -- 172.21.15.53:0/1143664003 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f159400dc30 con 0x7f15a40703d0 2024-02-19T21:56:41.743 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:41.734+0000 7f15a159a700 1 -- 172.21.15.53:0/1143664003 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1594011d30 con 0x7f15a40703d0 2024-02-19T21:56:41.743 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:41.735+0000 7f15a8b60700 1 -- 172.21.15.53:0/1143664003 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f15a40703d0 msgr2=0x7f15a40feab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:56:41.743 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:41.735+0000 7f15a8b60700 1 --2- 172.21.15.53:0/1143664003 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f15a40703d0 0x7f15a40feab0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f159400b5f0 tx=0x7f1594006ee0).stop 2024-02-19T21:56:41.743 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:41.735+0000 7f15a8b60700 1 -- 172.21.15.53:0/1143664003 shutdown_connections 2024-02-19T21:56:41.743 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:41.735+0000 7f15a8b60700 1 --2- 172.21.15.53:0/1143664003 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f15a40703d0 0x7f15a40feab0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:56:41.743 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:41.735+0000 7f15a8b60700 1 -- 172.21.15.53:0/1143664003 >> 172.21.15.53:0/1143664003 conn(0x7f15a40f6530 msgr2=0x7f15a40f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:56:41.743 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:41.735+0000 7f15a8b60700 1 -- 172.21.15.53:0/1143664003 shutdown_connections 2024-02-19T21:56:41.743 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:41.735+0000 7f15a8b60700 1 -- 172.21.15.53:0/1143664003 wait complete. 2024-02-19T21:56:41.743 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:41.735+0000 7f15a8b60700 1 Processor -- start 2024-02-19T21:56:41.743 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:41.735+0000 7f15a8b60700 1 -- start start 2024-02-19T21:56:41.743 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:41.735+0000 7f15a8b60700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f15a40703d0 0x7f15a40fc530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:56:41.744 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:41.735+0000 7f15a8b60700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15a40fa9b0 con 0x7f15a40703d0 2024-02-19T21:56:41.744 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:41.736+0000 7f15a259c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f15a40703d0 0x7f15a40fc530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:56:41.744 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:41.736+0000 7f15a259c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f15a40703d0 0x7f15a40fc530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:33932/0 (socket says 172.21.15.53:33932) 2024-02-19T21:56:41.744 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:41.736+0000 7f15a259c700 1 -- 172.21.15.53:0/4061325236 learned_addr learned my addr 172.21.15.53:0/4061325236 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-02-19T21:56:41.744 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:41.736+0000 7f15a259c700 1 -- 172.21.15.53:0/4061325236 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f159400b200 con 0x7f15a40703d0 2024-02-19T21:56:41.744 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:41.736+0000 7f15a259c700 1 --2- 172.21.15.53:0/4061325236 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f15a40703d0 0x7f15a40fc530 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f159400c010 tx=0x7f1594006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:56:41.744 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:41.736+0000 7f1592ffd700 1 -- 172.21.15.53:0/4061325236 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f159400da60 con 0x7f15a40703d0 2024-02-19T21:56:41.744 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:41.736+0000 7f1592ffd700 1 -- 172.21.15.53:0/4061325236 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f159401b8e0 con 0x7f15a40703d0 2024-02-19T21:56:41.744 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:41.736+0000 7f15a8b60700 1 -- 172.21.15.53:0/4061325236 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15a40fab50 con 0x7f15a40703d0 2024-02-19T21:56:41.744 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:41.736+0000 7f1592ffd700 1 -- 172.21.15.53:0/4061325236 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f159401a990 con 0x7f15a40703d0 2024-02-19T21:56:41.744 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:41.736+0000 7f15a8b60700 1 -- 172.21.15.53:0/4061325236 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15a40fafe0 con 0x7f15a40703d0 2024-02-19T21:56:41.744 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:41.737+0000 7f1592ffd700 1 -- 172.21.15.53:0/4061325236 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f159400f070 con 0x7f15a40703d0 2024-02-19T21:56:41.744 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:41.737+0000 7f15a8b60700 1 -- 172.21.15.53:0/4061325236 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15a4045ad0 con 0x7f15a40703d0 2024-02-19T21:56:41.744 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:41.737+0000 7f1592ffd700 1 --2- 172.21.15.53:0/4061325236 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f1588030070 0x7f1588032530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:56:41.745 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:41.737+0000 7f1592ffd700 1 -- 172.21.15.53:0/4061325236 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f15940418c0 con 0x7f15a40703d0 2024-02-19T21:56:41.745 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:41.737+0000 7f15a1d9b700 1 --2- 172.21.15.53:0/4061325236 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f1588030070 0x7f1588032530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:56:41.745 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:41.737+0000 7f15a1d9b700 1 --2- 172.21.15.53:0/4061325236 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f1588030070 0x7f1588032530 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f15980096a0 tx=0x7f1598006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:56:41.745 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:41.741+0000 7f1592ffd700 1 -- 172.21.15.53:0/4061325236 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1594010360 con 0x7f15a40703d0 2024-02-19T21:56:41.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:56:41 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:56:40.237603+0000 mgr.smithi053.fyhcdi (mgr.14162) 4 : audit [DBG] from='client.14174 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T21:56:41.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:56:41 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:56:40.482456+0000 mon.smithi053 (mon.0) 110 : audit [INF] from='mgr.14162 172.21.15.53:0/2860358250' entity='mgr.smithi053.fyhcdi' 2024-02-19T21:56:41.903 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:41.902+0000 7f15a8b60700 1 -- 172.21.15.53:0/4061325236 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7f15a40fffa0 con 0x7f15a40703d0 2024-02-19T21:56:41.903 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:41.903+0000 7f1592ffd700 1 -- 172.21.15.53:0/4061325236 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v10) v1 ==== 112+0+5 (secure 0 0 0) 0x7f1594016030 con 0x7f15a40703d0 2024-02-19T21:56:41.903 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 8443 2024-02-19T21:56:41.905 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:41.905+0000 7f15a8b60700 1 -- 172.21.15.53:0/4061325236 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f1588030070 msgr2=0x7f1588032530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:56:41.905 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:41.905+0000 7f15a8b60700 1 --2- 172.21.15.53:0/4061325236 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f1588030070 0x7f1588032530 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f15980096a0 tx=0x7f1598006b40).stop 2024-02-19T21:56:41.906 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:41.905+0000 7f15a8b60700 1 -- 172.21.15.53:0/4061325236 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f15a40703d0 msgr2=0x7f15a40fc530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:56:41.906 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:41.905+0000 7f15a8b60700 1 --2- 172.21.15.53:0/4061325236 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f15a40703d0 0x7f15a40fc530 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f159400c010 tx=0x7f1594006ee0).stop 2024-02-19T21:56:41.906 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:41.905+0000 7f15a8b60700 1 -- 172.21.15.53:0/4061325236 shutdown_connections 2024-02-19T21:56:41.906 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:41.905+0000 7f15a8b60700 1 --2- 172.21.15.53:0/4061325236 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f1588030070 0x7f1588032530 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:56:41.906 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:41.905+0000 7f15a8b60700 1 --2- 172.21.15.53:0/4061325236 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f15a40703d0 0x7f15a40fc530 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:56:41.906 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:41.905+0000 7f15a8b60700 1 -- 172.21.15.53:0/4061325236 >> 172.21.15.53:0/4061325236 conn(0x7f15a40f6530 msgr2=0x7f15a40f7ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:56:41.906 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:41.905+0000 7f15a8b60700 1 -- 172.21.15.53:0/4061325236 shutdown_connections 2024-02-19T21:56:41.906 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-02-19T21:56:41.905+0000 7f15a8b60700 1 -- 172.21.15.53:0/4061325236 wait complete. 2024-02-19T21:56:42.578 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: systemctl is-enabled firewalld.service 2024-02-19T21:56:42.587 INFO:teuthology.orchestra.run.smithi053.stderr:systemctl: disabled 2024-02-19T21:56:42.588 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: systemctl is-active firewalld.service 2024-02-19T21:56:42.595 INFO:teuthology.orchestra.run.smithi053.stderr:systemctl: inactive 2024-02-19T21:56:42.596 INFO:teuthology.orchestra.run.smithi053.stderr:firewalld.service is not enabled 2024-02-19T21:56:42.596 INFO:teuthology.orchestra.run.smithi053.stderr:Not possible to open ports <[8443]>. firewalld.service is not available 2024-02-19T21:56:42.599 INFO:teuthology.orchestra.run.smithi053.stderr:Ceph Dashboard is now available at: 2024-02-19T21:56:42.599 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-02-19T21:56:42.599 INFO:teuthology.orchestra.run.smithi053.stderr: URL: https://smithi053.front.sepia.ceph.com:8443/ 2024-02-19T21:56:42.599 INFO:teuthology.orchestra.run.smithi053.stderr: User: admin 2024-02-19T21:56:42.599 INFO:teuthology.orchestra.run.smithi053.stderr: Password: v7hc30q2yl 2024-02-19T21:56:42.599 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-02-19T21:56:42.600 INFO:teuthology.orchestra.run.smithi053.stderr:You can access the Ceph CLI with: 2024-02-19T21:56:42.600 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-02-19T21:56:42.600 INFO:teuthology.orchestra.run.smithi053.stderr: sudo /home/ubuntu/cephtest/cephadm shell --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-02-19T21:56:42.600 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-02-19T21:56:42.600 INFO:teuthology.orchestra.run.smithi053.stderr:Please consider enabling telemetry to help improve Ceph: 2024-02-19T21:56:42.600 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-02-19T21:56:42.600 INFO:teuthology.orchestra.run.smithi053.stderr: ceph telemetry on 2024-02-19T21:56:42.600 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-02-19T21:56:42.601 INFO:teuthology.orchestra.run.smithi053.stderr:For more information see: 2024-02-19T21:56:42.601 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-02-19T21:56:42.601 INFO:teuthology.orchestra.run.smithi053.stderr: https://docs.ceph.com/docs/pacific/mgr/telemetry/ 2024-02-19T21:56:42.601 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-02-19T21:56:42.601 INFO:teuthology.orchestra.run.smithi053.stderr:Bootstrap complete. 2024-02-19T21:56:42.601 INFO:teuthology.orchestra.run.smithi053.stderr:Releasing lock 140182440573416 on /run/cephadm/867b6b52-cf71-11ee-95bb-87774f69a715.lock 2024-02-19T21:56:42.601 INFO:teuthology.orchestra.run.smithi053.stderr:Lock 140182440573416 released on /run/cephadm/867b6b52-cf71-11ee-95bb-87774f69a715.lock 2024-02-19T21:56:42.637 INFO:tasks.cephadm:Fetching config... 2024-02-19T21:56:42.637 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-02-19T21:56:42.637 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-02-19T21:56:42.695 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-02-19T21:56:42.696 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-02-19T21:56:42.696 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-02-19T21:56:42.755 INFO:tasks.cephadm:Fetching mon keyring... 2024-02-19T21:56:42.755 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-02-19T21:56:42.755 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/var/lib/ceph/867b6b52-cf71-11ee-95bb-87774f69a715/mon.smithi053/keyring of=/dev/stdout 2024-02-19T21:56:42.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:56:42 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T21:56:41.483750+0000 mon.smithi053 (mon.0) 111 : cluster [DBG] mgrmap e17: smithi053.fyhcdi(active, since 6s) 2024-02-19T21:56:42.794 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:56:42 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:56:41.904082+0000 mon.smithi053 (mon.0) 112 : audit [DBG] from='client.? 172.21.15.53:0/4061325236' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-02-19T21:56:42.821 INFO:tasks.cephadm:Fetching pub ssh key... 2024-02-19T21:56:42.821 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-02-19T21:56:42.821 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-02-19T21:56:42.881 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-02-19T21:56:42.882 DEBUG:teuthology.orchestra.run.smithi053:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCkTewxITfw3HAfoqPo+JypGfCGvIqbS/6zUjdSdPTXQmlHWITF5B654zSBv0upEXbeRLCzE8BMFNKrNb2A3vSNUhtusIyw8WuSnCruenuVcHSTNVwyWN8+b02qhsWOSaa+0oIbIxNUQlLsyhpYr003e2aIVLkJMrgeo9xwM2H2A4jHGzxMTKepB+EbQBjrWxC+pgWGiHYUdG0BPxX2mpSfPolHnlzaVRMsfBCMuHM1Ou4uIr/to/BFZGdG/b0E0n7p1T6z46WJ2kNIYJkaWf4+RLLNjm7gNwtEQHcPi0J5axeLu0gghi3lj62K8DeB2utogf2fgc8PSK4xvcsqJkkpXUcAdy7iCIDNSXWkscR2b+cnsQURj1uuIpp0JwNTuhhZLeCQcqK4uD9/9EMqy6hcqmav3z1UXSMnaWZNY9E6AzU6NcF5j2x4+lhQ+jPppxn8c4A1XJ4HYhcnXu3elEnqFVG4mxLFYVb/uqO4JhlYWd54b12/8wesBUdeslCqaOE= ceph-867b6b52-cf71-11ee-95bb-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-02-19T21:56:42.962 INFO:teuthology.orchestra.run.smithi053.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCkTewxITfw3HAfoqPo+JypGfCGvIqbS/6zUjdSdPTXQmlHWITF5B654zSBv0upEXbeRLCzE8BMFNKrNb2A3vSNUhtusIyw8WuSnCruenuVcHSTNVwyWN8+b02qhsWOSaa+0oIbIxNUQlLsyhpYr003e2aIVLkJMrgeo9xwM2H2A4jHGzxMTKepB+EbQBjrWxC+pgWGiHYUdG0BPxX2mpSfPolHnlzaVRMsfBCMuHM1Ou4uIr/to/BFZGdG/b0E0n7p1T6z46WJ2kNIYJkaWf4+RLLNjm7gNwtEQHcPi0J5axeLu0gghi3lj62K8DeB2utogf2fgc8PSK4xvcsqJkkpXUcAdy7iCIDNSXWkscR2b+cnsQURj1uuIpp0JwNTuhhZLeCQcqK4uD9/9EMqy6hcqmav3z1UXSMnaWZNY9E6AzU6NcF5j2x4+lhQ+jPppxn8c4A1XJ4HYhcnXu3elEnqFVG4mxLFYVb/uqO4JhlYWd54b12/8wesBUdeslCqaOE= ceph-867b6b52-cf71-11ee-95bb-87774f69a715 2024-02-19T21:56:42.973 DEBUG:teuthology.orchestra.run.smithi086:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCkTewxITfw3HAfoqPo+JypGfCGvIqbS/6zUjdSdPTXQmlHWITF5B654zSBv0upEXbeRLCzE8BMFNKrNb2A3vSNUhtusIyw8WuSnCruenuVcHSTNVwyWN8+b02qhsWOSaa+0oIbIxNUQlLsyhpYr003e2aIVLkJMrgeo9xwM2H2A4jHGzxMTKepB+EbQBjrWxC+pgWGiHYUdG0BPxX2mpSfPolHnlzaVRMsfBCMuHM1Ou4uIr/to/BFZGdG/b0E0n7p1T6z46WJ2kNIYJkaWf4+RLLNjm7gNwtEQHcPi0J5axeLu0gghi3lj62K8DeB2utogf2fgc8PSK4xvcsqJkkpXUcAdy7iCIDNSXWkscR2b+cnsQURj1uuIpp0JwNTuhhZLeCQcqK4uD9/9EMqy6hcqmav3z1UXSMnaWZNY9E6AzU6NcF5j2x4+lhQ+jPppxn8c4A1XJ4HYhcnXu3elEnqFVG4mxLFYVb/uqO4JhlYWd54b12/8wesBUdeslCqaOE= ceph-867b6b52-cf71-11ee-95bb-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-02-19T21:56:43.016 INFO:teuthology.orchestra.run.smithi086.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCkTewxITfw3HAfoqPo+JypGfCGvIqbS/6zUjdSdPTXQmlHWITF5B654zSBv0upEXbeRLCzE8BMFNKrNb2A3vSNUhtusIyw8WuSnCruenuVcHSTNVwyWN8+b02qhsWOSaa+0oIbIxNUQlLsyhpYr003e2aIVLkJMrgeo9xwM2H2A4jHGzxMTKepB+EbQBjrWxC+pgWGiHYUdG0BPxX2mpSfPolHnlzaVRMsfBCMuHM1Ou4uIr/to/BFZGdG/b0E0n7p1T6z46WJ2kNIYJkaWf4+RLLNjm7gNwtEQHcPi0J5axeLu0gghi3lj62K8DeB2utogf2fgc8PSK4xvcsqJkkpXUcAdy7iCIDNSXWkscR2b+cnsQURj1uuIpp0JwNTuhhZLeCQcqK4uD9/9EMqy6hcqmav3z1UXSMnaWZNY9E6AzU6NcF5j2x4+lhQ+jPppxn8c4A1XJ4HYhcnXu3elEnqFVG4mxLFYVb/uqO4JhlYWd54b12/8wesBUdeslCqaOE= ceph-867b6b52-cf71-11ee-95bb-87774f69a715 2024-02-19T21:56:43.028 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-02-19T21:56:45.853 INFO:teuthology.orchestra.run.smithi053.stderr:2024-02-19T21:56:45.852+0000 7f1d041bc700 1 -- 172.21.15.53:0/1236824259 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1cfc0f6510 msgr2=0x7f1cfc0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:56:45.853 INFO:teuthology.orchestra.run.smithi053.stderr:2024-02-19T21:56:45.852+0000 7f1d041bc700 1 --2- 172.21.15.53:0/1236824259 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1cfc0f6510 0x7f1cfc0f6930 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f1cf8009960 tx=0x7f1cf8004d10).stop 2024-02-19T21:56:45.853 INFO:teuthology.orchestra.run.smithi053.stderr:2024-02-19T21:56:45.852+0000 7f1d041bc700 1 -- 172.21.15.53:0/1236824259 shutdown_connections 2024-02-19T21:56:45.854 INFO:teuthology.orchestra.run.smithi053.stderr:2024-02-19T21:56:45.852+0000 7f1d041bc700 1 --2- 172.21.15.53:0/1236824259 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1cfc0f6510 0x7f1cfc0f6930 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:56:45.854 INFO:teuthology.orchestra.run.smithi053.stderr:2024-02-19T21:56:45.852+0000 7f1d041bc700 1 -- 172.21.15.53:0/1236824259 >> 172.21.15.53:0/1236824259 conn(0x7f1cfc0f1c70 msgr2=0x7f1cfc0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:56:45.854 INFO:teuthology.orchestra.run.smithi053.stderr:2024-02-19T21:56:45.852+0000 7f1d041bc700 1 -- 172.21.15.53:0/1236824259 shutdown_connections 2024-02-19T21:56:45.854 INFO:teuthology.orchestra.run.smithi053.stderr:2024-02-19T21:56:45.852+0000 7f1d041bc700 1 -- 172.21.15.53:0/1236824259 wait complete. 2024-02-19T21:56:45.854 INFO:teuthology.orchestra.run.smithi053.stderr:2024-02-19T21:56:45.853+0000 7f1d041bc700 1 Processor -- start 2024-02-19T21:56:45.854 INFO:teuthology.orchestra.run.smithi053.stderr:2024-02-19T21:56:45.853+0000 7f1d041bc700 1 -- start start 2024-02-19T21:56:45.854 INFO:teuthology.orchestra.run.smithi053.stderr:2024-02-19T21:56:45.853+0000 7f1d041bc700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1cfc0f6510 0x7f1cfc1067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:56:45.854 INFO:teuthology.orchestra.run.smithi053.stderr:2024-02-19T21:56:45.853+0000 7f1d041bc700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cfc106cd0 con 0x7f1cfc0f6510 2024-02-19T21:56:45.854 INFO:teuthology.orchestra.run.smithi053.stderr:2024-02-19T21:56:45.853+0000 7f1d01f58700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1cfc0f6510 0x7f1cfc1067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:56:45.854 INFO:teuthology.orchestra.run.smithi053.stderr:2024-02-19T21:56:45.853+0000 7f1d01f58700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1cfc0f6510 0x7f1cfc1067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:33960/0 (socket says 172.21.15.53:33960) 2024-02-19T21:56:45.854 INFO:teuthology.orchestra.run.smithi053.stderr:2024-02-19T21:56:45.853+0000 7f1d01f58700 1 -- 172.21.15.53:0/1897842864 learned_addr learned my addr 172.21.15.53:0/1897842864 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-02-19T21:56:45.854 INFO:teuthology.orchestra.run.smithi053.stderr:2024-02-19T21:56:45.854+0000 7f1d01f58700 1 -- 172.21.15.53:0/1897842864 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cf8009640 con 0x7f1cfc0f6510 2024-02-19T21:56:45.854 INFO:teuthology.orchestra.run.smithi053.stderr:2024-02-19T21:56:45.854+0000 7f1d01f58700 1 --2- 172.21.15.53:0/1897842864 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1cfc0f6510 0x7f1cfc1067e0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f1cf8004b70 tx=0x7f1cf8004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:56:45.856 INFO:teuthology.orchestra.run.smithi053.stderr:2024-02-19T21:56:45.854+0000 7f1cf27fc700 1 -- 172.21.15.53:0/1897842864 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1cf8018070 con 0x7f1cfc0f6510 2024-02-19T21:56:45.856 INFO:teuthology.orchestra.run.smithi053.stderr:2024-02-19T21:56:45.854+0000 7f1cf27fc700 1 -- 172.21.15.53:0/1897842864 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1cf800b970 con 0x7f1cfc0f6510 2024-02-19T21:56:45.856 INFO:teuthology.orchestra.run.smithi053.stderr:2024-02-19T21:56:45.854+0000 7f1d041bc700 1 -- 172.21.15.53:0/1897842864 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cfc106ed0 con 0x7f1cfc0f6510 2024-02-19T21:56:45.856 INFO:teuthology.orchestra.run.smithi053.stderr:2024-02-19T21:56:45.854+0000 7f1cf27fc700 1 -- 172.21.15.53:0/1897842864 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1cf801c560 con 0x7f1cfc0f6510 2024-02-19T21:56:45.856 INFO:teuthology.orchestra.run.smithi053.stderr:2024-02-19T21:56:45.854+0000 7f1d041bc700 1 -- 172.21.15.53:0/1897842864 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cfc107360 con 0x7f1cfc0f6510 2024-02-19T21:56:45.856 INFO:teuthology.orchestra.run.smithi053.stderr:2024-02-19T21:56:45.855+0000 7f1cf27fc700 1 -- 172.21.15.53:0/1897842864 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1cf801c6c0 con 0x7f1cfc0f6510 2024-02-19T21:56:45.856 INFO:teuthology.orchestra.run.smithi053.stderr:2024-02-19T21:56:45.855+0000 7f1cf27fc700 1 --2- 172.21.15.53:0/1897842864 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f1ce802ff60 0x7f1ce8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:56:45.857 INFO:teuthology.orchestra.run.smithi053.stderr:2024-02-19T21:56:45.855+0000 7f1d041bc700 1 -- 172.21.15.53:0/1897842864 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1cfc045ad0 con 0x7f1cfc0f6510 2024-02-19T21:56:45.857 INFO:teuthology.orchestra.run.smithi053.stderr:2024-02-19T21:56:45.855+0000 7f1cf27fc700 1 -- 172.21.15.53:0/1897842864 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1cf8041270 con 0x7f1cfc0f6510 2024-02-19T21:56:45.857 INFO:teuthology.orchestra.run.smithi053.stderr:2024-02-19T21:56:45.855+0000 7f1d01757700 1 --2- 172.21.15.53:0/1897842864 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f1ce802ff60 0x7f1ce8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:56:45.857 INFO:teuthology.orchestra.run.smithi053.stderr:2024-02-19T21:56:45.855+0000 7f1d01757700 1 --2- 172.21.15.53:0/1897842864 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f1ce802ff60 0x7f1ce8032420 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f1cec0096a0 tx=0x7f1cec006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:56:45.859 INFO:teuthology.orchestra.run.smithi053.stderr:2024-02-19T21:56:45.859+0000 7f1cf27fc700 1 -- 172.21.15.53:0/1897842864 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1cf806a2b0 con 0x7f1cfc0f6510 2024-02-19T21:56:46.020 INFO:teuthology.orchestra.run.smithi053.stderr:2024-02-19T21:56:46.020+0000 7f1d041bc700 1 -- 172.21.15.53:0/1897842864 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7f1cfc0fc0a0 con 0x7f1cfc0f6510 2024-02-19T21:56:46.022 INFO:teuthology.orchestra.run.smithi053.stderr:2024-02-19T21:56:46.023+0000 7f1cf27fc700 1 -- 172.21.15.53:0/1897842864 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v11)=0 v11) v1 ==== 125+0+0 (secure 0 0 0) 0x7f1cf8012c00 con 0x7f1cfc0f6510 2024-02-19T21:56:46.024 INFO:teuthology.orchestra.run.smithi053.stderr:2024-02-19T21:56:46.024+0000 7f1d041bc700 1 -- 172.21.15.53:0/1897842864 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f1ce802ff60 msgr2=0x7f1ce8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:56:46.024 INFO:teuthology.orchestra.run.smithi053.stderr:2024-02-19T21:56:46.024+0000 7f1d041bc700 1 --2- 172.21.15.53:0/1897842864 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f1ce802ff60 0x7f1ce8032420 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f1cec0096a0 tx=0x7f1cec006b40).stop 2024-02-19T21:56:46.024 INFO:teuthology.orchestra.run.smithi053.stderr:2024-02-19T21:56:46.024+0000 7f1d041bc700 1 -- 172.21.15.53:0/1897842864 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1cfc0f6510 msgr2=0x7f1cfc1067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:56:46.024 INFO:teuthology.orchestra.run.smithi053.stderr:2024-02-19T21:56:46.024+0000 7f1d041bc700 1 --2- 172.21.15.53:0/1897842864 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1cfc0f6510 0x7f1cfc1067e0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f1cf8004b70 tx=0x7f1cf8004d10).stop 2024-02-19T21:56:46.025 INFO:teuthology.orchestra.run.smithi053.stderr:2024-02-19T21:56:46.024+0000 7f1d041bc700 1 -- 172.21.15.53:0/1897842864 shutdown_connections 2024-02-19T21:56:46.025 INFO:teuthology.orchestra.run.smithi053.stderr:2024-02-19T21:56:46.024+0000 7f1d041bc700 1 --2- 172.21.15.53:0/1897842864 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f1ce802ff60 0x7f1ce8032420 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:56:46.025 INFO:teuthology.orchestra.run.smithi053.stderr:2024-02-19T21:56:46.024+0000 7f1d041bc700 1 --2- 172.21.15.53:0/1897842864 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1cfc0f6510 0x7f1cfc1067e0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:56:46.025 INFO:teuthology.orchestra.run.smithi053.stderr:2024-02-19T21:56:46.024+0000 7f1d041bc700 1 -- 172.21.15.53:0/1897842864 >> 172.21.15.53:0/1897842864 conn(0x7f1cfc0f1c70 msgr2=0x7f1cfc0f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:56:46.025 INFO:teuthology.orchestra.run.smithi053.stderr:2024-02-19T21:56:46.025+0000 7f1d041bc700 1 -- 172.21.15.53:0/1897842864 shutdown_connections 2024-02-19T21:56:46.025 INFO:teuthology.orchestra.run.smithi053.stderr:2024-02-19T21:56:46.025+0000 7f1d041bc700 1 -- 172.21.15.53:0/1897842864 wait complete. 2024-02-19T21:56:46.293 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:56:45 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:56:44.902294+0000 mon.smithi053 (mon.0) 113 : audit [INF] from='mgr.14162 172.21.15.53:0/2860358250' entity='mgr.smithi053.fyhcdi' 2024-02-19T21:56:46.294 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:56:45 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:56:45.300732+0000 mon.smithi053 (mon.0) 114 : audit [INF] from='mgr.14162 172.21.15.53:0/2860358250' entity='mgr.smithi053.fyhcdi' 2024-02-19T21:56:46.294 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:56:45 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cephadm 2024-02-19T21:56:45.306775+0000 mgr.smithi053.fyhcdi (mgr.14162) 5 : cephadm [INF] Deploying daemon alertmanager.smithi053 on smithi053 2024-02-19T21:56:46.726 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi086 2024-02-19T21:56:46.726 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-02-19T21:56:46.726 DEBUG:teuthology.orchestra.run.smithi086:> dd of=/etc/ceph/ceph.conf 2024-02-19T21:56:46.748 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-02-19T21:56:46.748 DEBUG:teuthology.orchestra.run.smithi086:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-02-19T21:56:46.809 INFO:tasks.cephadm:Adding host smithi086 to orchestrator... 2024-02-19T21:56:46.809 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph orch host add smithi086 2024-02-19T21:56:47.293 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:56:47 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:56:46.023968+0000 mon.smithi053 (mon.0) 115 : audit [INF] from='client.? 172.21.15.53:0/1897842864' entity='client.admin' 2024-02-19T21:56:56.794 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:56:56 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T21:56:55.438621+0000 mgr.smithi053.fyhcdi (mgr.14162) 6 : cluster [DBG] pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:56:58.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:56:58 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T21:56:57.438954+0000 mgr.smithi053.fyhcdi (mgr.14162) 7 : cluster [DBG] pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:57:00.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:57:00 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T21:56:59.439276+0000 mgr.smithi053.fyhcdi (mgr.14162) 8 : cluster [DBG] pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:57:02.794 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:57:02 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T21:57:01.439548+0000 mgr.smithi053.fyhcdi (mgr.14162) 9 : cluster [DBG] pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:57:04.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:57:04 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T21:57:03.439829+0000 mgr.smithi053.fyhcdi (mgr.14162) 10 : cluster [DBG] pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:57:06.794 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:57:06 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T21:57:05.440219+0000 mgr.smithi053.fyhcdi (mgr.14162) 11 : cluster [DBG] pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:57:08.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:57:08 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T21:57:07.440452+0000 mgr.smithi053.fyhcdi (mgr.14162) 12 : cluster [DBG] pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:57:10.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:57:10 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T21:57:09.440768+0000 mgr.smithi053.fyhcdi (mgr.14162) 13 : cluster [DBG] pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:57:11.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:10.999+0000 7f7fa828c700 1 -- 172.21.15.86:0/1883503502 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7fa00f5410 msgr2=0x7f7fa00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:57:11.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:10.999+0000 7f7fa828c700 1 --2- 172.21.15.86:0/1883503502 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7fa00f5410 0x7f7fa00f5830 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f7f90009fb0 tx=0x7f7f90009b30).stop 2024-02-19T21:57:11.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:10.999+0000 7f7fa828c700 1 -- 172.21.15.86:0/1883503502 shutdown_connections 2024-02-19T21:57:11.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:10.999+0000 7f7fa828c700 1 --2- 172.21.15.86:0/1883503502 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7fa00f5410 0x7f7fa00f5830 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:57:11.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:10.999+0000 7f7fa828c700 1 -- 172.21.15.86:0/1883503502 >> 172.21.15.86:0/1883503502 conn(0x7f7fa00f0b70 msgr2=0x7f7fa00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:57:11.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:11.000+0000 7f7fa828c700 1 -- 172.21.15.86:0/1883503502 shutdown_connections 2024-02-19T21:57:11.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:11.000+0000 7f7fa828c700 1 -- 172.21.15.86:0/1883503502 wait complete. 2024-02-19T21:57:11.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:11.000+0000 7f7fa828c700 1 Processor -- start 2024-02-19T21:57:11.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:11.000+0000 7f7fa828c700 1 -- start start 2024-02-19T21:57:11.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:11.000+0000 7f7fa828c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7fa00f5410 0x7f7fa01056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:57:11.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:11.000+0000 7f7fa828c700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fa0105bd0 con 0x7f7fa00f5410 2024-02-19T21:57:11.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:11.001+0000 7f7fa6028700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7fa00f5410 0x7f7fa01056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:57:11.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:11.001+0000 7f7fa6028700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7fa00f5410 0x7f7fa01056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:60076/0 (socket says 172.21.15.86:60076) 2024-02-19T21:57:11.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:11.001+0000 7f7fa6028700 1 -- 172.21.15.86:0/2469828628 learned_addr learned my addr 172.21.15.86:0/2469828628 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T21:57:11.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:11.001+0000 7f7fa6028700 1 -- 172.21.15.86:0/2469828628 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f90005040 con 0x7f7fa00f5410 2024-02-19T21:57:11.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:11.001+0000 7f7fa6028700 1 --2- 172.21.15.86:0/2469828628 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7fa00f5410 0x7f7fa01056e0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f7f90005010 tx=0x7f7f9000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:57:11.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:11.001+0000 7f7f96ffd700 1 -- 172.21.15.86:0/2469828628 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7f90009320 con 0x7f7fa00f5410 2024-02-19T21:57:11.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:11.001+0000 7f7fa828c700 1 -- 172.21.15.86:0/2469828628 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fa0105dd0 con 0x7f7fa00f5410 2024-02-19T21:57:11.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:11.001+0000 7f7f96ffd700 1 -- 172.21.15.86:0/2469828628 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7f90013380 con 0x7f7fa00f5410 2024-02-19T21:57:11.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:11.001+0000 7f7f96ffd700 1 -- 172.21.15.86:0/2469828628 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7f90014e50 con 0x7f7fa00f5410 2024-02-19T21:57:11.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:11.001+0000 7f7fa828c700 1 -- 172.21.15.86:0/2469828628 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fa0106260 con 0x7f7fa00f5410 2024-02-19T21:57:11.004 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:11.002+0000 7f7f96ffd700 1 -- 172.21.15.86:0/2469828628 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7f90014420 con 0x7f7fa00f5410 2024-02-19T21:57:11.004 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:11.002+0000 7f7fa828c700 1 -- 172.21.15.86:0/2469828628 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fa0045ad0 con 0x7f7fa00f5410 2024-02-19T21:57:11.004 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:11.002+0000 7f7f96ffd700 1 --2- 172.21.15.86:0/2469828628 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f7f8c02ff10 0x7f7f8c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:57:11.004 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:11.002+0000 7f7f96ffd700 1 -- 172.21.15.86:0/2469828628 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7f90040f40 con 0x7f7fa00f5410 2024-02-19T21:57:11.004 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:11.002+0000 7f7fa5827700 1 --2- 172.21.15.86:0/2469828628 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f7f8c02ff10 0x7f7f8c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:57:11.004 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:11.003+0000 7f7fa5827700 1 --2- 172.21.15.86:0/2469828628 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f7f8c02ff10 0x7f7f8c0323d0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f7f9c0096a0 tx=0x7f7f9c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:57:11.006 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:11.007+0000 7f7f96ffd700 1 -- 172.21.15.86:0/2469828628 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7f900095f0 con 0x7f7fa00f5410 2024-02-19T21:57:11.177 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:11.177+0000 7f7fa828c700 1 -- 172.21.15.86:0/2469828628 --> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi086", "target": ["mon-mgr", ""]}) v1 -- 0x7f7fa00f1d20 con 0x7f7f8c02ff10 2024-02-19T21:57:11.774 INFO:teuthology.orchestra.run.smithi086.stdout:Added host 'smithi086' 2024-02-19T21:57:11.774 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:11.773+0000 7f7f96ffd700 1 -- 172.21.15.86:0/2469828628 <== mgr.14162 v2:172.21.15.53:6800/1543499597 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+23 (secure 0 0 0) 0x7f7fa00f1d20 con 0x7f7f8c02ff10 2024-02-19T21:57:11.775 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:11.775+0000 7f7fa828c700 1 -- 172.21.15.86:0/2469828628 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f7f8c02ff10 msgr2=0x7f7f8c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:57:11.775 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:11.775+0000 7f7fa828c700 1 --2- 172.21.15.86:0/2469828628 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f7f8c02ff10 0x7f7f8c0323d0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f7f9c0096a0 tx=0x7f7f9c006b40).stop 2024-02-19T21:57:11.775 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:11.775+0000 7f7fa828c700 1 -- 172.21.15.86:0/2469828628 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7fa00f5410 msgr2=0x7f7fa01056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:57:11.775 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:11.775+0000 7f7fa828c700 1 --2- 172.21.15.86:0/2469828628 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7fa00f5410 0x7f7fa01056e0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f7f90005010 tx=0x7f7f9000b040).stop 2024-02-19T21:57:11.776 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:11.775+0000 7f7fa828c700 1 -- 172.21.15.86:0/2469828628 shutdown_connections 2024-02-19T21:57:11.776 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:11.775+0000 7f7fa828c700 1 --2- 172.21.15.86:0/2469828628 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f7f8c02ff10 0x7f7f8c0323d0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:57:11.776 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:11.775+0000 7f7fa828c700 1 --2- 172.21.15.86:0/2469828628 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7fa00f5410 0x7f7fa01056e0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:57:11.776 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:11.775+0000 7f7fa828c700 1 -- 172.21.15.86:0/2469828628 >> 172.21.15.86:0/2469828628 conn(0x7f7fa00f0b70 msgr2=0x7f7fa00f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:57:11.776 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:11.775+0000 7f7fa828c700 1 -- 172.21.15.86:0/2469828628 shutdown_connections 2024-02-19T21:57:11.776 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:11.775+0000 7f7fa828c700 1 -- 172.21.15.86:0/2469828628 wait complete. 2024-02-19T21:57:12.346 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph orch host ls --format=json 2024-02-19T21:57:12.794 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:57:12 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:57:11.179675+0000 mgr.smithi053.fyhcdi (mgr.14162) 14 : audit [DBG] from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi086", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T21:57:12.794 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:57:12 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T21:57:11.441259+0000 mgr.smithi053.fyhcdi (mgr.14162) 15 : cluster [DBG] pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:57:12.794 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:57:12 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:57:11.773494+0000 mon.smithi053 (mon.0) 116 : audit [INF] from='mgr.14162 172.21.15.53:0/2860358250' entity='mgr.smithi053.fyhcdi' 2024-02-19T21:57:13.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:57:13 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cephadm 2024-02-19T21:57:11.773944+0000 mgr.smithi053.fyhcdi (mgr.14162) 16 : cephadm [INF] Added host smithi086 2024-02-19T21:57:14.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:14.103+0000 7fe1d0dd7700 1 -- 172.21.15.86:0/1326865028 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe1cc0f5410 msgr2=0x7fe1cc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:57:14.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:14.103+0000 7fe1d0dd7700 1 --2- 172.21.15.86:0/1326865028 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe1cc0f5410 0x7fe1cc0f5830 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7fe1c0009fb0 tx=0x7fe1c0009b30).stop 2024-02-19T21:57:14.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:14.103+0000 7fe1d0dd7700 1 -- 172.21.15.86:0/1326865028 shutdown_connections 2024-02-19T21:57:14.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:14.103+0000 7fe1d0dd7700 1 --2- 172.21.15.86:0/1326865028 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe1cc0f5410 0x7fe1cc0f5830 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:57:14.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:14.103+0000 7fe1d0dd7700 1 -- 172.21.15.86:0/1326865028 >> 172.21.15.86:0/1326865028 conn(0x7fe1cc0f0b70 msgr2=0x7fe1cc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:57:14.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:14.103+0000 7fe1d0dd7700 1 -- 172.21.15.86:0/1326865028 shutdown_connections 2024-02-19T21:57:14.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:14.103+0000 7fe1d0dd7700 1 -- 172.21.15.86:0/1326865028 wait complete. 2024-02-19T21:57:14.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:14.104+0000 7fe1d0dd7700 1 Processor -- start 2024-02-19T21:57:14.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:14.104+0000 7fe1d0dd7700 1 -- start start 2024-02-19T21:57:14.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:14.104+0000 7fe1d0dd7700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe1cc0f5410 0x7fe1cc1067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:57:14.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:14.104+0000 7fe1d0dd7700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1cc106cd0 con 0x7fe1cc0f5410 2024-02-19T21:57:14.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:14.104+0000 7fe1ca59c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe1cc0f5410 0x7fe1cc1067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:57:14.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:14.104+0000 7fe1ca59c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe1cc0f5410 0x7fe1cc1067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:60094/0 (socket says 172.21.15.86:60094) 2024-02-19T21:57:14.106 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:14.104+0000 7fe1ca59c700 1 -- 172.21.15.86:0/2605402705 learned_addr learned my addr 172.21.15.86:0/2605402705 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T21:57:14.106 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:14.104+0000 7fe1ca59c700 1 -- 172.21.15.86:0/2605402705 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1c0005040 con 0x7fe1cc0f5410 2024-02-19T21:57:14.106 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:14.104+0000 7fe1ca59c700 1 --2- 172.21.15.86:0/2605402705 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe1cc0f5410 0x7fe1cc1067e0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7fe1c0005010 tx=0x7fe1c000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:57:14.106 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:14.105+0000 7fe1baffd700 1 -- 172.21.15.86:0/2605402705 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe1c0009320 con 0x7fe1cc0f5410 2024-02-19T21:57:14.106 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:14.105+0000 7fe1d0dd7700 1 -- 172.21.15.86:0/2605402705 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1cc106ed0 con 0x7fe1cc0f5410 2024-02-19T21:57:14.106 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:14.105+0000 7fe1baffd700 1 -- 172.21.15.86:0/2605402705 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe1c0013380 con 0x7fe1cc0f5410 2024-02-19T21:57:14.106 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:14.105+0000 7fe1baffd700 1 -- 172.21.15.86:0/2605402705 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe1c0014e50 con 0x7fe1cc0f5410 2024-02-19T21:57:14.106 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:14.105+0000 7fe1d0dd7700 1 -- 172.21.15.86:0/2605402705 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1cc107360 con 0x7fe1cc0f5410 2024-02-19T21:57:14.107 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:14.105+0000 7fe1baffd700 1 -- 172.21.15.86:0/2605402705 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe1c00095f0 con 0x7fe1cc0f5410 2024-02-19T21:57:14.107 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:14.105+0000 7fe1d0dd7700 1 -- 172.21.15.86:0/2605402705 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1cc045ad0 con 0x7fe1cc0f5410 2024-02-19T21:57:14.107 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:14.106+0000 7fe1baffd700 1 --2- 172.21.15.86:0/2605402705 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fe1b402ff10 0x7fe1b40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:57:14.107 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:14.106+0000 7fe1baffd700 1 -- 172.21.15.86:0/2605402705 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe1c0040e90 con 0x7fe1cc0f5410 2024-02-19T21:57:14.107 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:14.106+0000 7fe1c9d9b700 1 --2- 172.21.15.86:0/2605402705 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fe1b402ff10 0x7fe1b40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:57:14.107 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:14.106+0000 7fe1c9d9b700 1 --2- 172.21.15.86:0/2605402705 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fe1b402ff10 0x7fe1b40323d0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7fe1bc0096a0 tx=0x7fe1bc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:57:14.110 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:14.110+0000 7fe1baffd700 1 -- 172.21.15.86:0/2605402705 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe1c0014420 con 0x7fe1cc0f5410 2024-02-19T21:57:14.281 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:14.281+0000 7fe1d0dd7700 1 -- 172.21.15.86:0/2605402705 --> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe1cc0f1d20 con 0x7fe1b402ff10 2024-02-19T21:57:14.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:14.282+0000 7fe1baffd700 1 -- 172.21.15.86:0/2605402705 <== mgr.14162 v2:172.21.15.53:6800/1543499597 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+156 (secure 0 0 0) 0x7fe1cc0f1d20 con 0x7fe1b402ff10 2024-02-19T21:57:14.282 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T21:57:14.283 INFO:teuthology.orchestra.run.smithi086.stdout:[{"addr": "172.21.15.53", "hostname": "smithi053", "labels": [], "status": ""}, {"addr": "smithi086", "hostname": "smithi086", "labels": [], "status": ""}] 2024-02-19T21:57:14.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:14.284+0000 7fe1d0dd7700 1 -- 172.21.15.86:0/2605402705 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fe1b402ff10 msgr2=0x7fe1b40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:57:14.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:14.284+0000 7fe1d0dd7700 1 --2- 172.21.15.86:0/2605402705 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fe1b402ff10 0x7fe1b40323d0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7fe1bc0096a0 tx=0x7fe1bc006b40).stop 2024-02-19T21:57:14.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:14.284+0000 7fe1d0dd7700 1 -- 172.21.15.86:0/2605402705 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe1cc0f5410 msgr2=0x7fe1cc1067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:57:14.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:14.284+0000 7fe1d0dd7700 1 --2- 172.21.15.86:0/2605402705 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe1cc0f5410 0x7fe1cc1067e0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7fe1c0005010 tx=0x7fe1c000b040).stop 2024-02-19T21:57:14.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:14.284+0000 7fe1d0dd7700 1 -- 172.21.15.86:0/2605402705 shutdown_connections 2024-02-19T21:57:14.285 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:14.284+0000 7fe1d0dd7700 1 --2- 172.21.15.86:0/2605402705 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fe1b402ff10 0x7fe1b40323d0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:57:14.285 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:14.284+0000 7fe1d0dd7700 1 --2- 172.21.15.86:0/2605402705 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe1cc0f5410 0x7fe1cc1067e0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:57:14.285 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:14.284+0000 7fe1d0dd7700 1 -- 172.21.15.86:0/2605402705 >> 172.21.15.86:0/2605402705 conn(0x7fe1cc0f0b70 msgr2=0x7fe1cc0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:57:14.285 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:14.284+0000 7fe1d0dd7700 1 -- 172.21.15.86:0/2605402705 shutdown_connections 2024-02-19T21:57:14.285 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:14.284+0000 7fe1d0dd7700 1 -- 172.21.15.86:0/2605402705 wait complete. 2024-02-19T21:57:14.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:57:14 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T21:57:13.441617+0000 mgr.smithi053.fyhcdi (mgr.14162) 17 : cluster [DBG] pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:57:14.834 INFO:tasks.cephadm:Setting crush tunables to default 2024-02-19T21:57:14.834 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph osd crush tunables default 2024-02-19T21:57:15.544 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:57:15 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:57:14.283343+0000 mgr.smithi053.fyhcdi (mgr.14162) 18 : audit [DBG] from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-19T21:57:16.708 INFO:teuthology.orchestra.run.smithi053.stderr:2024-02-19T21:57:16.706+0000 7f44a7906700 1 -- 172.21.15.53:0/1346219610 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f44a00f5410 msgr2=0x7f44a00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:57:16.708 INFO:teuthology.orchestra.run.smithi053.stderr:2024-02-19T21:57:16.706+0000 7f44a7906700 1 --2- 172.21.15.53:0/1346219610 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f44a00f5410 0x7f44a00f5830 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f4494009fb0 tx=0x7f4494009b30).stop 2024-02-19T21:57:16.708 INFO:teuthology.orchestra.run.smithi053.stderr:2024-02-19T21:57:16.706+0000 7f44a7906700 1 -- 172.21.15.53:0/1346219610 shutdown_connections 2024-02-19T21:57:16.708 INFO:teuthology.orchestra.run.smithi053.stderr:2024-02-19T21:57:16.706+0000 7f44a7906700 1 --2- 172.21.15.53:0/1346219610 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f44a00f5410 0x7f44a00f5830 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:57:16.708 INFO:teuthology.orchestra.run.smithi053.stderr:2024-02-19T21:57:16.706+0000 7f44a7906700 1 -- 172.21.15.53:0/1346219610 >> 172.21.15.53:0/1346219610 conn(0x7f44a00f0b70 msgr2=0x7f44a00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:57:16.708 INFO:teuthology.orchestra.run.smithi053.stderr:2024-02-19T21:57:16.706+0000 7f44a7906700 1 -- 172.21.15.53:0/1346219610 shutdown_connections 2024-02-19T21:57:16.708 INFO:teuthology.orchestra.run.smithi053.stderr:2024-02-19T21:57:16.706+0000 7f44a7906700 1 -- 172.21.15.53:0/1346219610 wait complete. 2024-02-19T21:57:16.708 INFO:teuthology.orchestra.run.smithi053.stderr:2024-02-19T21:57:16.707+0000 7f44a7906700 1 Processor -- start 2024-02-19T21:57:16.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-02-19T21:57:16.707+0000 7f44a7906700 1 -- start start 2024-02-19T21:57:16.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-02-19T21:57:16.707+0000 7f44a7906700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f44a00f5410 0x7f44a00fd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:57:16.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-02-19T21:57:16.707+0000 7f44a7906700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44a00fb960 con 0x7f44a00f5410 2024-02-19T21:57:16.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-02-19T21:57:16.707+0000 7f44a56a2700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f44a00f5410 0x7f44a00fd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:57:16.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-02-19T21:57:16.707+0000 7f44a56a2700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f44a00f5410 0x7f44a00fd4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:53682/0 (socket says 172.21.15.53:53682) 2024-02-19T21:57:16.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-02-19T21:57:16.707+0000 7f44a56a2700 1 -- 172.21.15.53:0/2226984826 learned_addr learned my addr 172.21.15.53:0/2226984826 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-02-19T21:57:16.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-02-19T21:57:16.707+0000 7f44a56a2700 1 -- 172.21.15.53:0/2226984826 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4494005040 con 0x7f44a00f5410 2024-02-19T21:57:16.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-02-19T21:57:16.707+0000 7f44a56a2700 1 --2- 172.21.15.53:0/2226984826 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f44a00f5410 0x7f44a00fd4e0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f4494005010 tx=0x7f449400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:57:16.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-02-19T21:57:16.707+0000 7f4491ffb700 1 -- 172.21.15.53:0/2226984826 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4494009320 con 0x7f44a00f5410 2024-02-19T21:57:16.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-02-19T21:57:16.708+0000 7f4491ffb700 1 -- 172.21.15.53:0/2226984826 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4494013380 con 0x7f44a00f5410 2024-02-19T21:57:16.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-02-19T21:57:16.708+0000 7f44a7906700 1 -- 172.21.15.53:0/2226984826 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44a00fbb00 con 0x7f44a00f5410 2024-02-19T21:57:16.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-02-19T21:57:16.708+0000 7f4491ffb700 1 -- 172.21.15.53:0/2226984826 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4494014e50 con 0x7f44a00f5410 2024-02-19T21:57:16.710 INFO:teuthology.orchestra.run.smithi053.stderr:2024-02-19T21:57:16.708+0000 7f44a7906700 1 -- 172.21.15.53:0/2226984826 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44a00fbf90 con 0x7f44a00f5410 2024-02-19T21:57:16.710 INFO:teuthology.orchestra.run.smithi053.stderr:2024-02-19T21:57:16.709+0000 7f44a7906700 1 -- 172.21.15.53:0/2226984826 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44a0059100 con 0x7f44a00f5410 2024-02-19T21:57:16.710 INFO:teuthology.orchestra.run.smithi053.stderr:2024-02-19T21:57:16.709+0000 7f4491ffb700 1 -- 172.21.15.53:0/2226984826 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f44940095f0 con 0x7f44a00f5410 2024-02-19T21:57:16.710 INFO:teuthology.orchestra.run.smithi053.stderr:2024-02-19T21:57:16.709+0000 7f4491ffb700 1 --2- 172.21.15.53:0/2226984826 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f448c02ff10 0x7f448c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:57:16.711 INFO:teuthology.orchestra.run.smithi053.stderr:2024-02-19T21:57:16.709+0000 7f4491ffb700 1 -- 172.21.15.53:0/2226984826 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f44940410a0 con 0x7f44a00f5410 2024-02-19T21:57:16.711 INFO:teuthology.orchestra.run.smithi053.stderr:2024-02-19T21:57:16.709+0000 7f44a4ea1700 1 --2- 172.21.15.53:0/2226984826 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f448c02ff10 0x7f448c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:57:16.711 INFO:teuthology.orchestra.run.smithi053.stderr:2024-02-19T21:57:16.710+0000 7f44a4ea1700 1 --2- 172.21.15.53:0/2226984826 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f448c02ff10 0x7f448c0323d0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f449c0096a0 tx=0x7f449c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:57:16.713 INFO:teuthology.orchestra.run.smithi053.stderr:2024-02-19T21:57:16.713+0000 7f4491ffb700 1 -- 172.21.15.53:0/2226984826 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4494014420 con 0x7f44a00f5410 2024-02-19T21:57:16.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:57:16 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T21:57:15.441989+0000 mgr.smithi053.fyhcdi (mgr.14162) 19 : cluster [DBG] pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:57:16.880 INFO:teuthology.orchestra.run.smithi053.stderr:2024-02-19T21:57:16.880+0000 7f44a7906700 1 -- 172.21.15.53:0/2226984826 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f44a00fafa0 con 0x7f44a00f5410 2024-02-19T21:57:17.503 INFO:teuthology.orchestra.run.smithi053.stderr:2024-02-19T21:57:17.502+0000 7f4491ffb700 1 -- 172.21.15.53:0/2226984826 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v5) v1 ==== 124+0+0 (secure 0 0 0) 0x7f449400f990 con 0x7f44a00f5410 2024-02-19T21:57:17.505 INFO:teuthology.orchestra.run.smithi053.stderr:2024-02-19T21:57:17.504+0000 7f44a7906700 1 -- 172.21.15.53:0/2226984826 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f448c02ff10 msgr2=0x7f448c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:57:17.505 INFO:teuthology.orchestra.run.smithi053.stderr:2024-02-19T21:57:17.504+0000 7f44a7906700 1 --2- 172.21.15.53:0/2226984826 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f448c02ff10 0x7f448c0323d0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f449c0096a0 tx=0x7f449c006b40).stop 2024-02-19T21:57:17.505 INFO:teuthology.orchestra.run.smithi053.stderr:2024-02-19T21:57:17.504+0000 7f44a7906700 1 -- 172.21.15.53:0/2226984826 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f44a00f5410 msgr2=0x7f44a00fd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:57:17.505 INFO:teuthology.orchestra.run.smithi053.stderr:2024-02-19T21:57:17.504+0000 7f44a7906700 1 --2- 172.21.15.53:0/2226984826 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f44a00f5410 0x7f44a00fd4e0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f4494005010 tx=0x7f449400b040).stop 2024-02-19T21:57:17.505 INFO:teuthology.orchestra.run.smithi053.stderr:2024-02-19T21:57:17.505+0000 7f44a7906700 1 -- 172.21.15.53:0/2226984826 shutdown_connections 2024-02-19T21:57:17.505 INFO:teuthology.orchestra.run.smithi053.stderr:2024-02-19T21:57:17.505+0000 7f44a7906700 1 --2- 172.21.15.53:0/2226984826 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f448c02ff10 0x7f448c0323d0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:57:17.505 INFO:teuthology.orchestra.run.smithi053.stderr:2024-02-19T21:57:17.505+0000 7f44a7906700 1 --2- 172.21.15.53:0/2226984826 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f44a00f5410 0x7f44a00fd4e0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:57:17.505 INFO:teuthology.orchestra.run.smithi053.stderr:2024-02-19T21:57:17.505+0000 7f44a7906700 1 -- 172.21.15.53:0/2226984826 >> 172.21.15.53:0/2226984826 conn(0x7f44a00f0b70 msgr2=0x7f44a005fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:57:17.505 INFO:teuthology.orchestra.run.smithi053.stderr:2024-02-19T21:57:17.505+0000 7f44a7906700 1 -- 172.21.15.53:0/2226984826 shutdown_connections 2024-02-19T21:57:17.505 INFO:teuthology.orchestra.run.smithi053.stderr:2024-02-19T21:57:17.505+0000 7f44a7906700 1 -- 172.21.15.53:0/2226984826 wait complete. 2024-02-19T21:57:17.505 INFO:teuthology.orchestra.run.smithi053.stderr:adjusted tunables profile to default 2024-02-19T21:57:17.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:57:17 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:57:16.881569+0000 mon.smithi053 (mon.0) 117 : audit [INF] from='client.? 172.21.15.53:0/2226984826' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-02-19T21:57:18.095 INFO:tasks.cephadm:Adding mon.smithi053 on smithi053 2024-02-19T21:57:18.095 INFO:tasks.cephadm:Adding mon.smithi086 on smithi086 2024-02-19T21:57:18.096 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph orch apply mon '2;smithi053:172.21.15.53=smithi053;smithi086:172.21.15.86=smithi086' 2024-02-19T21:57:18.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:57:18 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T21:57:17.442307+0000 mgr.smithi053.fyhcdi (mgr.14162) 20 : cluster [DBG] pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:57:18.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:57:18 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:57:17.503521+0000 mon.smithi053 (mon.0) 118 : audit [INF] from='client.? 172.21.15.53:0/2226984826' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-02-19T21:57:18.794 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:57:18 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T21:57:17.503579+0000 mon.smithi053 (mon.0) 119 : cluster [DBG] osdmap e5: 0 total, 0 up, 0 in 2024-02-19T21:57:19.966 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:19.961+0000 7f35756e3700 1 -- 172.21.15.86:0/688820278 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f35700f6510 msgr2=0x7f35700f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:57:19.966 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:19.961+0000 7f35756e3700 1 --2- 172.21.15.86:0/688820278 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f35700f6510 0x7f35700f6930 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f3564009fb0 tx=0x7f3564009b30).stop 2024-02-19T21:57:19.966 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:19.961+0000 7f35756e3700 1 -- 172.21.15.86:0/688820278 shutdown_connections 2024-02-19T21:57:19.966 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:19.961+0000 7f35756e3700 1 --2- 172.21.15.86:0/688820278 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f35700f6510 0x7f35700f6930 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:57:19.966 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:19.961+0000 7f35756e3700 1 -- 172.21.15.86:0/688820278 >> 172.21.15.86:0/688820278 conn(0x7f35700f1c70 msgr2=0x7f35700f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:57:19.966 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:19.961+0000 7f35756e3700 1 -- 172.21.15.86:0/688820278 shutdown_connections 2024-02-19T21:57:19.967 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:19.961+0000 7f35756e3700 1 -- 172.21.15.86:0/688820278 wait complete. 2024-02-19T21:57:19.967 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:19.962+0000 7f35756e3700 1 Processor -- start 2024-02-19T21:57:19.967 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:19.962+0000 7f35756e3700 1 -- start start 2024-02-19T21:57:19.967 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:19.962+0000 7f35756e3700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f35700f6510 0x7f35701067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:57:19.967 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:19.962+0000 7f35756e3700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3570106cd0 con 0x7f35700f6510 2024-02-19T21:57:19.967 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:19.962+0000 7f356effd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f35700f6510 0x7f35701067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:57:19.967 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:19.963+0000 7f356effd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f35700f6510 0x7f35701067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:36628/0 (socket says 172.21.15.86:36628) 2024-02-19T21:57:19.967 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:19.963+0000 7f356effd700 1 -- 172.21.15.86:0/1773491625 learned_addr learned my addr 172.21.15.86:0/1773491625 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T21:57:19.968 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:19.963+0000 7f356effd700 1 -- 172.21.15.86:0/1773491625 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3564005040 con 0x7f35700f6510 2024-02-19T21:57:19.968 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:19.963+0000 7f356effd700 1 --2- 172.21.15.86:0/1773491625 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f35700f6510 0x7f35701067e0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f3564005010 tx=0x7f356400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:57:19.968 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:19.963+0000 7f35577fe700 1 -- 172.21.15.86:0/1773491625 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3564009320 con 0x7f35700f6510 2024-02-19T21:57:19.968 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:19.963+0000 7f35756e3700 1 -- 172.21.15.86:0/1773491625 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3570106ed0 con 0x7f35700f6510 2024-02-19T21:57:19.968 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:19.963+0000 7f35577fe700 1 -- 172.21.15.86:0/1773491625 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3564013380 con 0x7f35700f6510 2024-02-19T21:57:19.968 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:19.963+0000 7f35577fe700 1 -- 172.21.15.86:0/1773491625 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3564014e50 con 0x7f35700f6510 2024-02-19T21:57:19.969 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:19.963+0000 7f35756e3700 1 -- 172.21.15.86:0/1773491625 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3570107360 con 0x7f35700f6510 2024-02-19T21:57:19.969 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:19.964+0000 7f35577fe700 1 -- 172.21.15.86:0/1773491625 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f35640263a0 con 0x7f35700f6510 2024-02-19T21:57:19.969 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:19.964+0000 7f35756e3700 1 -- 172.21.15.86:0/1773491625 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3570045ad0 con 0x7f35700f6510 2024-02-19T21:57:19.969 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:19.964+0000 7f35577fe700 1 --2- 172.21.15.86:0/1773491625 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f355802ff10 0x7f35580323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:57:19.969 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:19.964+0000 7f35577fe700 1 -- 172.21.15.86:0/1773491625 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3564041270 con 0x7f35700f6510 2024-02-19T21:57:19.969 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:19.964+0000 7f356e7fc700 1 --2- 172.21.15.86:0/1773491625 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f355802ff10 0x7f35580323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:57:19.969 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:19.965+0000 7f356e7fc700 1 --2- 172.21.15.86:0/1773491625 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f355802ff10 0x7f35580323d0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f35600096a0 tx=0x7f3560006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:57:19.970 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:19.969+0000 7f35577fe700 1 -- 172.21.15.86:0/1773491625 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f35640126f0 con 0x7f35700f6510 2024-02-19T21:57:20.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:20.144+0000 7f35756e3700 1 -- 172.21.15.86:0/1773491625 --> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi053:172.21.15.53=smithi053;smithi086:172.21.15.86=smithi086", "target": ["mon-mgr", ""]}) v1 -- 0x7f35700f2ed0 con 0x7f355802ff10 2024-02-19T21:57:20.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:20.150+0000 7f35577fe700 1 -- 172.21.15.86:0/1773491625 <== mgr.14162 v2:172.21.15.53:6800/1543499597 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f35700f2ed0 con 0x7f355802ff10 2024-02-19T21:57:20.150 INFO:teuthology.orchestra.run.smithi086.stdout:Scheduled mon update... 2024-02-19T21:57:20.151 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:20.151+0000 7f35756e3700 1 -- 172.21.15.86:0/1773491625 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f355802ff10 msgr2=0x7f35580323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:57:20.151 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:20.151+0000 7f35756e3700 1 --2- 172.21.15.86:0/1773491625 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f355802ff10 0x7f35580323d0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f35600096a0 tx=0x7f3560006b40).stop 2024-02-19T21:57:20.151 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:20.151+0000 7f35756e3700 1 -- 172.21.15.86:0/1773491625 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f35700f6510 msgr2=0x7f35701067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:57:20.151 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:20.151+0000 7f35756e3700 1 --2- 172.21.15.86:0/1773491625 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f35700f6510 0x7f35701067e0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f3564005010 tx=0x7f356400b040).stop 2024-02-19T21:57:20.151 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:20.151+0000 7f35756e3700 1 -- 172.21.15.86:0/1773491625 shutdown_connections 2024-02-19T21:57:20.152 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:20.151+0000 7f35756e3700 1 --2- 172.21.15.86:0/1773491625 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f355802ff10 0x7f35580323d0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:57:20.152 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:20.151+0000 7f35756e3700 1 --2- 172.21.15.86:0/1773491625 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f35700f6510 0x7f35701067e0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:57:20.152 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:20.151+0000 7f35756e3700 1 -- 172.21.15.86:0/1773491625 >> 172.21.15.86:0/1773491625 conn(0x7f35700f1c70 msgr2=0x7f35700f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:57:20.152 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:20.151+0000 7f35756e3700 1 -- 172.21.15.86:0/1773491625 shutdown_connections 2024-02-19T21:57:20.152 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:20.151+0000 7f35756e3700 1 -- 172.21.15.86:0/1773491625 wait complete. 2024-02-19T21:57:20.747 DEBUG:teuthology.orchestra.run.smithi086:mon.smithi086> sudo journalctl -f -n 0 -u ceph-867b6b52-cf71-11ee-95bb-87774f69a715@mon.smithi086.service 2024-02-19T21:57:20.749 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T21:57:20.749 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T21:57:20.778 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:-- Logs begin at Mon 2024-02-19 21:44:50 UTC. -- 2024-02-19T21:57:20.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:57:20 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T21:57:19.442532+0000 mgr.smithi053.fyhcdi (mgr.14162) 21 : cluster [DBG] pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:57:20.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:57:20 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:57:20.150350+0000 mon.smithi053 (mon.0) 120 : audit [INF] from='mgr.14162 172.21.15.53:0/2860358250' entity='mgr.smithi053.fyhcdi' 2024-02-19T21:57:21.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:57:21 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:57:20.146280+0000 mgr.smithi053.fyhcdi (mgr.14162) 22 : audit [DBG] from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi053:172.21.15.53=smithi053;smithi086:172.21.15.86=smithi086", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T21:57:21.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:57:21 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cephadm 2024-02-19T21:57:20.147721+0000 mgr.smithi053.fyhcdi (mgr.14162) 23 : cephadm [INF] Saving service mon spec with placement smithi053:172.21.15.53=smithi053;smithi086:172.21.15.86=smithi086;count:2 2024-02-19T21:57:22.631 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:22.629+0000 7f5bc653a700 1 -- 172.21.15.86:0/525227848 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5bc0060280 msgr2=0x7f5bc00f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:57:22.631 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:22.629+0000 7f5bc653a700 1 --2- 172.21.15.86:0/525227848 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5bc0060280 0x7f5bc00f90f0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f5bb0009fb0 tx=0x7f5bb0009b30).stop 2024-02-19T21:57:22.631 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:22.629+0000 7f5bc653a700 1 -- 172.21.15.86:0/525227848 shutdown_connections 2024-02-19T21:57:22.631 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:22.629+0000 7f5bc653a700 1 --2- 172.21.15.86:0/525227848 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5bc0060280 0x7f5bc00f90f0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:57:22.631 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:22.629+0000 7f5bc653a700 1 -- 172.21.15.86:0/525227848 >> 172.21.15.86:0/525227848 conn(0x7f5bc00f0b70 msgr2=0x7f5bc00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:57:22.631 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:22.629+0000 7f5bc653a700 1 -- 172.21.15.86:0/525227848 shutdown_connections 2024-02-19T21:57:22.631 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:22.629+0000 7f5bc653a700 1 -- 172.21.15.86:0/525227848 wait complete. 2024-02-19T21:57:22.631 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:22.630+0000 7f5bc653a700 1 Processor -- start 2024-02-19T21:57:22.631 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:22.630+0000 7f5bc653a700 1 -- start start 2024-02-19T21:57:22.631 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:22.630+0000 7f5bc653a700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5bc0060280 0x7f5bc0106780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:57:22.632 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:22.630+0000 7f5bc653a700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bc0106c70 con 0x7f5bc0060280 2024-02-19T21:57:22.632 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:22.630+0000 7f5bbffff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5bc0060280 0x7f5bc0106780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:57:22.632 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:22.630+0000 7f5bbffff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5bc0060280 0x7f5bc0106780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:36642/0 (socket says 172.21.15.86:36642) 2024-02-19T21:57:22.632 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:22.630+0000 7f5bbffff700 1 -- 172.21.15.86:0/3822976176 learned_addr learned my addr 172.21.15.86:0/3822976176 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T21:57:22.632 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:22.630+0000 7f5bbffff700 1 -- 172.21.15.86:0/3822976176 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bb0005040 con 0x7f5bc0060280 2024-02-19T21:57:22.632 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:22.631+0000 7f5bbffff700 1 --2- 172.21.15.86:0/3822976176 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5bc0060280 0x7f5bc0106780 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f5bb00089f0 tx=0x7f5bb000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:57:22.632 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:22.631+0000 7f5bbcff9700 1 -- 172.21.15.86:0/3822976176 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5bb0009320 con 0x7f5bc0060280 2024-02-19T21:57:22.632 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:22.631+0000 7f5bc653a700 1 -- 172.21.15.86:0/3822976176 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bc0106e70 con 0x7f5bc0060280 2024-02-19T21:57:22.632 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:22.631+0000 7f5bbcff9700 1 -- 172.21.15.86:0/3822976176 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5bb0013380 con 0x7f5bc0060280 2024-02-19T21:57:22.632 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:22.631+0000 7f5bbcff9700 1 -- 172.21.15.86:0/3822976176 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5bb0014e20 con 0x7f5bc0060280 2024-02-19T21:57:22.632 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:22.631+0000 7f5bc653a700 1 -- 172.21.15.86:0/3822976176 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bc0107300 con 0x7f5bc0060280 2024-02-19T21:57:22.633 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:22.632+0000 7f5bbcff9700 1 -- 172.21.15.86:0/3822976176 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5bb00095f0 con 0x7f5bc0060280 2024-02-19T21:57:22.633 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:22.632+0000 7f5bc653a700 1 -- 172.21.15.86:0/3822976176 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5bc0045ad0 con 0x7f5bc0060280 2024-02-19T21:57:22.633 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:22.632+0000 7f5bbcff9700 1 --2- 172.21.15.86:0/3822976176 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f5bac02ff10 0x7f5bac0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:57:22.633 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:22.632+0000 7f5bbcff9700 1 -- 172.21.15.86:0/3822976176 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5bb0040fb0 con 0x7f5bc0060280 2024-02-19T21:57:22.633 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:22.632+0000 7f5bbf7fe700 1 --2- 172.21.15.86:0/3822976176 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f5bac02ff10 0x7f5bac0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:57:22.633 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:22.632+0000 7f5bbf7fe700 1 --2- 172.21.15.86:0/3822976176 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f5bac02ff10 0x7f5bac0323d0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f5ba80096a0 tx=0x7f5ba8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:57:22.636 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:22.637+0000 7f5bbcff9700 1 -- 172.21.15.86:0/3822976176 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5bb006d2b0 con 0x7f5bc0060280 2024-02-19T21:57:22.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:57:22 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T21:57:21.442795+0000 mgr.smithi053.fyhcdi (mgr.14162) 24 : cluster [DBG] pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:57:22.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:22.841+0000 7f5bc653a700 1 -- 172.21.15.86:0/3822976176 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5bc01081b0 con 0x7f5bc0060280 2024-02-19T21:57:22.842 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:22.842+0000 7f5bbcff9700 1 -- 172.21.15.86:0/3822976176 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5bb001d980 con 0x7f5bc0060280 2024-02-19T21:57:22.842 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T21:57:22.842 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T21:57:22.844 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:22.843+0000 7f5bc653a700 1 -- 172.21.15.86:0/3822976176 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f5bac02ff10 msgr2=0x7f5bac0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:57:22.844 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:22.843+0000 7f5bc653a700 1 --2- 172.21.15.86:0/3822976176 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f5bac02ff10 0x7f5bac0323d0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f5ba80096a0 tx=0x7f5ba8006b40).stop 2024-02-19T21:57:22.844 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:22.844+0000 7f5bc653a700 1 -- 172.21.15.86:0/3822976176 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5bc0060280 msgr2=0x7f5bc0106780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:57:22.844 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:22.844+0000 7f5bc653a700 1 --2- 172.21.15.86:0/3822976176 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5bc0060280 0x7f5bc0106780 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f5bb00089f0 tx=0x7f5bb000b040).stop 2024-02-19T21:57:22.844 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:22.844+0000 7f5bc653a700 1 -- 172.21.15.86:0/3822976176 shutdown_connections 2024-02-19T21:57:22.844 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:22.844+0000 7f5bc653a700 1 --2- 172.21.15.86:0/3822976176 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f5bac02ff10 0x7f5bac0323d0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:57:22.844 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:22.844+0000 7f5bc653a700 1 --2- 172.21.15.86:0/3822976176 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5bc0060280 0x7f5bc0106780 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:57:22.844 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:22.844+0000 7f5bc653a700 1 -- 172.21.15.86:0/3822976176 >> 172.21.15.86:0/3822976176 conn(0x7f5bc00f0b70 msgr2=0x7f5bc00f15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:57:22.844 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:22.844+0000 7f5bc653a700 1 -- 172.21.15.86:0/3822976176 shutdown_connections 2024-02-19T21:57:22.844 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:22.844+0000 7f5bc653a700 1 -- 172.21.15.86:0/3822976176 wait complete. 2024-02-19T21:57:22.845 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T21:57:23.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:57:23 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:57:22.843279+0000 mon.smithi053 (mon.0) 121 : audit [DBG] from='client.? 172.21.15.86:0/3822976176' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T21:57:24.503 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T21:57:24.503 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T21:57:24.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:57:24 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T21:57:23.443182+0000 mgr.smithi053.fyhcdi (mgr.14162) 25 : cluster [DBG] pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:57:26.306 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:26.304+0000 7f90ae399700 1 -- 172.21.15.86:0/1707808808 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f90a80f5410 msgr2=0x7f90a80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:57:26.307 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:26.304+0000 7f90ae399700 1 --2- 172.21.15.86:0/1707808808 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f90a80f5410 0x7f90a80f5830 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7f9098009fb0 tx=0x7f9098009b30).stop 2024-02-19T21:57:26.307 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:26.305+0000 7f90ae399700 1 -- 172.21.15.86:0/1707808808 shutdown_connections 2024-02-19T21:57:26.307 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:26.305+0000 7f90ae399700 1 --2- 172.21.15.86:0/1707808808 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f90a80f5410 0x7f90a80f5830 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:57:26.307 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:26.305+0000 7f90ae399700 1 -- 172.21.15.86:0/1707808808 >> 172.21.15.86:0/1707808808 conn(0x7f90a80f0b70 msgr2=0x7f90a80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:57:26.307 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:26.305+0000 7f90ae399700 1 -- 172.21.15.86:0/1707808808 shutdown_connections 2024-02-19T21:57:26.307 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:26.305+0000 7f90ae399700 1 -- 172.21.15.86:0/1707808808 wait complete. 2024-02-19T21:57:26.307 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:26.305+0000 7f90ae399700 1 Processor -- start 2024-02-19T21:57:26.307 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:26.305+0000 7f90ae399700 1 -- start start 2024-02-19T21:57:26.307 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:26.305+0000 7f90ae399700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f90a80f5410 0x7f90a8109510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:57:26.307 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:26.305+0000 7f90ae399700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90a8060240 con 0x7f90a80f5410 2024-02-19T21:57:26.307 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:26.306+0000 7f90a7fff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f90a80f5410 0x7f90a8109510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:57:26.307 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:26.306+0000 7f90a7fff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f90a80f5410 0x7f90a8109510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:36668/0 (socket says 172.21.15.86:36668) 2024-02-19T21:57:26.307 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:26.306+0000 7f90a7fff700 1 -- 172.21.15.86:0/606182687 learned_addr learned my addr 172.21.15.86:0/606182687 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T21:57:26.308 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:26.306+0000 7f90a7fff700 1 -- 172.21.15.86:0/606182687 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9098005040 con 0x7f90a80f5410 2024-02-19T21:57:26.308 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:26.306+0000 7f90a7fff700 1 --2- 172.21.15.86:0/606182687 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f90a80f5410 0x7f90a8109510 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f9098005010 tx=0x7f909800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:57:26.308 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:26.306+0000 7f90a4ff9700 1 -- 172.21.15.86:0/606182687 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9098009320 con 0x7f90a80f5410 2024-02-19T21:57:26.308 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:26.306+0000 7f90ae399700 1 -- 172.21.15.86:0/606182687 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90a8060440 con 0x7f90a80f5410 2024-02-19T21:57:26.308 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:26.306+0000 7f90a4ff9700 1 -- 172.21.15.86:0/606182687 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9098013380 con 0x7f90a80f5410 2024-02-19T21:57:26.308 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:26.306+0000 7f90a4ff9700 1 -- 172.21.15.86:0/606182687 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9098014e50 con 0x7f90a80f5410 2024-02-19T21:57:26.308 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:26.306+0000 7f90ae399700 1 -- 172.21.15.86:0/606182687 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90a8109bc0 con 0x7f90a80f5410 2024-02-19T21:57:26.309 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:26.307+0000 7f90a4ff9700 1 -- 172.21.15.86:0/606182687 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f90980095f0 con 0x7f90a80f5410 2024-02-19T21:57:26.309 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:26.307+0000 7f90ae399700 1 -- 172.21.15.86:0/606182687 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90a8045ad0 con 0x7f90a80f5410 2024-02-19T21:57:26.309 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:26.307+0000 7f90a4ff9700 1 --2- 172.21.15.86:0/606182687 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f909002ff10 0x7f90900323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:57:26.309 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:26.307+0000 7f90a4ff9700 1 -- 172.21.15.86:0/606182687 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9098040ef0 con 0x7f90a80f5410 2024-02-19T21:57:26.309 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:26.308+0000 7f90a77fe700 1 --2- 172.21.15.86:0/606182687 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f909002ff10 0x7f90900323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:57:26.309 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:26.308+0000 7f90a77fe700 1 --2- 172.21.15.86:0/606182687 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f909002ff10 0x7f90900323d0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f909c0096a0 tx=0x7f909c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:57:26.312 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:26.312+0000 7f90a4ff9700 1 -- 172.21.15.86:0/606182687 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f909801dbc0 con 0x7f90a80f5410 2024-02-19T21:57:26.517 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:26.517+0000 7f90ae399700 1 -- 172.21.15.86:0/606182687 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f90a810a3d0 con 0x7f90a80f5410 2024-02-19T21:57:26.518 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:26.518+0000 7f90a4ff9700 1 -- 172.21.15.86:0/606182687 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9098010230 con 0x7f90a80f5410 2024-02-19T21:57:26.518 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T21:57:26.519 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T21:57:26.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:26.519+0000 7f90ae399700 1 -- 172.21.15.86:0/606182687 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f909002ff10 msgr2=0x7f90900323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:57:26.520 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:26.519+0000 7f90ae399700 1 --2- 172.21.15.86:0/606182687 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f909002ff10 0x7f90900323d0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f909c0096a0 tx=0x7f909c006b40).stop 2024-02-19T21:57:26.520 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:26.519+0000 7f90ae399700 1 -- 172.21.15.86:0/606182687 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f90a80f5410 msgr2=0x7f90a8109510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:57:26.520 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:26.519+0000 7f90ae399700 1 --2- 172.21.15.86:0/606182687 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f90a80f5410 0x7f90a8109510 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f9098005010 tx=0x7f909800b040).stop 2024-02-19T21:57:26.520 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:26.520+0000 7f90ae399700 1 -- 172.21.15.86:0/606182687 shutdown_connections 2024-02-19T21:57:26.520 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:26.520+0000 7f90ae399700 1 --2- 172.21.15.86:0/606182687 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f909002ff10 0x7f90900323d0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:57:26.520 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:26.520+0000 7f90ae399700 1 --2- 172.21.15.86:0/606182687 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f90a80f5410 0x7f90a8109510 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:57:26.520 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:26.520+0000 7f90ae399700 1 -- 172.21.15.86:0/606182687 >> 172.21.15.86:0/606182687 conn(0x7f90a80f0b70 msgr2=0x7f90a80f15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:57:26.520 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:26.520+0000 7f90ae399700 1 -- 172.21.15.86:0/606182687 shutdown_connections 2024-02-19T21:57:26.520 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:26.520+0000 7f90ae399700 1 -- 172.21.15.86:0/606182687 wait complete. 2024-02-19T21:57:26.521 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T21:57:26.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:57:26 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T21:57:25.443558+0000 mgr.smithi053.fyhcdi (mgr.14162) 26 : cluster [DBG] pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:57:27.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:57:27 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:57:26.519255+0000 mon.smithi053 (mon.0) 122 : audit [DBG] from='client.? 172.21.15.86:0/606182687' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T21:57:28.197 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T21:57:28.197 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T21:57:28.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:57:28 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T21:57:27.443938+0000 mgr.smithi053.fyhcdi (mgr.14162) 27 : cluster [DBG] pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:57:29.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:29.854+0000 7f16418b4700 1 -- 172.21.15.86:0/545815073 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f163c0f4880 msgr2=0x7f163c0f4ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:57:29.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:29.854+0000 7f16418b4700 1 --2- 172.21.15.86:0/545815073 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f163c0f4880 0x7f163c0f4ca0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f1630009fb0 tx=0x7f1630009b30).stop 2024-02-19T21:57:29.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:29.854+0000 7f16418b4700 1 -- 172.21.15.86:0/545815073 shutdown_connections 2024-02-19T21:57:29.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:29.854+0000 7f16418b4700 1 --2- 172.21.15.86:0/545815073 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f163c0f4880 0x7f163c0f4ca0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:57:29.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:29.854+0000 7f16418b4700 1 -- 172.21.15.86:0/545815073 >> 172.21.15.86:0/545815073 conn(0x7f163c0f02e0 msgr2=0x7f163c0f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:57:29.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:29.854+0000 7f16418b4700 1 -- 172.21.15.86:0/545815073 shutdown_connections 2024-02-19T21:57:29.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:29.854+0000 7f16418b4700 1 -- 172.21.15.86:0/545815073 wait complete. 2024-02-19T21:57:29.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:29.855+0000 7f16418b4700 1 Processor -- start 2024-02-19T21:57:29.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:29.855+0000 7f16418b4700 1 -- start start 2024-02-19T21:57:29.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:29.855+0000 7f16418b4700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f163c0f4880 0x7f163c1067c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:57:29.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:29.855+0000 7f16418b4700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f163c106cb0 con 0x7f163c0f4880 2024-02-19T21:57:29.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:29.855+0000 7f163affd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f163c0f4880 0x7f163c1067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:57:29.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:29.855+0000 7f163affd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f163c0f4880 0x7f163c1067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:48806/0 (socket says 172.21.15.86:48806) 2024-02-19T21:57:29.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:29.855+0000 7f163affd700 1 -- 172.21.15.86:0/3397018361 learned_addr learned my addr 172.21.15.86:0/3397018361 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T21:57:29.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:29.856+0000 7f163affd700 1 -- 172.21.15.86:0/3397018361 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1630005040 con 0x7f163c0f4880 2024-02-19T21:57:29.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:29.856+0000 7f163affd700 1 --2- 172.21.15.86:0/3397018361 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f163c0f4880 0x7f163c1067c0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f16300089f0 tx=0x7f163000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:57:29.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:29.856+0000 7f16237fe700 1 -- 172.21.15.86:0/3397018361 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1630009320 con 0x7f163c0f4880 2024-02-19T21:57:29.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:29.856+0000 7f16237fe700 1 -- 172.21.15.86:0/3397018361 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1630013380 con 0x7f163c0f4880 2024-02-19T21:57:29.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:29.856+0000 7f16237fe700 1 -- 172.21.15.86:0/3397018361 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1630014e20 con 0x7f163c0f4880 2024-02-19T21:57:29.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:29.856+0000 7f16418b4700 1 -- 172.21.15.86:0/3397018361 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f163c106eb0 con 0x7f163c0f4880 2024-02-19T21:57:29.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:29.856+0000 7f16418b4700 1 -- 172.21.15.86:0/3397018361 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f163c107340 con 0x7f163c0f4880 2024-02-19T21:57:29.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:29.857+0000 7f16237fe700 1 -- 172.21.15.86:0/3397018361 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f16300095f0 con 0x7f163c0f4880 2024-02-19T21:57:29.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:29.857+0000 7f16418b4700 1 -- 172.21.15.86:0/3397018361 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f163c045ad0 con 0x7f163c0f4880 2024-02-19T21:57:29.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:29.857+0000 7f16237fe700 1 --2- 172.21.15.86:0/3397018361 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f162402ffb0 0x7f1624032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:57:29.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:29.857+0000 7f16237fe700 1 -- 172.21.15.86:0/3397018361 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1630040f00 con 0x7f163c0f4880 2024-02-19T21:57:29.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:29.857+0000 7f163a7fc700 1 --2- 172.21.15.86:0/3397018361 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f162402ffb0 0x7f1624032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:57:29.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:29.858+0000 7f163a7fc700 1 --2- 172.21.15.86:0/3397018361 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f162402ffb0 0x7f1624032470 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f162c0096a0 tx=0x7f162c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:57:29.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:29.862+0000 7f16237fe700 1 -- 172.21.15.86:0/3397018361 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f163001d7f0 con 0x7f163c0f4880 2024-02-19T21:57:30.067 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:30.068+0000 7f16418b4700 1 -- 172.21.15.86:0/3397018361 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f163c1081f0 con 0x7f163c0f4880 2024-02-19T21:57:30.068 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:30.068+0000 7f16237fe700 1 -- 172.21.15.86:0/3397018361 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1630010230 con 0x7f163c0f4880 2024-02-19T21:57:30.068 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T21:57:30.068 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T21:57:30.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:30.070+0000 7f16418b4700 1 -- 172.21.15.86:0/3397018361 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f162402ffb0 msgr2=0x7f1624032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:57:30.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:30.070+0000 7f16418b4700 1 --2- 172.21.15.86:0/3397018361 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f162402ffb0 0x7f1624032470 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f162c0096a0 tx=0x7f162c006b40).stop 2024-02-19T21:57:30.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:30.070+0000 7f16418b4700 1 -- 172.21.15.86:0/3397018361 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f163c0f4880 msgr2=0x7f163c1067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:57:30.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:30.070+0000 7f16418b4700 1 --2- 172.21.15.86:0/3397018361 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f163c0f4880 0x7f163c1067c0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f16300089f0 tx=0x7f163000b040).stop 2024-02-19T21:57:30.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:30.070+0000 7f16418b4700 1 -- 172.21.15.86:0/3397018361 shutdown_connections 2024-02-19T21:57:30.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:30.070+0000 7f16418b4700 1 --2- 172.21.15.86:0/3397018361 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f162402ffb0 0x7f1624032470 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:57:30.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:30.070+0000 7f16418b4700 1 --2- 172.21.15.86:0/3397018361 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f163c0f4880 0x7f163c1067c0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:57:30.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:30.070+0000 7f16418b4700 1 -- 172.21.15.86:0/3397018361 >> 172.21.15.86:0/3397018361 conn(0x7f163c0f02e0 msgr2=0x7f163c0f0d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:57:30.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:30.070+0000 7f16418b4700 1 -- 172.21.15.86:0/3397018361 shutdown_connections 2024-02-19T21:57:30.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:30.070+0000 7f16418b4700 1 -- 172.21.15.86:0/3397018361 wait complete. 2024-02-19T21:57:30.071 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T21:57:30.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:57:30 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T21:57:29.444273+0000 mgr.smithi053.fyhcdi (mgr.14162) 28 : cluster [DBG] pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:57:30.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:57:30 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:57:30.069736+0000 mon.smithi053 (mon.0) 123 : audit [DBG] from='client.? 172.21.15.86:0/3397018361' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T21:57:31.661 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T21:57:31.661 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T21:57:32.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:57:32 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T21:57:31.444655+0000 mgr.smithi053.fyhcdi (mgr.14162) 29 : cluster [DBG] pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:57:34.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:57:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T21:57:33.445048+0000 mgr.smithi053.fyhcdi (mgr.14162) 30 : cluster [DBG] pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:57:36.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:57:36 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T21:57:35.445301+0000 mgr.smithi053.fyhcdi (mgr.14162) 31 : cluster [DBG] pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:57:36.794 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:57:36 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:57:35.596753+0000 mon.smithi053 (mon.0) 124 : audit [INF] from='mgr.14162 172.21.15.53:0/2860358250' entity='mgr.smithi053.fyhcdi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi053.fyhcdi/mirror_snapshot_schedule"}]: dispatch 2024-02-19T21:57:36.794 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:57:36 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:57:35.600504+0000 mon.smithi053 (mon.0) 125 : audit [INF] from='mgr.14162 172.21.15.53:0/2860358250' entity='mgr.smithi053.fyhcdi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi053.fyhcdi/trash_purge_schedule"}]: dispatch 2024-02-19T21:57:38.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:57:38 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T21:57:37.445642+0000 mgr.smithi053.fyhcdi (mgr.14162) 32 : cluster [DBG] pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:57:40.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:57:40 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T21:57:39.445964+0000 mgr.smithi053.fyhcdi (mgr.14162) 33 : cluster [DBG] pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:57:42.767 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:42.765+0000 7f6a18047700 1 -- 172.21.15.86:0/3548222876 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6a100f6510 msgr2=0x7f6a100f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:57:42.767 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:42.765+0000 7f6a18047700 1 --2- 172.21.15.86:0/3548222876 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6a100f6510 0x7f6a100f6930 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f6a08009960 tx=0x7f6a08004d10).stop 2024-02-19T21:57:42.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:42.765+0000 7f6a18047700 1 -- 172.21.15.86:0/3548222876 shutdown_connections 2024-02-19T21:57:42.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:42.765+0000 7f6a18047700 1 --2- 172.21.15.86:0/3548222876 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6a100f6510 0x7f6a100f6930 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:57:42.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:42.765+0000 7f6a18047700 1 -- 172.21.15.86:0/3548222876 >> 172.21.15.86:0/3548222876 conn(0x7f6a100f1c70 msgr2=0x7f6a100f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:57:42.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:42.765+0000 7f6a18047700 1 -- 172.21.15.86:0/3548222876 shutdown_connections 2024-02-19T21:57:42.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:42.766+0000 7f6a18047700 1 -- 172.21.15.86:0/3548222876 wait complete. 2024-02-19T21:57:42.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:42.766+0000 7f6a18047700 1 Processor -- start 2024-02-19T21:57:42.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:42.766+0000 7f6a18047700 1 -- start start 2024-02-19T21:57:42.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:42.766+0000 7f6a18047700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6a100f6510 0x7f6a10108a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:57:42.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:42.766+0000 7f6a18047700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a10108f00 con 0x7f6a100f6510 2024-02-19T21:57:42.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:42.766+0000 7f6a15de3700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6a100f6510 0x7f6a10108a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:57:42.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:42.766+0000 7f6a15de3700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6a100f6510 0x7f6a10108a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:53670/0 (socket says 172.21.15.86:53670) 2024-02-19T21:57:42.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:42.766+0000 7f6a15de3700 1 -- 172.21.15.86:0/4141544167 learned_addr learned my addr 172.21.15.86:0/4141544167 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T21:57:42.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:42.767+0000 7f6a15de3700 1 -- 172.21.15.86:0/4141544167 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a08009640 con 0x7f6a100f6510 2024-02-19T21:57:42.769 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:42.767+0000 7f6a15de3700 1 --2- 172.21.15.86:0/4141544167 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6a100f6510 0x7f6a10108a10 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f6a08004a10 tx=0x7f6a08004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:57:42.769 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:42.767+0000 7f6a067fc700 1 -- 172.21.15.86:0/4141544167 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6a08018070 con 0x7f6a100f6510 2024-02-19T21:57:42.769 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:42.767+0000 7f6a18047700 1 -- 172.21.15.86:0/4141544167 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a10109100 con 0x7f6a100f6510 2024-02-19T21:57:42.769 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:42.767+0000 7f6a067fc700 1 -- 172.21.15.86:0/4141544167 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6a08007e70 con 0x7f6a100f6510 2024-02-19T21:57:42.769 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:42.767+0000 7f6a067fc700 1 -- 172.21.15.86:0/4141544167 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6a0801d420 con 0x7f6a100f6510 2024-02-19T21:57:42.769 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:42.767+0000 7f6a18047700 1 -- 172.21.15.86:0/4141544167 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a10109590 con 0x7f6a100f6510 2024-02-19T21:57:42.770 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:42.768+0000 7f6a067fc700 1 -- 172.21.15.86:0/4141544167 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6a0801d6e0 con 0x7f6a100f6510 2024-02-19T21:57:42.770 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:42.768+0000 7f6a18047700 1 -- 172.21.15.86:0/4141544167 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a10045ad0 con 0x7f6a100f6510 2024-02-19T21:57:42.770 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:42.768+0000 7f6a067fc700 1 --2- 172.21.15.86:0/4141544167 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f69fc02ff60 0x7f69fc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:57:42.770 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:42.768+0000 7f6a067fc700 1 -- 172.21.15.86:0/4141544167 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6a080416e0 con 0x7f6a100f6510 2024-02-19T21:57:42.770 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:42.768+0000 7f6a155e2700 1 --2- 172.21.15.86:0/4141544167 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f69fc02ff60 0x7f69fc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:57:42.770 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:42.769+0000 7f6a155e2700 1 --2- 172.21.15.86:0/4141544167 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f69fc02ff60 0x7f69fc032420 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f6a000096a0 tx=0x7f6a00006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:57:42.772 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:42.774+0000 7f6a067fc700 1 -- 172.21.15.86:0/4141544167 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6a0801daa0 con 0x7f6a100f6510 2024-02-19T21:57:42.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:57:42 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T21:57:41.446351+0000 mgr.smithi053.fyhcdi (mgr.14162) 34 : cluster [DBG] pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:57:42.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:42.978+0000 7f6a18047700 1 -- 172.21.15.86:0/4141544167 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6a1006b150 con 0x7f6a100f6510 2024-02-19T21:57:42.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:42.978+0000 7f6a067fc700 1 -- 172.21.15.86:0/4141544167 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6a08011320 con 0x7f6a100f6510 2024-02-19T21:57:42.979 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T21:57:42.979 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T21:57:42.980 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:42.980+0000 7f6a18047700 1 -- 172.21.15.86:0/4141544167 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f69fc02ff60 msgr2=0x7f69fc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:57:42.980 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:42.980+0000 7f6a18047700 1 --2- 172.21.15.86:0/4141544167 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f69fc02ff60 0x7f69fc032420 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f6a000096a0 tx=0x7f6a00006b40).stop 2024-02-19T21:57:42.980 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:42.980+0000 7f6a18047700 1 -- 172.21.15.86:0/4141544167 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6a100f6510 msgr2=0x7f6a10108a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:57:42.980 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:42.980+0000 7f6a18047700 1 --2- 172.21.15.86:0/4141544167 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6a100f6510 0x7f6a10108a10 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f6a08004a10 tx=0x7f6a08004d10).stop 2024-02-19T21:57:42.980 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:42.980+0000 7f6a18047700 1 -- 172.21.15.86:0/4141544167 shutdown_connections 2024-02-19T21:57:42.980 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:42.980+0000 7f6a18047700 1 --2- 172.21.15.86:0/4141544167 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f69fc02ff60 0x7f69fc032420 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:57:42.980 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:42.980+0000 7f6a18047700 1 --2- 172.21.15.86:0/4141544167 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6a100f6510 0x7f6a10108a10 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:57:42.980 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:42.980+0000 7f6a18047700 1 -- 172.21.15.86:0/4141544167 >> 172.21.15.86:0/4141544167 conn(0x7f6a100f1c70 msgr2=0x7f6a100f2ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:57:42.980 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:42.980+0000 7f6a18047700 1 -- 172.21.15.86:0/4141544167 shutdown_connections 2024-02-19T21:57:42.980 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:42.980+0000 7f6a18047700 1 -- 172.21.15.86:0/4141544167 wait complete. 2024-02-19T21:57:42.981 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T21:57:43.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:57:43 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:57:42.979720+0000 mon.smithi053 (mon.0) 126 : audit [DBG] from='client.? 172.21.15.86:0/4141544167' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T21:57:44.497 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T21:57:44.498 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T21:57:44.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:57:44 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T21:57:43.446636+0000 mgr.smithi053.fyhcdi (mgr.14162) 35 : cluster [DBG] pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:57:46.218 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:46.217+0000 7f2532507700 1 -- 172.21.15.86:0/4190414993 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f252c0f4740 msgr2=0x7f252c0f4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:57:46.218 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:46.217+0000 7f2532507700 1 --2- 172.21.15.86:0/4190414993 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f252c0f4740 0x7f252c0f4b60 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f2520009fb0 tx=0x7f2520009b30).stop 2024-02-19T21:57:46.218 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:46.217+0000 7f2532507700 1 -- 172.21.15.86:0/4190414993 shutdown_connections 2024-02-19T21:57:46.218 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:46.217+0000 7f2532507700 1 --2- 172.21.15.86:0/4190414993 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f252c0f4740 0x7f252c0f4b60 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:57:46.218 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:46.217+0000 7f2532507700 1 -- 172.21.15.86:0/4190414993 >> 172.21.15.86:0/4190414993 conn(0x7f252c0f0260 msgr2=0x7f252c0f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:57:46.218 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:46.217+0000 7f2532507700 1 -- 172.21.15.86:0/4190414993 shutdown_connections 2024-02-19T21:57:46.218 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:46.217+0000 7f2532507700 1 -- 172.21.15.86:0/4190414993 wait complete. 2024-02-19T21:57:46.218 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:46.218+0000 7f2532507700 1 Processor -- start 2024-02-19T21:57:46.218 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:46.218+0000 7f2532507700 1 -- start start 2024-02-19T21:57:46.218 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:46.218+0000 7f2532507700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f252c0f4740 0x7f252c0f72f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:57:46.218 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:46.218+0000 7f2532507700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f252c0f5770 con 0x7f252c0f4740 2024-02-19T21:57:46.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:46.218+0000 7f252bfff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f252c0f4740 0x7f252c0f72f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:57:46.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:46.218+0000 7f252bfff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f252c0f4740 0x7f252c0f72f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:53688/0 (socket says 172.21.15.86:53688) 2024-02-19T21:57:46.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:46.218+0000 7f252bfff700 1 -- 172.21.15.86:0/1920369254 learned_addr learned my addr 172.21.15.86:0/1920369254 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T21:57:46.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:46.218+0000 7f252bfff700 1 -- 172.21.15.86:0/1920369254 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2520005040 con 0x7f252c0f4740 2024-02-19T21:57:46.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:46.219+0000 7f252bfff700 1 --2- 172.21.15.86:0/1920369254 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f252c0f4740 0x7f252c0f72f0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f2520005010 tx=0x7f252000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:57:46.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:46.219+0000 7f2528ff9700 1 -- 172.21.15.86:0/1920369254 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2520009320 con 0x7f252c0f4740 2024-02-19T21:57:46.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:46.219+0000 7f2532507700 1 -- 172.21.15.86:0/1920369254 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f252c0f5910 con 0x7f252c0f4740 2024-02-19T21:57:46.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:46.219+0000 7f2528ff9700 1 -- 172.21.15.86:0/1920369254 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2520013380 con 0x7f252c0f4740 2024-02-19T21:57:46.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:46.219+0000 7f2528ff9700 1 -- 172.21.15.86:0/1920369254 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2520014e50 con 0x7f252c0f4740 2024-02-19T21:57:46.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:46.219+0000 7f2532507700 1 -- 172.21.15.86:0/1920369254 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f252c0f5da0 con 0x7f252c0f4740 2024-02-19T21:57:46.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:46.220+0000 7f2528ff9700 1 -- 172.21.15.86:0/1920369254 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f25200095f0 con 0x7f252c0f4740 2024-02-19T21:57:46.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:46.220+0000 7f2532507700 1 -- 172.21.15.86:0/1920369254 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f252c059100 con 0x7f252c0f4740 2024-02-19T21:57:46.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:46.220+0000 7f2528ff9700 1 --2- 172.21.15.86:0/1920369254 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f251402ff10 0x7f25140323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:57:46.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:46.220+0000 7f2528ff9700 1 -- 172.21.15.86:0/1920369254 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2520040f80 con 0x7f252c0f4740 2024-02-19T21:57:46.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:46.220+0000 7f252b7fe700 1 --2- 172.21.15.86:0/1920369254 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f251402ff10 0x7f25140323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:57:46.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:46.220+0000 7f252b7fe700 1 --2- 172.21.15.86:0/1920369254 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f251402ff10 0x7f25140323d0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7f251c0096a0 tx=0x7f251c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:57:46.223 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:46.224+0000 7f2528ff9700 1 -- 172.21.15.86:0/1920369254 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2520014420 con 0x7f252c0f4740 2024-02-19T21:57:46.428 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:46.430+0000 7f2532507700 1 -- 172.21.15.86:0/1920369254 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f252c045ad0 con 0x7f252c0f4740 2024-02-19T21:57:46.429 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:46.430+0000 7f2528ff9700 1 -- 172.21.15.86:0/1920369254 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2520010230 con 0x7f252c0f4740 2024-02-19T21:57:46.429 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T21:57:46.430 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T21:57:46.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:46.432+0000 7f2532507700 1 -- 172.21.15.86:0/1920369254 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f251402ff10 msgr2=0x7f25140323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:57:46.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:46.432+0000 7f2532507700 1 --2- 172.21.15.86:0/1920369254 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f251402ff10 0x7f25140323d0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7f251c0096a0 tx=0x7f251c006b40).stop 2024-02-19T21:57:46.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:46.432+0000 7f2532507700 1 -- 172.21.15.86:0/1920369254 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f252c0f4740 msgr2=0x7f252c0f72f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:57:46.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:46.432+0000 7f2532507700 1 --2- 172.21.15.86:0/1920369254 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f252c0f4740 0x7f252c0f72f0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f2520005010 tx=0x7f252000b040).stop 2024-02-19T21:57:46.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:46.432+0000 7f2532507700 1 -- 172.21.15.86:0/1920369254 shutdown_connections 2024-02-19T21:57:46.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:46.432+0000 7f2532507700 1 --2- 172.21.15.86:0/1920369254 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f251402ff10 0x7f25140323d0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:57:46.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:46.432+0000 7f2532507700 1 --2- 172.21.15.86:0/1920369254 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f252c0f4740 0x7f252c0f72f0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:57:46.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:46.432+0000 7f2532507700 1 -- 172.21.15.86:0/1920369254 >> 172.21.15.86:0/1920369254 conn(0x7f252c0f0260 msgr2=0x7f252c0f0cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:57:46.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:46.432+0000 7f2532507700 1 -- 172.21.15.86:0/1920369254 shutdown_connections 2024-02-19T21:57:46.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:46.432+0000 7f2532507700 1 -- 172.21.15.86:0/1920369254 wait complete. 2024-02-19T21:57:46.432 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T21:57:46.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:57:46 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T21:57:45.446958+0000 mgr.smithi053.fyhcdi (mgr.14162) 36 : cluster [DBG] pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:57:46.794 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:57:46 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:57:46.430433+0000 mon.smithi053 (mon.0) 127 : audit [DBG] from='client.? 172.21.15.86:0/1920369254' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T21:57:47.991 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T21:57:47.992 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T21:57:48.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:57:48 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T21:57:47.447265+0000 mgr.smithi053.fyhcdi (mgr.14162) 37 : cluster [DBG] pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:57:49.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:49.852+0000 7fef884d8700 1 -- 172.21.15.86:0/3696472968 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fef800f6510 msgr2=0x7fef800f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:57:49.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:49.852+0000 7fef884d8700 1 --2- 172.21.15.86:0/3696472968 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fef800f6510 0x7fef800f6930 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7fef7c009fb0 tx=0x7fef7c009b30).stop 2024-02-19T21:57:49.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:49.852+0000 7fef884d8700 1 -- 172.21.15.86:0/3696472968 shutdown_connections 2024-02-19T21:57:49.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:49.852+0000 7fef884d8700 1 --2- 172.21.15.86:0/3696472968 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fef800f6510 0x7fef800f6930 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:57:49.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:49.852+0000 7fef884d8700 1 -- 172.21.15.86:0/3696472968 >> 172.21.15.86:0/3696472968 conn(0x7fef800f1c70 msgr2=0x7fef800f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:57:49.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:49.852+0000 7fef884d8700 1 -- 172.21.15.86:0/3696472968 shutdown_connections 2024-02-19T21:57:49.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:49.852+0000 7fef884d8700 1 -- 172.21.15.86:0/3696472968 wait complete. 2024-02-19T21:57:49.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:49.852+0000 7fef884d8700 1 Processor -- start 2024-02-19T21:57:49.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:49.852+0000 7fef884d8700 1 -- start start 2024-02-19T21:57:49.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:49.853+0000 7fef884d8700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fef800f6510 0x7fef801067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:57:49.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:49.853+0000 7fef884d8700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef80106cd0 con 0x7fef800f6510 2024-02-19T21:57:49.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:49.853+0000 7fef86274700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fef800f6510 0x7fef801067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:57:49.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:49.853+0000 7fef86274700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fef800f6510 0x7fef801067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:32870/0 (socket says 172.21.15.86:32870) 2024-02-19T21:57:49.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:49.853+0000 7fef86274700 1 -- 172.21.15.86:0/3563131081 learned_addr learned my addr 172.21.15.86:0/3563131081 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T21:57:49.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:49.853+0000 7fef86274700 1 -- 172.21.15.86:0/3563131081 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef7c005040 con 0x7fef800f6510 2024-02-19T21:57:49.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:49.853+0000 7fef86274700 1 --2- 172.21.15.86:0/3563131081 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fef800f6510 0x7fef801067e0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7fef7c005010 tx=0x7fef7c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:57:49.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:49.853+0000 7fef76ffd700 1 -- 172.21.15.86:0/3563131081 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fef7c009320 con 0x7fef800f6510 2024-02-19T21:57:49.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:49.853+0000 7fef884d8700 1 -- 172.21.15.86:0/3563131081 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef80106ed0 con 0x7fef800f6510 2024-02-19T21:57:49.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:49.853+0000 7fef76ffd700 1 -- 172.21.15.86:0/3563131081 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fef7c013380 con 0x7fef800f6510 2024-02-19T21:57:49.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:49.853+0000 7fef76ffd700 1 -- 172.21.15.86:0/3563131081 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fef7c014e50 con 0x7fef800f6510 2024-02-19T21:57:49.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:49.853+0000 7fef884d8700 1 -- 172.21.15.86:0/3563131081 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef80107360 con 0x7fef800f6510 2024-02-19T21:57:49.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:49.854+0000 7fef76ffd700 1 -- 172.21.15.86:0/3563131081 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fef7c0095f0 con 0x7fef800f6510 2024-02-19T21:57:49.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:49.854+0000 7fef76ffd700 1 --2- 172.21.15.86:0/3563131081 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fef6c02ff10 0x7fef6c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:57:49.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:49.854+0000 7fef884d8700 1 -- 172.21.15.86:0/3563131081 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef80045ad0 con 0x7fef800f6510 2024-02-19T21:57:49.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:49.854+0000 7fef76ffd700 1 -- 172.21.15.86:0/3563131081 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fef7c040f80 con 0x7fef800f6510 2024-02-19T21:57:49.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:49.855+0000 7fef85a73700 1 --2- 172.21.15.86:0/3563131081 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fef6c02ff10 0x7fef6c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:57:49.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:49.855+0000 7fef85a73700 1 --2- 172.21.15.86:0/3563131081 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fef6c02ff10 0x7fef6c0323d0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7fef700096a0 tx=0x7fef70006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:57:49.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:49.860+0000 7fef76ffd700 1 -- 172.21.15.86:0/3563131081 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fef7c014420 con 0x7fef800f6510 2024-02-19T21:57:50.065 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:50.064+0000 7fef884d8700 1 -- 172.21.15.86:0/3563131081 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fef80108210 con 0x7fef800f6510 2024-02-19T21:57:50.065 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:50.064+0000 7fef76ffd700 1 -- 172.21.15.86:0/3563131081 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fef7c02c090 con 0x7fef800f6510 2024-02-19T21:57:50.066 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T21:57:50.066 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T21:57:50.067 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:50.066+0000 7fef884d8700 1 -- 172.21.15.86:0/3563131081 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fef6c02ff10 msgr2=0x7fef6c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:57:50.067 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:50.066+0000 7fef884d8700 1 --2- 172.21.15.86:0/3563131081 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fef6c02ff10 0x7fef6c0323d0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7fef700096a0 tx=0x7fef70006b40).stop 2024-02-19T21:57:50.067 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:50.066+0000 7fef884d8700 1 -- 172.21.15.86:0/3563131081 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fef800f6510 msgr2=0x7fef801067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:57:50.067 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:50.066+0000 7fef884d8700 1 --2- 172.21.15.86:0/3563131081 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fef800f6510 0x7fef801067e0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7fef7c005010 tx=0x7fef7c00b040).stop 2024-02-19T21:57:50.067 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:50.066+0000 7fef884d8700 1 -- 172.21.15.86:0/3563131081 shutdown_connections 2024-02-19T21:57:50.067 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:50.066+0000 7fef884d8700 1 --2- 172.21.15.86:0/3563131081 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fef6c02ff10 0x7fef6c0323d0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:57:50.067 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:50.066+0000 7fef884d8700 1 --2- 172.21.15.86:0/3563131081 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fef800f6510 0x7fef801067e0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:57:50.067 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:50.066+0000 7fef884d8700 1 -- 172.21.15.86:0/3563131081 >> 172.21.15.86:0/3563131081 conn(0x7fef800f1c70 msgr2=0x7fef800f26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:57:50.067 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:50.066+0000 7fef884d8700 1 -- 172.21.15.86:0/3563131081 shutdown_connections 2024-02-19T21:57:50.067 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:50.066+0000 7fef884d8700 1 -- 172.21.15.86:0/3563131081 wait complete. 2024-02-19T21:57:50.067 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T21:57:50.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:57:50 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T21:57:49.447545+0000 mgr.smithi053.fyhcdi (mgr.14162) 38 : cluster [DBG] pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:57:50.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:57:50 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:57:50.065812+0000 mon.smithi053 (mon.0) 128 : audit [DBG] from='client.? 172.21.15.86:0/3563131081' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T21:57:51.701 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T21:57:51.702 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T21:57:52.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:57:52 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T21:57:51.447910+0000 mgr.smithi053.fyhcdi (mgr.14162) 39 : cluster [DBG] pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:57:53.493 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:53.491+0000 7fb286c20700 1 -- 172.21.15.86:0/2220054097 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb2800f5410 msgr2=0x7fb2800f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:57:53.493 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:53.491+0000 7fb286c20700 1 --2- 172.21.15.86:0/2220054097 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb2800f5410 0x7fb2800f5830 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7fb274009fb0 tx=0x7fb274009b30).stop 2024-02-19T21:57:53.494 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:53.491+0000 7fb286c20700 1 -- 172.21.15.86:0/2220054097 shutdown_connections 2024-02-19T21:57:53.494 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:53.491+0000 7fb286c20700 1 --2- 172.21.15.86:0/2220054097 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb2800f5410 0x7fb2800f5830 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:57:53.494 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:53.491+0000 7fb286c20700 1 -- 172.21.15.86:0/2220054097 >> 172.21.15.86:0/2220054097 conn(0x7fb2800f0b70 msgr2=0x7fb2800f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:57:53.494 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:53.492+0000 7fb286c20700 1 -- 172.21.15.86:0/2220054097 shutdown_connections 2024-02-19T21:57:53.494 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:53.492+0000 7fb286c20700 1 -- 172.21.15.86:0/2220054097 wait complete. 2024-02-19T21:57:53.494 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:53.492+0000 7fb286c20700 1 Processor -- start 2024-02-19T21:57:53.494 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:53.492+0000 7fb286c20700 1 -- start start 2024-02-19T21:57:53.494 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:53.492+0000 7fb286c20700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb2800f5410 0x7fb2801089e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:57:53.494 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:53.492+0000 7fb286c20700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb280108ed0 con 0x7fb2800f5410 2024-02-19T21:57:53.494 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:53.493+0000 7fb2849bc700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb2800f5410 0x7fb2801089e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:57:53.494 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:53.493+0000 7fb2849bc700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb2800f5410 0x7fb2801089e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:32892/0 (socket says 172.21.15.86:32892) 2024-02-19T21:57:53.494 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:53.493+0000 7fb2849bc700 1 -- 172.21.15.86:0/3944037518 learned_addr learned my addr 172.21.15.86:0/3944037518 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T21:57:53.494 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:53.493+0000 7fb2849bc700 1 -- 172.21.15.86:0/3944037518 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb274005040 con 0x7fb2800f5410 2024-02-19T21:57:53.495 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:53.493+0000 7fb2849bc700 1 --2- 172.21.15.86:0/3944037518 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb2800f5410 0x7fb2801089e0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7fb274005010 tx=0x7fb27400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:57:53.495 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:53.493+0000 7fb27d7fa700 1 -- 172.21.15.86:0/3944037518 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb274009320 con 0x7fb2800f5410 2024-02-19T21:57:53.495 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:53.493+0000 7fb27d7fa700 1 -- 172.21.15.86:0/3944037518 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb274013380 con 0x7fb2800f5410 2024-02-19T21:57:53.495 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:53.493+0000 7fb286c20700 1 -- 172.21.15.86:0/3944037518 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2801090d0 con 0x7fb2800f5410 2024-02-19T21:57:53.495 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:53.493+0000 7fb27d7fa700 1 -- 172.21.15.86:0/3944037518 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb274014e50 con 0x7fb2800f5410 2024-02-19T21:57:53.495 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:53.493+0000 7fb286c20700 1 -- 172.21.15.86:0/3944037518 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb280109560 con 0x7fb2800f5410 2024-02-19T21:57:53.495 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:53.494+0000 7fb27d7fa700 1 -- 172.21.15.86:0/3944037518 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb2740095f0 con 0x7fb2800f5410 2024-02-19T21:57:53.496 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:53.494+0000 7fb286c20700 1 -- 172.21.15.86:0/3944037518 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb280045ad0 con 0x7fb2800f5410 2024-02-19T21:57:53.496 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:53.494+0000 7fb27d7fa700 1 --2- 172.21.15.86:0/3944037518 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fb27002ff00 0x7fb2700323c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:57:53.496 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:53.494+0000 7fb27d7fa700 1 -- 172.21.15.86:0/3944037518 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb274040fa0 con 0x7fb2800f5410 2024-02-19T21:57:53.496 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:53.494+0000 7fb27ffff700 1 --2- 172.21.15.86:0/3944037518 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fb27002ff00 0x7fb2700323c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:57:53.496 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:53.495+0000 7fb27ffff700 1 --2- 172.21.15.86:0/3944037518 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fb27002ff00 0x7fb2700323c0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7fb26c0096a0 tx=0x7fb26c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:57:53.499 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:53.499+0000 7fb27d7fa700 1 -- 172.21.15.86:0/3944037518 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb27401dbc0 con 0x7fb2800f5410 2024-02-19T21:57:53.704 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:53.704+0000 7fb286c20700 1 -- 172.21.15.86:0/3944037518 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb28010a410 con 0x7fb2800f5410 2024-02-19T21:57:53.705 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:53.705+0000 7fb27d7fa700 1 -- 172.21.15.86:0/3944037518 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb274010230 con 0x7fb2800f5410 2024-02-19T21:57:53.705 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T21:57:53.705 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T21:57:53.706 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:53.706+0000 7fb286c20700 1 -- 172.21.15.86:0/3944037518 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fb27002ff00 msgr2=0x7fb2700323c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:57:53.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:53.706+0000 7fb286c20700 1 --2- 172.21.15.86:0/3944037518 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fb27002ff00 0x7fb2700323c0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7fb26c0096a0 tx=0x7fb26c006b40).stop 2024-02-19T21:57:53.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:53.707+0000 7fb286c20700 1 -- 172.21.15.86:0/3944037518 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb2800f5410 msgr2=0x7fb2801089e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:57:53.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:53.707+0000 7fb286c20700 1 --2- 172.21.15.86:0/3944037518 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb2800f5410 0x7fb2801089e0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7fb274005010 tx=0x7fb27400b040).stop 2024-02-19T21:57:53.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:53.707+0000 7fb286c20700 1 -- 172.21.15.86:0/3944037518 shutdown_connections 2024-02-19T21:57:53.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:53.707+0000 7fb286c20700 1 --2- 172.21.15.86:0/3944037518 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fb27002ff00 0x7fb2700323c0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:57:53.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:53.707+0000 7fb286c20700 1 --2- 172.21.15.86:0/3944037518 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb2800f5410 0x7fb2801089e0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:57:53.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:53.707+0000 7fb286c20700 1 -- 172.21.15.86:0/3944037518 >> 172.21.15.86:0/3944037518 conn(0x7fb2800f0b70 msgr2=0x7fb2800f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:57:53.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:53.707+0000 7fb286c20700 1 -- 172.21.15.86:0/3944037518 shutdown_connections 2024-02-19T21:57:53.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:53.707+0000 7fb286c20700 1 -- 172.21.15.86:0/3944037518 wait complete. 2024-02-19T21:57:53.707 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T21:57:54.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:57:54 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T21:57:53.448193+0000 mgr.smithi053.fyhcdi (mgr.14162) 40 : cluster [DBG] pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:57:54.794 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:57:54 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:57:53.705994+0000 mon.smithi053 (mon.0) 129 : audit [DBG] from='client.? 172.21.15.86:0/3944037518' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T21:57:55.365 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T21:57:55.365 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T21:57:56.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:57:56 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T21:57:55.448387+0000 mgr.smithi053.fyhcdi (mgr.14162) 41 : cluster [DBG] pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:57:57.126 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:57.125+0000 7fabe8f8c700 1 -- 172.21.15.86:0/2927513682 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fabe40f6510 msgr2=0x7fabe40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:57:57.126 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:57.125+0000 7fabe8f8c700 1 --2- 172.21.15.86:0/2927513682 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fabe40f6510 0x7fabe40f6930 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7fabd8009fb0 tx=0x7fabd8009b30).stop 2024-02-19T21:57:57.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:57.125+0000 7fabe8f8c700 1 -- 172.21.15.86:0/2927513682 shutdown_connections 2024-02-19T21:57:57.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:57.125+0000 7fabe8f8c700 1 --2- 172.21.15.86:0/2927513682 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fabe40f6510 0x7fabe40f6930 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:57:57.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:57.125+0000 7fabe8f8c700 1 -- 172.21.15.86:0/2927513682 >> 172.21.15.86:0/2927513682 conn(0x7fabe40f1c70 msgr2=0x7fabe40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:57:57.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:57.125+0000 7fabe8f8c700 1 -- 172.21.15.86:0/2927513682 shutdown_connections 2024-02-19T21:57:57.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:57.125+0000 7fabe8f8c700 1 -- 172.21.15.86:0/2927513682 wait complete. 2024-02-19T21:57:57.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:57.125+0000 7fabe8f8c700 1 Processor -- start 2024-02-19T21:57:57.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:57.125+0000 7fabe8f8c700 1 -- start start 2024-02-19T21:57:57.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:57.126+0000 7fabe8f8c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fabe40f6510 0x7fabe41067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:57:57.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:57.126+0000 7fabe8f8c700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabe4106cd0 con 0x7fabe40f6510 2024-02-19T21:57:57.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:57.126+0000 7fabe259c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fabe40f6510 0x7fabe41067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:57:57.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:57.126+0000 7fabe259c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fabe40f6510 0x7fabe41067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:32896/0 (socket says 172.21.15.86:32896) 2024-02-19T21:57:57.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:57.126+0000 7fabe259c700 1 -- 172.21.15.86:0/3449544868 learned_addr learned my addr 172.21.15.86:0/3449544868 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T21:57:57.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:57.126+0000 7fabe259c700 1 -- 172.21.15.86:0/3449544868 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabd8005040 con 0x7fabe40f6510 2024-02-19T21:57:57.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:57.126+0000 7fabe259c700 1 --2- 172.21.15.86:0/3449544868 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fabe40f6510 0x7fabe41067e0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7fabd8005010 tx=0x7fabd800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:57:57.128 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:57.126+0000 7fabd2ffd700 1 -- 172.21.15.86:0/3449544868 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fabd8009320 con 0x7fabe40f6510 2024-02-19T21:57:57.128 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:57.126+0000 7fabd2ffd700 1 -- 172.21.15.86:0/3449544868 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fabd8013380 con 0x7fabe40f6510 2024-02-19T21:57:57.128 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:57.126+0000 7fabe8f8c700 1 -- 172.21.15.86:0/3449544868 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabe4106ed0 con 0x7fabe40f6510 2024-02-19T21:57:57.128 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:57.126+0000 7fabd2ffd700 1 -- 172.21.15.86:0/3449544868 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fabd8014e50 con 0x7fabe40f6510 2024-02-19T21:57:57.128 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:57.127+0000 7fabe8f8c700 1 -- 172.21.15.86:0/3449544868 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabe4107360 con 0x7fabe40f6510 2024-02-19T21:57:57.129 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:57.127+0000 7fabd2ffd700 1 -- 172.21.15.86:0/3449544868 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fabd8014420 con 0x7fabe40f6510 2024-02-19T21:57:57.129 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:57.127+0000 7fabe8f8c700 1 -- 172.21.15.86:0/3449544868 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabe4045ad0 con 0x7fabe40f6510 2024-02-19T21:57:57.129 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:57.128+0000 7fabd2ffd700 1 --2- 172.21.15.86:0/3449544868 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fabcc02ff10 0x7fabcc0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:57:57.129 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:57.128+0000 7fabd2ffd700 1 -- 172.21.15.86:0/3449544868 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fabd8040e30 con 0x7fabe40f6510 2024-02-19T21:57:57.129 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:57.128+0000 7fabe1d9b700 1 --2- 172.21.15.86:0/3449544868 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fabcc02ff10 0x7fabcc0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:57:57.129 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:57.128+0000 7fabe1d9b700 1 --2- 172.21.15.86:0/3449544868 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fabcc02ff10 0x7fabcc0323d0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7fabd40096a0 tx=0x7fabd4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:57:57.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:57.132+0000 7fabd2ffd700 1 -- 172.21.15.86:0/3449544868 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fabd80095f0 con 0x7fabe40f6510 2024-02-19T21:57:57.337 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:57.337+0000 7fabe8f8c700 1 -- 172.21.15.86:0/3449544868 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fabe4108210 con 0x7fabe40f6510 2024-02-19T21:57:57.338 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:57.338+0000 7fabd2ffd700 1 -- 172.21.15.86:0/3449544868 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fabd801db70 con 0x7fabe40f6510 2024-02-19T21:57:57.338 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T21:57:57.338 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T21:57:57.340 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:57.340+0000 7fabe8f8c700 1 -- 172.21.15.86:0/3449544868 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fabcc02ff10 msgr2=0x7fabcc0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:57:57.340 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:57.340+0000 7fabe8f8c700 1 --2- 172.21.15.86:0/3449544868 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fabcc02ff10 0x7fabcc0323d0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7fabd40096a0 tx=0x7fabd4006b40).stop 2024-02-19T21:57:57.340 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:57.340+0000 7fabe8f8c700 1 -- 172.21.15.86:0/3449544868 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fabe40f6510 msgr2=0x7fabe41067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:57:57.340 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:57.340+0000 7fabe8f8c700 1 --2- 172.21.15.86:0/3449544868 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fabe40f6510 0x7fabe41067e0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7fabd8005010 tx=0x7fabd800b040).stop 2024-02-19T21:57:57.340 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:57.340+0000 7fabe8f8c700 1 -- 172.21.15.86:0/3449544868 shutdown_connections 2024-02-19T21:57:57.340 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:57.340+0000 7fabe8f8c700 1 --2- 172.21.15.86:0/3449544868 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fabcc02ff10 0x7fabcc0323d0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:57:57.340 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:57.340+0000 7fabe8f8c700 1 --2- 172.21.15.86:0/3449544868 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fabe40f6510 0x7fabe41067e0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:57:57.340 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:57.340+0000 7fabe8f8c700 1 -- 172.21.15.86:0/3449544868 >> 172.21.15.86:0/3449544868 conn(0x7fabe40f1c70 msgr2=0x7fabe40f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:57:57.340 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:57.340+0000 7fabe8f8c700 1 -- 172.21.15.86:0/3449544868 shutdown_connections 2024-02-19T21:57:57.341 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:57:57.340+0000 7fabe8f8c700 1 -- 172.21.15.86:0/3449544868 wait complete. 2024-02-19T21:57:57.341 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T21:57:57.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:57:57 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:57:57.339422+0000 mon.smithi053 (mon.0) 130 : audit [DBG] from='client.? 172.21.15.86:0/3449544868' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T21:57:58.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:57:58 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T21:57:57.448630+0000 mgr.smithi053.fyhcdi (mgr.14162) 42 : cluster [DBG] pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:57:58.975 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T21:57:58.976 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T21:58:00.724 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:00.724+0000 7fa714c1e700 1 -- 172.21.15.86:0/1722432740 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa710060180 msgr2=0x7fa7100f8f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:58:00.725 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:00.724+0000 7fa714c1e700 1 --2- 172.21.15.86:0/1722432740 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa710060180 0x7fa7100f8f80 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7fa700009fb0 tx=0x7fa700009b30).stop 2024-02-19T21:58:00.725 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:00.724+0000 7fa714c1e700 1 -- 172.21.15.86:0/1722432740 shutdown_connections 2024-02-19T21:58:00.725 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:00.724+0000 7fa714c1e700 1 --2- 172.21.15.86:0/1722432740 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa710060180 0x7fa7100f8f80 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:58:00.725 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:00.724+0000 7fa714c1e700 1 -- 172.21.15.86:0/1722432740 >> 172.21.15.86:0/1722432740 conn(0x7fa7100f09a0 msgr2=0x7fa7100f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:58:00.725 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:00.724+0000 7fa714c1e700 1 -- 172.21.15.86:0/1722432740 shutdown_connections 2024-02-19T21:58:00.725 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:00.724+0000 7fa714c1e700 1 -- 172.21.15.86:0/1722432740 wait complete. 2024-02-19T21:58:00.725 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:00.724+0000 7fa714c1e700 1 Processor -- start 2024-02-19T21:58:00.725 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:00.725+0000 7fa714c1e700 1 -- start start 2024-02-19T21:58:00.725 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:00.725+0000 7fa714c1e700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa710060180 0x7fa710106610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:58:00.725 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:00.725+0000 7fa714c1e700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa710106b00 con 0x7fa710060180 2024-02-19T21:58:00.727 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:00.725+0000 7fa70f7fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa710060180 0x7fa710106610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:58:00.727 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:00.725+0000 7fa70f7fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa710060180 0x7fa710106610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:54298/0 (socket says 172.21.15.86:54298) 2024-02-19T21:58:00.727 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:00.725+0000 7fa70f7fe700 1 -- 172.21.15.86:0/2052485964 learned_addr learned my addr 172.21.15.86:0/2052485964 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T21:58:00.727 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:00.725+0000 7fa70f7fe700 1 -- 172.21.15.86:0/2052485964 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa700005040 con 0x7fa710060180 2024-02-19T21:58:00.727 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:00.726+0000 7fa70f7fe700 1 --2- 172.21.15.86:0/2052485964 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa710060180 0x7fa710106610 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7fa700005010 tx=0x7fa70000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:58:00.727 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:00.726+0000 7fa6f7fff700 1 -- 172.21.15.86:0/2052485964 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa700009320 con 0x7fa710060180 2024-02-19T21:58:00.727 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:00.726+0000 7fa6f7fff700 1 -- 172.21.15.86:0/2052485964 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa700013380 con 0x7fa710060180 2024-02-19T21:58:00.727 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:00.726+0000 7fa6f7fff700 1 -- 172.21.15.86:0/2052485964 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa700014e50 con 0x7fa710060180 2024-02-19T21:58:00.727 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:00.726+0000 7fa714c1e700 1 -- 172.21.15.86:0/2052485964 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa710106d00 con 0x7fa710060180 2024-02-19T21:58:00.728 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:00.726+0000 7fa714c1e700 1 -- 172.21.15.86:0/2052485964 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa710107190 con 0x7fa710060180 2024-02-19T21:58:00.728 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:00.727+0000 7fa6f7fff700 1 -- 172.21.15.86:0/2052485964 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa7000095f0 con 0x7fa710060180 2024-02-19T21:58:00.728 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:00.727+0000 7fa714c1e700 1 -- 172.21.15.86:0/2052485964 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa710045ad0 con 0x7fa710060180 2024-02-19T21:58:00.728 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:00.727+0000 7fa6f7fff700 1 --2- 172.21.15.86:0/2052485964 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fa6f802ff10 0x7fa6f80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:58:00.728 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:00.727+0000 7fa6f7fff700 1 -- 172.21.15.86:0/2052485964 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa700040e60 con 0x7fa710060180 2024-02-19T21:58:00.728 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:00.727+0000 7fa70effd700 1 --2- 172.21.15.86:0/2052485964 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fa6f802ff10 0x7fa6f80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:58:00.728 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:00.728+0000 7fa70effd700 1 --2- 172.21.15.86:0/2052485964 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fa6f802ff10 0x7fa6f80323d0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7fa7040096a0 tx=0x7fa704006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:58:00.731 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:00.732+0000 7fa6f7fff700 1 -- 172.21.15.86:0/2052485964 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa700014420 con 0x7fa710060180 2024-02-19T21:58:00.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:58:00 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T21:57:59.448851+0000 mgr.smithi053.fyhcdi (mgr.14162) 43 : cluster [DBG] pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:58:00.937 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:00.937+0000 7fa714c1e700 1 -- 172.21.15.86:0/2052485964 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa710108040 con 0x7fa710060180 2024-02-19T21:58:00.937 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:00.938+0000 7fa6f7fff700 1 -- 172.21.15.86:0/2052485964 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa70001d980 con 0x7fa710060180 2024-02-19T21:58:00.938 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T21:58:00.938 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T21:58:00.939 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:00.939+0000 7fa714c1e700 1 -- 172.21.15.86:0/2052485964 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fa6f802ff10 msgr2=0x7fa6f80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:58:00.939 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:00.939+0000 7fa714c1e700 1 --2- 172.21.15.86:0/2052485964 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fa6f802ff10 0x7fa6f80323d0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7fa7040096a0 tx=0x7fa704006b40).stop 2024-02-19T21:58:00.939 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:00.939+0000 7fa714c1e700 1 -- 172.21.15.86:0/2052485964 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa710060180 msgr2=0x7fa710106610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:58:00.939 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:00.939+0000 7fa714c1e700 1 --2- 172.21.15.86:0/2052485964 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa710060180 0x7fa710106610 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7fa700005010 tx=0x7fa70000b040).stop 2024-02-19T21:58:00.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:00.939+0000 7fa714c1e700 1 -- 172.21.15.86:0/2052485964 shutdown_connections 2024-02-19T21:58:00.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:00.939+0000 7fa714c1e700 1 --2- 172.21.15.86:0/2052485964 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fa6f802ff10 0x7fa6f80323d0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:58:00.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:00.939+0000 7fa714c1e700 1 --2- 172.21.15.86:0/2052485964 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa710060180 0x7fa710106610 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:58:00.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:00.939+0000 7fa714c1e700 1 -- 172.21.15.86:0/2052485964 >> 172.21.15.86:0/2052485964 conn(0x7fa7100f09a0 msgr2=0x7fa7100f1410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:58:00.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:00.939+0000 7fa714c1e700 1 -- 172.21.15.86:0/2052485964 shutdown_connections 2024-02-19T21:58:00.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:00.939+0000 7fa714c1e700 1 -- 172.21.15.86:0/2052485964 wait complete. 2024-02-19T21:58:00.940 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T21:58:01.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:58:01 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:58:00.938840+0000 mon.smithi053 (mon.0) 131 : audit [DBG] from='client.? 172.21.15.86:0/2052485964' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T21:58:02.546 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T21:58:02.546 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T21:58:02.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:58:02 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T21:58:01.449070+0000 mgr.smithi053.fyhcdi (mgr.14162) 44 : cluster [DBG] pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:58:04.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:04.274+0000 7fcc6e40b700 1 -- 172.21.15.86:0/1056416349 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcc680f5410 msgr2=0x7fcc680f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:58:04.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:04.274+0000 7fcc6e40b700 1 --2- 172.21.15.86:0/1056416349 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcc680f5410 0x7fcc680f5830 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7fcc58009fb0 tx=0x7fcc58009b30).stop 2024-02-19T21:58:04.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:04.274+0000 7fcc6e40b700 1 -- 172.21.15.86:0/1056416349 shutdown_connections 2024-02-19T21:58:04.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:04.274+0000 7fcc6e40b700 1 --2- 172.21.15.86:0/1056416349 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcc680f5410 0x7fcc680f5830 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:58:04.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:04.274+0000 7fcc6e40b700 1 -- 172.21.15.86:0/1056416349 >> 172.21.15.86:0/1056416349 conn(0x7fcc680f0b70 msgr2=0x7fcc680f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:58:04.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:04.274+0000 7fcc6e40b700 1 -- 172.21.15.86:0/1056416349 shutdown_connections 2024-02-19T21:58:04.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:04.274+0000 7fcc6e40b700 1 -- 172.21.15.86:0/1056416349 wait complete. 2024-02-19T21:58:04.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:04.275+0000 7fcc6e40b700 1 Processor -- start 2024-02-19T21:58:04.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:04.275+0000 7fcc6e40b700 1 -- start start 2024-02-19T21:58:04.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:04.275+0000 7fcc6e40b700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcc680f5410 0x7fcc681094e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:58:04.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:04.275+0000 7fcc6e40b700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc68060240 con 0x7fcc680f5410 2024-02-19T21:58:04.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:04.275+0000 7fcc67fff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcc680f5410 0x7fcc681094e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:58:04.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:04.275+0000 7fcc67fff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcc680f5410 0x7fcc681094e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:54318/0 (socket says 172.21.15.86:54318) 2024-02-19T21:58:04.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:04.275+0000 7fcc67fff700 1 -- 172.21.15.86:0/2677987331 learned_addr learned my addr 172.21.15.86:0/2677987331 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T21:58:04.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:04.276+0000 7fcc67fff700 1 -- 172.21.15.86:0/2677987331 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc58005040 con 0x7fcc680f5410 2024-02-19T21:58:04.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:04.276+0000 7fcc67fff700 1 --2- 172.21.15.86:0/2677987331 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcc680f5410 0x7fcc681094e0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7fcc58005010 tx=0x7fcc5800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:58:04.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:04.276+0000 7fcc64ff9700 1 -- 172.21.15.86:0/2677987331 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcc58009320 con 0x7fcc680f5410 2024-02-19T21:58:04.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:04.276+0000 7fcc6e40b700 1 -- 172.21.15.86:0/2677987331 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc68060440 con 0x7fcc680f5410 2024-02-19T21:58:04.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:04.276+0000 7fcc6e40b700 1 -- 172.21.15.86:0/2677987331 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc68109b90 con 0x7fcc680f5410 2024-02-19T21:58:04.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:04.276+0000 7fcc64ff9700 1 -- 172.21.15.86:0/2677987331 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcc58013380 con 0x7fcc680f5410 2024-02-19T21:58:04.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:04.276+0000 7fcc64ff9700 1 -- 172.21.15.86:0/2677987331 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcc58014df0 con 0x7fcc680f5410 2024-02-19T21:58:04.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:04.277+0000 7fcc64ff9700 1 -- 172.21.15.86:0/2677987331 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fcc580095f0 con 0x7fcc680f5410 2024-02-19T21:58:04.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:04.277+0000 7fcc6e40b700 1 -- 172.21.15.86:0/2677987331 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc4c004fa0 con 0x7fcc680f5410 2024-02-19T21:58:04.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:04.277+0000 7fcc64ff9700 1 --2- 172.21.15.86:0/2677987331 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fcc5402ff60 0x7fcc54032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:58:04.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:04.277+0000 7fcc64ff9700 1 -- 172.21.15.86:0/2677987331 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fcc58040db0 con 0x7fcc680f5410 2024-02-19T21:58:04.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:04.277+0000 7fcc677fe700 1 --2- 172.21.15.86:0/2677987331 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fcc5402ff60 0x7fcc54032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:58:04.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:04.278+0000 7fcc677fe700 1 --2- 172.21.15.86:0/2677987331 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fcc5402ff60 0x7fcc54032420 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7fcc500096a0 tx=0x7fcc50006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:58:04.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:04.282+0000 7fcc64ff9700 1 -- 172.21.15.86:0/2677987331 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcc58014420 con 0x7fcc680f5410 2024-02-19T21:58:04.490 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:04.490+0000 7fcc6e40b700 1 -- 172.21.15.86:0/2677987331 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcc4c005670 con 0x7fcc680f5410 2024-02-19T21:58:04.490 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:04.490+0000 7fcc64ff9700 1 -- 172.21.15.86:0/2677987331 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fcc5800e2a0 con 0x7fcc680f5410 2024-02-19T21:58:04.490 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T21:58:04.491 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T21:58:04.492 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:04.492+0000 7fcc6e40b700 1 -- 172.21.15.86:0/2677987331 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fcc5402ff60 msgr2=0x7fcc54032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:58:04.492 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:04.492+0000 7fcc6e40b700 1 --2- 172.21.15.86:0/2677987331 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fcc5402ff60 0x7fcc54032420 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7fcc500096a0 tx=0x7fcc50006b40).stop 2024-02-19T21:58:04.492 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:04.492+0000 7fcc6e40b700 1 -- 172.21.15.86:0/2677987331 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcc680f5410 msgr2=0x7fcc681094e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:58:04.492 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:04.492+0000 7fcc6e40b700 1 --2- 172.21.15.86:0/2677987331 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcc680f5410 0x7fcc681094e0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7fcc58005010 tx=0x7fcc5800b040).stop 2024-02-19T21:58:04.492 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:04.492+0000 7fcc6e40b700 1 -- 172.21.15.86:0/2677987331 shutdown_connections 2024-02-19T21:58:04.492 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:04.492+0000 7fcc6e40b700 1 --2- 172.21.15.86:0/2677987331 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fcc5402ff60 0x7fcc54032420 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:58:04.492 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:04.492+0000 7fcc6e40b700 1 --2- 172.21.15.86:0/2677987331 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcc680f5410 0x7fcc681094e0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:58:04.493 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:04.492+0000 7fcc6e40b700 1 -- 172.21.15.86:0/2677987331 >> 172.21.15.86:0/2677987331 conn(0x7fcc680f0b70 msgr2=0x7fcc680f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:58:04.493 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:04.492+0000 7fcc6e40b700 1 -- 172.21.15.86:0/2677987331 shutdown_connections 2024-02-19T21:58:04.493 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:04.492+0000 7fcc6e40b700 1 -- 172.21.15.86:0/2677987331 wait complete. 2024-02-19T21:58:04.493 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T21:58:04.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:58:04 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T21:58:03.449330+0000 mgr.smithi053.fyhcdi (mgr.14162) 45 : cluster [DBG] pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:58:05.794 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:58:05 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:58:04.491716+0000 mon.smithi053 (mon.0) 132 : audit [DBG] from='client.? 172.21.15.86:0/2677987331' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T21:58:06.046 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T21:58:06.047 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T21:58:06.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:58:06 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T21:58:05.449598+0000 mgr.smithi053.fyhcdi (mgr.14162) 46 : cluster [DBG] pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:58:07.960 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:07.960+0000 7fcdcbfff700 1 -- 172.21.15.86:0/2635135797 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcdc4013990 con 0x7fcdd40f5410 2024-02-19T21:58:07.961 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:07.960+0000 7fcddb3bb700 1 -- 172.21.15.86:0/2635135797 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcdd40f5410 msgr2=0x7fcdd40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:58:07.961 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:07.960+0000 7fcddb3bb700 1 --2- 172.21.15.86:0/2635135797 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcdd40f5410 0x7fcdd40f5830 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7fcdc4009fb0 tx=0x7fcdc4009b30).stop 2024-02-19T21:58:07.961 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:07.960+0000 7fcddb3bb700 1 -- 172.21.15.86:0/2635135797 shutdown_connections 2024-02-19T21:58:07.961 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:07.960+0000 7fcddb3bb700 1 --2- 172.21.15.86:0/2635135797 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcdd40f5410 0x7fcdd40f5830 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:58:07.961 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:07.960+0000 7fcddb3bb700 1 -- 172.21.15.86:0/2635135797 >> 172.21.15.86:0/2635135797 conn(0x7fcdd40f0b70 msgr2=0x7fcdd40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:58:07.961 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:07.960+0000 7fcddb3bb700 1 -- 172.21.15.86:0/2635135797 shutdown_connections 2024-02-19T21:58:07.961 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:07.960+0000 7fcddb3bb700 1 -- 172.21.15.86:0/2635135797 wait complete. 2024-02-19T21:58:07.961 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:07.961+0000 7fcddb3bb700 1 Processor -- start 2024-02-19T21:58:07.961 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:07.961+0000 7fcddb3bb700 1 -- start start 2024-02-19T21:58:07.961 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:07.961+0000 7fcddb3bb700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcdd40f5410 0x7fcdd41067d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:58:07.961 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:07.961+0000 7fcddb3bb700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdd4106cc0 con 0x7fcdd40f5410 2024-02-19T21:58:07.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:07.961+0000 7fcdd9157700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcdd40f5410 0x7fcdd41067d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:58:07.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:07.961+0000 7fcdd9157700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcdd40f5410 0x7fcdd41067d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:54334/0 (socket says 172.21.15.86:54334) 2024-02-19T21:58:07.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:07.961+0000 7fcdd9157700 1 -- 172.21.15.86:0/1254762669 learned_addr learned my addr 172.21.15.86:0/1254762669 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T21:58:07.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:07.961+0000 7fcdd9157700 1 -- 172.21.15.86:0/1254762669 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdc4005040 con 0x7fcdd40f5410 2024-02-19T21:58:07.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:07.962+0000 7fcdd9157700 1 --2- 172.21.15.86:0/1254762669 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcdd40f5410 0x7fcdd41067d0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7fcdc4005010 tx=0x7fcdc400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:58:07.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:07.962+0000 7fcdc9ffb700 1 -- 172.21.15.86:0/1254762669 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcdc4009320 con 0x7fcdd40f5410 2024-02-19T21:58:07.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:07.962+0000 7fcddb3bb700 1 -- 172.21.15.86:0/1254762669 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdd4106ec0 con 0x7fcdd40f5410 2024-02-19T21:58:07.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:07.962+0000 7fcdc9ffb700 1 -- 172.21.15.86:0/1254762669 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcdc4013380 con 0x7fcdd40f5410 2024-02-19T21:58:07.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:07.962+0000 7fcdc9ffb700 1 -- 172.21.15.86:0/1254762669 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcdc4014e50 con 0x7fcdd40f5410 2024-02-19T21:58:07.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:07.962+0000 7fcddb3bb700 1 -- 172.21.15.86:0/1254762669 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdd4107350 con 0x7fcdd40f5410 2024-02-19T21:58:07.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:07.963+0000 7fcdc9ffb700 1 -- 172.21.15.86:0/1254762669 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fcdc40095f0 con 0x7fcdd40f5410 2024-02-19T21:58:07.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:07.963+0000 7fcdc9ffb700 1 --2- 172.21.15.86:0/1254762669 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fcdc002ff10 0x7fcdc00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:58:07.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:07.963+0000 7fcdc9ffb700 1 -- 172.21.15.86:0/1254762669 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fcdc4040fa0 con 0x7fcdd40f5410 2024-02-19T21:58:07.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:07.963+0000 7fcddb3bb700 1 -- 172.21.15.86:0/1254762669 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcdd4045ad0 con 0x7fcdd40f5410 2024-02-19T21:58:07.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:07.963+0000 7fcdd8956700 1 --2- 172.21.15.86:0/1254762669 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fcdc002ff10 0x7fcdc00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:58:07.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:07.964+0000 7fcdd8956700 1 --2- 172.21.15.86:0/1254762669 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fcdc002ff10 0x7fcdc00323d0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7fcdd00096a0 tx=0x7fcdd0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:58:07.968 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:07.968+0000 7fcdc9ffb700 1 -- 172.21.15.86:0/1254762669 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcdc4014420 con 0x7fcdd40f5410 2024-02-19T21:58:08.173 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:08.173+0000 7fcddb3bb700 1 -- 172.21.15.86:0/1254762669 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcdd41081a0 con 0x7fcdd40f5410 2024-02-19T21:58:08.174 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:08.174+0000 7fcdc9ffb700 1 -- 172.21.15.86:0/1254762669 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fcdc4010230 con 0x7fcdd40f5410 2024-02-19T21:58:08.174 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T21:58:08.174 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T21:58:08.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:08.176+0000 7fcddb3bb700 1 -- 172.21.15.86:0/1254762669 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fcdc002ff10 msgr2=0x7fcdc00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:58:08.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:08.176+0000 7fcddb3bb700 1 --2- 172.21.15.86:0/1254762669 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fcdc002ff10 0x7fcdc00323d0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7fcdd00096a0 tx=0x7fcdd0006b40).stop 2024-02-19T21:58:08.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:08.176+0000 7fcddb3bb700 1 -- 172.21.15.86:0/1254762669 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcdd40f5410 msgr2=0x7fcdd41067d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:58:08.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:08.176+0000 7fcddb3bb700 1 --2- 172.21.15.86:0/1254762669 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcdd40f5410 0x7fcdd41067d0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7fcdc4005010 tx=0x7fcdc400b040).stop 2024-02-19T21:58:08.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:08.176+0000 7fcddb3bb700 1 -- 172.21.15.86:0/1254762669 shutdown_connections 2024-02-19T21:58:08.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:08.176+0000 7fcddb3bb700 1 --2- 172.21.15.86:0/1254762669 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fcdc002ff10 0x7fcdc00323d0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:58:08.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:08.176+0000 7fcddb3bb700 1 --2- 172.21.15.86:0/1254762669 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcdd40f5410 0x7fcdd41067d0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:58:08.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:08.176+0000 7fcddb3bb700 1 -- 172.21.15.86:0/1254762669 >> 172.21.15.86:0/1254762669 conn(0x7fcdd40f0b70 msgr2=0x7fcdd405fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:58:08.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:08.176+0000 7fcddb3bb700 1 -- 172.21.15.86:0/1254762669 shutdown_connections 2024-02-19T21:58:08.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:08.176+0000 7fcddb3bb700 1 -- 172.21.15.86:0/1254762669 wait complete. 2024-02-19T21:58:08.177 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T21:58:08.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:58:08 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T21:58:07.449877+0000 mgr.smithi053.fyhcdi (mgr.14162) 47 : cluster [DBG] pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:58:08.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:58:08 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:58:08.175213+0000 mon.smithi053 (mon.0) 133 : audit [DBG] from='client.? 172.21.15.86:0/1254762669' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T21:58:10.202 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T21:58:10.202 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T21:58:10.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:58:10 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T21:58:09.450113+0000 mgr.smithi053.fyhcdi (mgr.14162) 48 : cluster [DBG] pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:58:11.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:11.891+0000 7fad5359e700 1 -- 172.21.15.86:0/3762100103 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fad540f51a0 msgr2=0x7fad540f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:58:11.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:11.891+0000 7fad5359e700 1 --2- 172.21.15.86:0/3762100103 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fad540f51a0 0x7fad540f55c0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7fad44009fb0 tx=0x7fad44009b30).stop 2024-02-19T21:58:11.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:11.891+0000 7fad5359e700 1 -- 172.21.15.86:0/3762100103 shutdown_connections 2024-02-19T21:58:11.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:11.891+0000 7fad5359e700 1 --2- 172.21.15.86:0/3762100103 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fad540f51a0 0x7fad540f55c0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:58:11.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:11.891+0000 7fad5359e700 1 -- 172.21.15.86:0/3762100103 >> 172.21.15.86:0/3762100103 conn(0x7fad540f09a0 msgr2=0x7fad540f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:58:11.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:11.891+0000 7fad5359e700 1 -- 172.21.15.86:0/3762100103 shutdown_connections 2024-02-19T21:58:11.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:11.891+0000 7fad5359e700 1 -- 172.21.15.86:0/3762100103 wait complete. 2024-02-19T21:58:11.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:11.891+0000 7fad5359e700 1 Processor -- start 2024-02-19T21:58:11.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:11.891+0000 7fad5359e700 1 -- start start 2024-02-19T21:58:11.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:11.891+0000 7fad5359e700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fad540f51a0 0x7fad541065d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:58:11.899 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:11.891+0000 7fad5359e700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad54106ac0 con 0x7fad540f51a0 2024-02-19T21:58:11.899 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:11.892+0000 7fad5259c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fad540f51a0 0x7fad541065d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:58:11.899 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:11.892+0000 7fad5259c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fad540f51a0 0x7fad541065d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:58404/0 (socket says 172.21.15.86:58404) 2024-02-19T21:58:11.899 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:11.892+0000 7fad5259c700 1 -- 172.21.15.86:0/470452434 learned_addr learned my addr 172.21.15.86:0/470452434 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T21:58:11.899 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:11.892+0000 7fad5259c700 1 -- 172.21.15.86:0/470452434 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad44005040 con 0x7fad540f51a0 2024-02-19T21:58:11.899 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:11.892+0000 7fad5259c700 1 --2- 172.21.15.86:0/470452434 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fad540f51a0 0x7fad541065d0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7fad44009830 tx=0x7fad4400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:58:11.899 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:11.892+0000 7fad42ffd700 1 -- 172.21.15.86:0/470452434 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fad44009320 con 0x7fad540f51a0 2024-02-19T21:58:11.899 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:11.892+0000 7fad42ffd700 1 -- 172.21.15.86:0/470452434 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fad44013380 con 0x7fad540f51a0 2024-02-19T21:58:11.899 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:11.892+0000 7fad5359e700 1 -- 172.21.15.86:0/470452434 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad54106cc0 con 0x7fad540f51a0 2024-02-19T21:58:11.899 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:11.892+0000 7fad42ffd700 1 -- 172.21.15.86:0/470452434 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fad44014420 con 0x7fad540f51a0 2024-02-19T21:58:11.899 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:11.892+0000 7fad5359e700 1 -- 172.21.15.86:0/470452434 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad54107150 con 0x7fad540f51a0 2024-02-19T21:58:11.899 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:11.893+0000 7fad42ffd700 1 -- 172.21.15.86:0/470452434 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fad4401e420 con 0x7fad540f51a0 2024-02-19T21:58:11.899 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:11.893+0000 7fad42ffd700 1 --2- 172.21.15.86:0/470452434 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fad3c02ffb0 0x7fad3c032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:58:11.900 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:11.893+0000 7fad5359e700 1 -- 172.21.15.86:0/470452434 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad54045ad0 con 0x7fad540f51a0 2024-02-19T21:58:11.900 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:11.893+0000 7fad42ffd700 1 -- 172.21.15.86:0/470452434 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fad44040e00 con 0x7fad540f51a0 2024-02-19T21:58:11.900 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:11.894+0000 7fad51d9b700 1 --2- 172.21.15.86:0/470452434 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fad3c02ffb0 0x7fad3c032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:58:11.900 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:11.894+0000 7fad51d9b700 1 --2- 172.21.15.86:0/470452434 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fad3c02ffb0 0x7fad3c032470 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7fad480096a0 tx=0x7fad48006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:58:11.900 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:11.898+0000 7fad42ffd700 1 -- 172.21.15.86:0/470452434 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fad4401c970 con 0x7fad540f51a0 2024-02-19T21:58:12.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:12.104+0000 7fad5359e700 1 -- 172.21.15.86:0/470452434 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fad540fad30 con 0x7fad540f51a0 2024-02-19T21:58:12.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:12.105+0000 7fad42ffd700 1 -- 172.21.15.86:0/470452434 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fad4400e2a0 con 0x7fad540f51a0 2024-02-19T21:58:12.105 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T21:58:12.106 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T21:58:12.107 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:12.107+0000 7fad5359e700 1 -- 172.21.15.86:0/470452434 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fad3c02ffb0 msgr2=0x7fad3c032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:58:12.107 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:12.107+0000 7fad5359e700 1 --2- 172.21.15.86:0/470452434 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fad3c02ffb0 0x7fad3c032470 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7fad480096a0 tx=0x7fad48006b40).stop 2024-02-19T21:58:12.107 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:12.107+0000 7fad5359e700 1 -- 172.21.15.86:0/470452434 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fad540f51a0 msgr2=0x7fad541065d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:58:12.107 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:12.107+0000 7fad5359e700 1 --2- 172.21.15.86:0/470452434 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fad540f51a0 0x7fad541065d0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7fad44009830 tx=0x7fad4400b040).stop 2024-02-19T21:58:12.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:12.107+0000 7fad5359e700 1 -- 172.21.15.86:0/470452434 shutdown_connections 2024-02-19T21:58:12.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:12.107+0000 7fad5359e700 1 --2- 172.21.15.86:0/470452434 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fad3c02ffb0 0x7fad3c032470 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:58:12.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:12.107+0000 7fad5359e700 1 --2- 172.21.15.86:0/470452434 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fad540f51a0 0x7fad541065d0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:58:12.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:12.107+0000 7fad5359e700 1 -- 172.21.15.86:0/470452434 >> 172.21.15.86:0/470452434 conn(0x7fad540f09a0 msgr2=0x7fad5405fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:58:12.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:12.107+0000 7fad5359e700 1 -- 172.21.15.86:0/470452434 shutdown_connections 2024-02-19T21:58:12.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:12.107+0000 7fad5359e700 1 -- 172.21.15.86:0/470452434 wait complete. 2024-02-19T21:58:12.108 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T21:58:12.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:58:12 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T21:58:11.450355+0000 mgr.smithi053.fyhcdi (mgr.14162) 49 : cluster [DBG] pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:58:12.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:58:12 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:58:12.106572+0000 mon.smithi053 (mon.0) 134 : audit [DBG] from='client.? 172.21.15.86:0/470452434' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T21:58:13.759 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T21:58:13.759 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T21:58:14.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:58:14 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T21:58:13.450629+0000 mgr.smithi053.fyhcdi (mgr.14162) 50 : cluster [DBG] pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:58:15.783 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:15.783+0000 7f35a759e700 1 -- 172.21.15.86:0/2479950580 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f35a80f51a0 msgr2=0x7f35a80f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:58:15.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:15.783+0000 7f35a759e700 1 --2- 172.21.15.86:0/2479950580 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f35a80f51a0 0x7f35a80f55c0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f3598009fb0 tx=0x7f3598009b30).stop 2024-02-19T21:58:15.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:15.783+0000 7f35a759e700 1 -- 172.21.15.86:0/2479950580 shutdown_connections 2024-02-19T21:58:15.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:15.783+0000 7f35a759e700 1 --2- 172.21.15.86:0/2479950580 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f35a80f51a0 0x7f35a80f55c0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:58:15.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:15.783+0000 7f35a759e700 1 -- 172.21.15.86:0/2479950580 >> 172.21.15.86:0/2479950580 conn(0x7f35a80f09a0 msgr2=0x7f35a80f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:58:15.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:15.783+0000 7f35a759e700 1 -- 172.21.15.86:0/2479950580 shutdown_connections 2024-02-19T21:58:15.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:15.783+0000 7f35a759e700 1 -- 172.21.15.86:0/2479950580 wait complete. 2024-02-19T21:58:15.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:15.783+0000 7f35a759e700 1 Processor -- start 2024-02-19T21:58:15.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:15.783+0000 7f35a759e700 1 -- start start 2024-02-19T21:58:15.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:15.783+0000 7f35a759e700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f35a80f51a0 0x7f35a81065d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:58:15.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:15.783+0000 7f35a759e700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35a8106ac0 con 0x7f35a80f51a0 2024-02-19T21:58:15.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:15.784+0000 7f35a659c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f35a80f51a0 0x7f35a81065d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:58:15.785 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:15.784+0000 7f35a659c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f35a80f51a0 0x7f35a81065d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:58432/0 (socket says 172.21.15.86:58432) 2024-02-19T21:58:15.785 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:15.784+0000 7f35a659c700 1 -- 172.21.15.86:0/3639082187 learned_addr learned my addr 172.21.15.86:0/3639082187 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T21:58:15.785 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:15.784+0000 7f35a659c700 1 -- 172.21.15.86:0/3639082187 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3598005040 con 0x7f35a80f51a0 2024-02-19T21:58:15.786 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:15.784+0000 7f35a659c700 1 --2- 172.21.15.86:0/3639082187 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f35a80f51a0 0x7f35a81065d0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7f35980089f0 tx=0x7f359800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:58:15.786 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:15.784+0000 7f3596ffd700 1 -- 172.21.15.86:0/3639082187 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3598009320 con 0x7f35a80f51a0 2024-02-19T21:58:15.786 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:15.784+0000 7f3596ffd700 1 -- 172.21.15.86:0/3639082187 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3598013380 con 0x7f35a80f51a0 2024-02-19T21:58:15.786 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:15.784+0000 7f35a759e700 1 -- 172.21.15.86:0/3639082187 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35a8106cc0 con 0x7f35a80f51a0 2024-02-19T21:58:15.786 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:15.784+0000 7f3596ffd700 1 -- 172.21.15.86:0/3639082187 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3598014e20 con 0x7f35a80f51a0 2024-02-19T21:58:15.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:15.784+0000 7f35a759e700 1 -- 172.21.15.86:0/3639082187 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35a8107150 con 0x7f35a80f51a0 2024-02-19T21:58:15.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:15.785+0000 7f3596ffd700 1 -- 172.21.15.86:0/3639082187 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f35980095f0 con 0x7f35a80f51a0 2024-02-19T21:58:15.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:15.785+0000 7f35a759e700 1 -- 172.21.15.86:0/3639082187 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35a8045ad0 con 0x7f35a80f51a0 2024-02-19T21:58:15.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:15.785+0000 7f3596ffd700 1 --2- 172.21.15.86:0/3639082187 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f359002ff10 0x7f35900323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:58:15.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:15.785+0000 7f3596ffd700 1 -- 172.21.15.86:0/3639082187 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3598040f10 con 0x7f35a80f51a0 2024-02-19T21:58:15.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:15.785+0000 7f35a5d9b700 1 --2- 172.21.15.86:0/3639082187 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f359002ff10 0x7f35900323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:58:15.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:15.786+0000 7f35a5d9b700 1 --2- 172.21.15.86:0/3639082187 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f359002ff10 0x7f35900323d0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7f359c0096a0 tx=0x7f359c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:58:15.790 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:15.790+0000 7f3596ffd700 1 -- 172.21.15.86:0/3639082187 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3598014420 con 0x7f35a80f51a0 2024-02-19T21:58:15.995 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:15.996+0000 7f35a759e700 1 -- 172.21.15.86:0/3639082187 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f35a80fad30 con 0x7f35a80f51a0 2024-02-19T21:58:15.996 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:15.996+0000 7f3596ffd700 1 -- 172.21.15.86:0/3639082187 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f359801d980 con 0x7f35a80f51a0 2024-02-19T21:58:15.996 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T21:58:15.996 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T21:58:15.998 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:15.998+0000 7f35a759e700 1 -- 172.21.15.86:0/3639082187 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f359002ff10 msgr2=0x7f35900323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:58:15.998 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:15.998+0000 7f35a759e700 1 --2- 172.21.15.86:0/3639082187 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f359002ff10 0x7f35900323d0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7f359c0096a0 tx=0x7f359c006b40).stop 2024-02-19T21:58:15.998 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:15.998+0000 7f35a759e700 1 -- 172.21.15.86:0/3639082187 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f35a80f51a0 msgr2=0x7f35a81065d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:58:15.998 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:15.998+0000 7f35a759e700 1 --2- 172.21.15.86:0/3639082187 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f35a80f51a0 0x7f35a81065d0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7f35980089f0 tx=0x7f359800b040).stop 2024-02-19T21:58:15.998 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:15.998+0000 7f35a759e700 1 -- 172.21.15.86:0/3639082187 shutdown_connections 2024-02-19T21:58:15.998 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:15.998+0000 7f35a759e700 1 --2- 172.21.15.86:0/3639082187 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f359002ff10 0x7f35900323d0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:58:15.998 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:15.998+0000 7f35a759e700 1 --2- 172.21.15.86:0/3639082187 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f35a80f51a0 0x7f35a81065d0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:58:15.998 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:15.998+0000 7f35a759e700 1 -- 172.21.15.86:0/3639082187 >> 172.21.15.86:0/3639082187 conn(0x7f35a80f09a0 msgr2=0x7f35a805fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:58:15.998 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:15.998+0000 7f35a759e700 1 -- 172.21.15.86:0/3639082187 shutdown_connections 2024-02-19T21:58:15.998 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:15.998+0000 7f35a759e700 1 -- 172.21.15.86:0/3639082187 wait complete. 2024-02-19T21:58:15.999 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T21:58:16.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:58:16 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T21:58:15.450889+0000 mgr.smithi053.fyhcdi (mgr.14162) 51 : cluster [DBG] pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:58:16.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:58:16 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:58:15.997383+0000 mon.smithi053 (mon.0) 135 : audit [DBG] from='client.? 172.21.15.86:0/3639082187' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T21:58:17.648 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T21:58:17.649 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T21:58:18.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:58:18 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T21:58:17.451153+0000 mgr.smithi053.fyhcdi (mgr.14162) 52 : cluster [DBG] pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:58:19.330 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:19.328+0000 7f6e7b25c700 1 -- 172.21.15.86:0/1564561879 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6e740f6480 msgr2=0x7f6e740f68a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:58:19.330 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:19.328+0000 7f6e7b25c700 1 --2- 172.21.15.86:0/1564561879 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6e740f6480 0x7f6e740f68a0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7f6e68009fb0 tx=0x7f6e68009b30).stop 2024-02-19T21:58:19.330 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:19.328+0000 7f6e7b25c700 1 -- 172.21.15.86:0/1564561879 shutdown_connections 2024-02-19T21:58:19.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:19.328+0000 7f6e7b25c700 1 --2- 172.21.15.86:0/1564561879 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6e740f6480 0x7f6e740f68a0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:58:19.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:19.328+0000 7f6e7b25c700 1 -- 172.21.15.86:0/1564561879 >> 172.21.15.86:0/1564561879 conn(0x7f6e740f1c00 msgr2=0x7f6e740f4020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:58:19.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:19.328+0000 7f6e7b25c700 1 -- 172.21.15.86:0/1564561879 shutdown_connections 2024-02-19T21:58:19.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:19.328+0000 7f6e7b25c700 1 -- 172.21.15.86:0/1564561879 wait complete. 2024-02-19T21:58:19.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:19.329+0000 7f6e7b25c700 1 Processor -- start 2024-02-19T21:58:19.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:19.329+0000 7f6e7b25c700 1 -- start start 2024-02-19T21:58:19.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:19.329+0000 7f6e7b25c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6e740f6480 0x7f6e741067f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:58:19.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:19.329+0000 7f6e7b25c700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e74106ce0 con 0x7f6e740f6480 2024-02-19T21:58:19.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:19.329+0000 7f6e78ff8700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6e740f6480 0x7f6e741067f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:58:19.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:19.329+0000 7f6e78ff8700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6e740f6480 0x7f6e741067f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:51008/0 (socket says 172.21.15.86:51008) 2024-02-19T21:58:19.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:19.329+0000 7f6e78ff8700 1 -- 172.21.15.86:0/2929603749 learned_addr learned my addr 172.21.15.86:0/2929603749 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T21:58:19.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:19.330+0000 7f6e78ff8700 1 -- 172.21.15.86:0/2929603749 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e68005040 con 0x7f6e740f6480 2024-02-19T21:58:19.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:19.330+0000 7f6e78ff8700 1 --2- 172.21.15.86:0/2929603749 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6e740f6480 0x7f6e741067f0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f6e68005010 tx=0x7f6e6800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:58:19.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:19.330+0000 7f6e717fa700 1 -- 172.21.15.86:0/2929603749 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6e68009320 con 0x7f6e740f6480 2024-02-19T21:58:19.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:19.330+0000 7f6e717fa700 1 -- 172.21.15.86:0/2929603749 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6e68013380 con 0x7f6e740f6480 2024-02-19T21:58:19.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:19.330+0000 7f6e7b25c700 1 -- 172.21.15.86:0/2929603749 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e74106ee0 con 0x7f6e740f6480 2024-02-19T21:58:19.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:19.330+0000 7f6e717fa700 1 -- 172.21.15.86:0/2929603749 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6e68014e50 con 0x7f6e740f6480 2024-02-19T21:58:19.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:19.330+0000 7f6e7b25c700 1 -- 172.21.15.86:0/2929603749 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e74107370 con 0x7f6e740f6480 2024-02-19T21:58:19.333 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:19.331+0000 7f6e717fa700 1 -- 172.21.15.86:0/2929603749 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6e68014420 con 0x7f6e740f6480 2024-02-19T21:58:19.333 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:19.331+0000 7f6e7b25c700 1 -- 172.21.15.86:0/2929603749 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e74045ad0 con 0x7f6e740f6480 2024-02-19T21:58:19.333 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:19.331+0000 7f6e717fa700 1 --2- 172.21.15.86:0/2929603749 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f6e6402ff10 0x7f6e640323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:58:19.333 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:19.331+0000 7f6e717fa700 1 -- 172.21.15.86:0/2929603749 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6e68040f70 con 0x7f6e740f6480 2024-02-19T21:58:19.333 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:19.331+0000 7f6e73fff700 1 --2- 172.21.15.86:0/2929603749 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f6e6402ff10 0x7f6e640323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:58:19.333 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:19.332+0000 7f6e73fff700 1 --2- 172.21.15.86:0/2929603749 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f6e6402ff10 0x7f6e640323d0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7f6e600096a0 tx=0x7f6e60006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:58:19.336 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:19.336+0000 7f6e717fa700 1 -- 172.21.15.86:0/2929603749 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6e680095f0 con 0x7f6e740f6480 2024-02-19T21:58:19.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:19.539+0000 7f6e7b25c700 1 -- 172.21.15.86:0/2929603749 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6e740fc090 con 0x7f6e740f6480 2024-02-19T21:58:19.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:19.539+0000 7f6e717fa700 1 -- 172.21.15.86:0/2929603749 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6e6801db70 con 0x7f6e740f6480 2024-02-19T21:58:19.539 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T21:58:19.539 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T21:58:19.541 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:19.541+0000 7f6e7b25c700 1 -- 172.21.15.86:0/2929603749 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f6e6402ff10 msgr2=0x7f6e640323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:58:19.541 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:19.541+0000 7f6e7b25c700 1 --2- 172.21.15.86:0/2929603749 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f6e6402ff10 0x7f6e640323d0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7f6e600096a0 tx=0x7f6e60006b40).stop 2024-02-19T21:58:19.541 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:19.541+0000 7f6e7b25c700 1 -- 172.21.15.86:0/2929603749 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6e740f6480 msgr2=0x7f6e741067f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:58:19.541 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:19.541+0000 7f6e7b25c700 1 --2- 172.21.15.86:0/2929603749 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6e740f6480 0x7f6e741067f0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f6e68005010 tx=0x7f6e6800b040).stop 2024-02-19T21:58:19.541 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:19.541+0000 7f6e7b25c700 1 -- 172.21.15.86:0/2929603749 shutdown_connections 2024-02-19T21:58:19.541 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:19.541+0000 7f6e7b25c700 1 --2- 172.21.15.86:0/2929603749 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f6e6402ff10 0x7f6e640323d0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:58:19.541 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:19.541+0000 7f6e7b25c700 1 --2- 172.21.15.86:0/2929603749 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6e740f6480 0x7f6e741067f0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:58:19.541 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:19.541+0000 7f6e7b25c700 1 -- 172.21.15.86:0/2929603749 >> 172.21.15.86:0/2929603749 conn(0x7f6e740f1c00 msgr2=0x7f6e740f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:58:19.541 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:19.541+0000 7f6e7b25c700 1 -- 172.21.15.86:0/2929603749 shutdown_connections 2024-02-19T21:58:19.541 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:19.541+0000 7f6e7b25c700 1 -- 172.21.15.86:0/2929603749 wait complete. 2024-02-19T21:58:19.542 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T21:58:20.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:58:20 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T21:58:19.451354+0000 mgr.smithi053.fyhcdi (mgr.14162) 53 : cluster [DBG] pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:58:20.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:58:20 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:58:19.540641+0000 mon.smithi053 (mon.0) 136 : audit [DBG] from='client.? 172.21.15.86:0/2929603749' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T21:58:21.155 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T21:58:21.155 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T21:58:22.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:58:22 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T21:58:21.451606+0000 mgr.smithi053.fyhcdi (mgr.14162) 54 : cluster [DBG] pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:58:22.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:22.897+0000 7f8d8df9d700 1 -- 172.21.15.86:0/3945819936 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8d880601e0 msgr2=0x7f8d880f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:58:22.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:22.897+0000 7f8d8df9d700 1 --2- 172.21.15.86:0/3945819936 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8d880601e0 0x7f8d880f8f20 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7f8d7c009fb0 tx=0x7f8d7c009b30).stop 2024-02-19T21:58:22.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:22.897+0000 7f8d8df9d700 1 -- 172.21.15.86:0/3945819936 shutdown_connections 2024-02-19T21:58:22.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:22.897+0000 7f8d8df9d700 1 --2- 172.21.15.86:0/3945819936 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8d880601e0 0x7f8d880f8f20 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:58:22.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:22.897+0000 7f8d8df9d700 1 -- 172.21.15.86:0/3945819936 >> 172.21.15.86:0/3945819936 conn(0x7f8d880f09a0 msgr2=0x7f8d880f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:58:22.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:22.897+0000 7f8d8df9d700 1 -- 172.21.15.86:0/3945819936 shutdown_connections 2024-02-19T21:58:22.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:22.897+0000 7f8d8df9d700 1 -- 172.21.15.86:0/3945819936 wait complete. 2024-02-19T21:58:22.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:22.897+0000 7f8d8df9d700 1 Processor -- start 2024-02-19T21:58:22.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:22.897+0000 7f8d8df9d700 1 -- start start 2024-02-19T21:58:22.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:22.898+0000 7f8d8df9d700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8d880601e0 0x7f8d88106600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:58:22.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:22.898+0000 7f8d8df9d700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d88106af0 con 0x7f8d880601e0 2024-02-19T21:58:22.899 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:22.898+0000 7f8d8cf9b700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8d880601e0 0x7f8d88106600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:58:22.899 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:22.898+0000 7f8d8cf9b700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8d880601e0 0x7f8d88106600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:51024/0 (socket says 172.21.15.86:51024) 2024-02-19T21:58:22.899 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:22.898+0000 7f8d8cf9b700 1 -- 172.21.15.86:0/2094183617 learned_addr learned my addr 172.21.15.86:0/2094183617 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T21:58:22.899 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:22.898+0000 7f8d8cf9b700 1 -- 172.21.15.86:0/2094183617 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d7c005040 con 0x7f8d880601e0 2024-02-19T21:58:22.899 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:22.898+0000 7f8d8cf9b700 1 --2- 172.21.15.86:0/2094183617 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8d880601e0 0x7f8d88106600 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f8d7c005010 tx=0x7f8d7c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:58:22.900 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:22.898+0000 7f8d857fa700 1 -- 172.21.15.86:0/2094183617 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8d7c009320 con 0x7f8d880601e0 2024-02-19T21:58:22.900 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:22.898+0000 7f8d8df9d700 1 -- 172.21.15.86:0/2094183617 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d88106cf0 con 0x7f8d880601e0 2024-02-19T21:58:22.900 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:22.898+0000 7f8d857fa700 1 -- 172.21.15.86:0/2094183617 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8d7c013380 con 0x7f8d880601e0 2024-02-19T21:58:22.900 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:22.898+0000 7f8d857fa700 1 -- 172.21.15.86:0/2094183617 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8d7c014e50 con 0x7f8d880601e0 2024-02-19T21:58:22.900 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:22.898+0000 7f8d8df9d700 1 -- 172.21.15.86:0/2094183617 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d88107180 con 0x7f8d880601e0 2024-02-19T21:58:22.900 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:22.899+0000 7f8d857fa700 1 -- 172.21.15.86:0/2094183617 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8d7c0263a0 con 0x7f8d880601e0 2024-02-19T21:58:22.900 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:22.899+0000 7f8d857fa700 1 --2- 172.21.15.86:0/2094183617 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f8d7802ff10 0x7f8d780323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:58:22.900 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:22.899+0000 7f8d8df9d700 1 -- 172.21.15.86:0/2094183617 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d88045ad0 con 0x7f8d880601e0 2024-02-19T21:58:22.901 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:22.899+0000 7f8d857fa700 1 -- 172.21.15.86:0/2094183617 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8d7c040fb0 con 0x7f8d880601e0 2024-02-19T21:58:22.901 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:22.900+0000 7f8d87fff700 1 --2- 172.21.15.86:0/2094183617 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f8d7802ff10 0x7f8d780323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:58:22.901 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:22.900+0000 7f8d87fff700 1 --2- 172.21.15.86:0/2094183617 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f8d7802ff10 0x7f8d780323d0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7f8d740096a0 tx=0x7f8d74006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:58:22.904 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:22.904+0000 7f8d857fa700 1 -- 172.21.15.86:0/2094183617 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8d7c012920 con 0x7f8d880601e0 2024-02-19T21:58:23.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:23.108+0000 7f8d8df9d700 1 -- 172.21.15.86:0/2094183617 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8d88108030 con 0x7f8d880601e0 2024-02-19T21:58:23.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:23.109+0000 7f8d857fa700 1 -- 172.21.15.86:0/2094183617 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8d7c00f340 con 0x7f8d880601e0 2024-02-19T21:58:23.109 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T21:58:23.109 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T21:58:23.110 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:23.110+0000 7f8d8df9d700 1 -- 172.21.15.86:0/2094183617 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f8d7802ff10 msgr2=0x7f8d780323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:58:23.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:23.110+0000 7f8d8df9d700 1 --2- 172.21.15.86:0/2094183617 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f8d7802ff10 0x7f8d780323d0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7f8d740096a0 tx=0x7f8d74006b40).stop 2024-02-19T21:58:23.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:23.110+0000 7f8d8df9d700 1 -- 172.21.15.86:0/2094183617 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8d880601e0 msgr2=0x7f8d88106600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:58:23.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:23.110+0000 7f8d8df9d700 1 --2- 172.21.15.86:0/2094183617 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8d880601e0 0x7f8d88106600 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f8d7c005010 tx=0x7f8d7c00b040).stop 2024-02-19T21:58:23.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:23.111+0000 7f8d8df9d700 1 -- 172.21.15.86:0/2094183617 shutdown_connections 2024-02-19T21:58:23.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:23.111+0000 7f8d8df9d700 1 --2- 172.21.15.86:0/2094183617 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f8d7802ff10 0x7f8d780323d0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:58:23.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:23.111+0000 7f8d8df9d700 1 --2- 172.21.15.86:0/2094183617 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8d880601e0 0x7f8d88106600 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:58:23.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:23.111+0000 7f8d8df9d700 1 -- 172.21.15.86:0/2094183617 >> 172.21.15.86:0/2094183617 conn(0x7f8d880f09a0 msgr2=0x7f8d880f13d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:58:23.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:23.111+0000 7f8d8df9d700 1 -- 172.21.15.86:0/2094183617 shutdown_connections 2024-02-19T21:58:23.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:23.111+0000 7f8d8df9d700 1 -- 172.21.15.86:0/2094183617 wait complete. 2024-02-19T21:58:23.112 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T21:58:23.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:58:23 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:58:23.110259+0000 mon.smithi053 (mon.0) 137 : audit [DBG] from='client.? 172.21.15.86:0/2094183617' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T21:58:24.689 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T21:58:24.690 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T21:58:24.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:58:24 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T21:58:23.451929+0000 mgr.smithi053.fyhcdi (mgr.14162) 55 : cluster [DBG] pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:58:26.430 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:26.429+0000 7f7605980700 1 -- 172.21.15.86:0/3811645802 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f76000f51a0 msgr2=0x7f76000f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:58:26.430 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:26.429+0000 7f7605980700 1 --2- 172.21.15.86:0/3811645802 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f76000f51a0 0x7f76000f55c0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f75f4009fb0 tx=0x7f75f4009b30).stop 2024-02-19T21:58:26.430 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:26.429+0000 7f7605980700 1 -- 172.21.15.86:0/3811645802 shutdown_connections 2024-02-19T21:58:26.430 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:26.429+0000 7f7605980700 1 --2- 172.21.15.86:0/3811645802 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f76000f51a0 0x7f76000f55c0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:58:26.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:26.429+0000 7f7605980700 1 -- 172.21.15.86:0/3811645802 >> 172.21.15.86:0/3811645802 conn(0x7f76000f09a0 msgr2=0x7f76000f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:58:26.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:26.430+0000 7f7605980700 1 -- 172.21.15.86:0/3811645802 shutdown_connections 2024-02-19T21:58:26.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:26.430+0000 7f7605980700 1 -- 172.21.15.86:0/3811645802 wait complete. 2024-02-19T21:58:26.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:26.430+0000 7f7605980700 1 Processor -- start 2024-02-19T21:58:26.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:26.430+0000 7f7605980700 1 -- start start 2024-02-19T21:58:26.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:26.430+0000 7f7605980700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f76000f51a0 0x7f76001054d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:58:26.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:26.430+0000 7f7605980700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76001059c0 con 0x7f76000f51a0 2024-02-19T21:58:26.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:26.431+0000 7f760497e700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f76000f51a0 0x7f76001054d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:58:26.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:26.431+0000 7f760497e700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f76000f51a0 0x7f76001054d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:51030/0 (socket says 172.21.15.86:51030) 2024-02-19T21:58:26.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:26.431+0000 7f760497e700 1 -- 172.21.15.86:0/577057317 learned_addr learned my addr 172.21.15.86:0/577057317 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T21:58:26.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:26.431+0000 7f760497e700 1 -- 172.21.15.86:0/577057317 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75f4005040 con 0x7f76000f51a0 2024-02-19T21:58:26.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:26.431+0000 7f760497e700 1 --2- 172.21.15.86:0/577057317 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f76000f51a0 0x7f76001054d0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f75f4009830 tx=0x7f75f400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:58:26.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:26.431+0000 7f75fd7fa700 1 -- 172.21.15.86:0/577057317 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f75f4009320 con 0x7f76000f51a0 2024-02-19T21:58:26.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:26.431+0000 7f7605980700 1 -- 172.21.15.86:0/577057317 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7600105bc0 con 0x7f76000f51a0 2024-02-19T21:58:26.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:26.431+0000 7f75fd7fa700 1 -- 172.21.15.86:0/577057317 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f75f4013380 con 0x7f76000f51a0 2024-02-19T21:58:26.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:26.431+0000 7f7605980700 1 -- 172.21.15.86:0/577057317 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7600106050 con 0x7f76000f51a0 2024-02-19T21:58:26.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:26.431+0000 7f75fd7fa700 1 -- 172.21.15.86:0/577057317 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f75f4014420 con 0x7f76000f51a0 2024-02-19T21:58:26.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:26.432+0000 7f75fd7fa700 1 -- 172.21.15.86:0/577057317 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f75f401e420 con 0x7f76000f51a0 2024-02-19T21:58:26.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:26.432+0000 7f7605980700 1 -- 172.21.15.86:0/577057317 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7600045ad0 con 0x7f76000f51a0 2024-02-19T21:58:26.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:26.432+0000 7f75fd7fa700 1 --2- 172.21.15.86:0/577057317 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f75f002ff60 0x7f75f0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:58:26.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:26.432+0000 7f75fd7fa700 1 -- 172.21.15.86:0/577057317 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f75f4040ee0 con 0x7f76000f51a0 2024-02-19T21:58:26.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:26.432+0000 7f75fffff700 1 --2- 172.21.15.86:0/577057317 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f75f002ff60 0x7f75f0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:58:26.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:26.433+0000 7f75fffff700 1 --2- 172.21.15.86:0/577057317 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f75f002ff60 0x7f75f0032420 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f75ec0096a0 tx=0x7f75ec006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:58:26.436 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:26.437+0000 7f75fd7fa700 1 -- 172.21.15.86:0/577057317 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f75f401c970 con 0x7f76000f51a0 2024-02-19T21:58:26.642 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:26.642+0000 7f7605980700 1 -- 172.21.15.86:0/577057317 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f76000fad30 con 0x7f76000f51a0 2024-02-19T21:58:26.642 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:26.643+0000 7f75fd7fa700 1 -- 172.21.15.86:0/577057317 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f75f402c2d0 con 0x7f76000f51a0 2024-02-19T21:58:26.643 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T21:58:26.643 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T21:58:26.644 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:26.644+0000 7f7605980700 1 -- 172.21.15.86:0/577057317 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f75f002ff60 msgr2=0x7f75f0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:58:26.644 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:26.644+0000 7f7605980700 1 --2- 172.21.15.86:0/577057317 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f75f002ff60 0x7f75f0032420 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f75ec0096a0 tx=0x7f75ec006b40).stop 2024-02-19T21:58:26.644 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:26.644+0000 7f7605980700 1 -- 172.21.15.86:0/577057317 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f76000f51a0 msgr2=0x7f76001054d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:58:26.644 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:26.644+0000 7f7605980700 1 --2- 172.21.15.86:0/577057317 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f76000f51a0 0x7f76001054d0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f75f4009830 tx=0x7f75f400b040).stop 2024-02-19T21:58:26.645 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:26.644+0000 7f7605980700 1 -- 172.21.15.86:0/577057317 shutdown_connections 2024-02-19T21:58:26.645 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:26.644+0000 7f7605980700 1 --2- 172.21.15.86:0/577057317 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f75f002ff60 0x7f75f0032420 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:58:26.645 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:26.644+0000 7f7605980700 1 --2- 172.21.15.86:0/577057317 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f76000f51a0 0x7f76001054d0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:58:26.645 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:26.644+0000 7f7605980700 1 -- 172.21.15.86:0/577057317 >> 172.21.15.86:0/577057317 conn(0x7f76000f09a0 msgr2=0x7f76000f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:58:26.645 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:26.644+0000 7f7605980700 1 -- 172.21.15.86:0/577057317 shutdown_connections 2024-02-19T21:58:26.645 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:26.644+0000 7f7605980700 1 -- 172.21.15.86:0/577057317 wait complete. 2024-02-19T21:58:26.645 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T21:58:26.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:58:26 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T21:58:25.452298+0000 mgr.smithi053.fyhcdi (mgr.14162) 56 : cluster [DBG] pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:58:27.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:58:27 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:58:26.643842+0000 mon.smithi053 (mon.0) 138 : audit [DBG] from='client.? 172.21.15.86:0/577057317' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T21:58:28.171 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T21:58:28.172 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T21:58:28.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:58:28 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T21:58:27.452646+0000 mgr.smithi053.fyhcdi (mgr.14162) 57 : cluster [DBG] pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:58:29.931 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:29.931+0000 7fbcbf2b7700 1 -- 172.21.15.86:0/3840224696 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbcb80f6510 msgr2=0x7fbcb80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:58:29.931 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:29.931+0000 7fbcbf2b7700 1 --2- 172.21.15.86:0/3840224696 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbcb80f6510 0x7fbcb80f6930 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7fbca8009fb0 tx=0x7fbca8009b30).stop 2024-02-19T21:58:29.931 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:29.931+0000 7fbcbf2b7700 1 -- 172.21.15.86:0/3840224696 shutdown_connections 2024-02-19T21:58:29.931 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:29.931+0000 7fbcbf2b7700 1 --2- 172.21.15.86:0/3840224696 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbcb80f6510 0x7fbcb80f6930 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:58:29.931 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:29.931+0000 7fbcbf2b7700 1 -- 172.21.15.86:0/3840224696 >> 172.21.15.86:0/3840224696 conn(0x7fbcb80f1c70 msgr2=0x7fbcb80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:58:29.932 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:29.931+0000 7fbcbf2b7700 1 -- 172.21.15.86:0/3840224696 shutdown_connections 2024-02-19T21:58:29.932 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:29.931+0000 7fbcbf2b7700 1 -- 172.21.15.86:0/3840224696 wait complete. 2024-02-19T21:58:29.932 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:29.932+0000 7fbcbf2b7700 1 Processor -- start 2024-02-19T21:58:29.932 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:29.932+0000 7fbcbf2b7700 1 -- start start 2024-02-19T21:58:29.932 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:29.932+0000 7fbcbf2b7700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbcb80f6510 0x7fbcb81067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:58:29.932 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:29.932+0000 7fbcbf2b7700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcb8106cd0 con 0x7fbcb80f6510 2024-02-19T21:58:29.932 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:29.932+0000 7fbcbd053700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbcb80f6510 0x7fbcb81067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:58:29.932 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:29.932+0000 7fbcbd053700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbcb80f6510 0x7fbcb81067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:50576/0 (socket says 172.21.15.86:50576) 2024-02-19T21:58:29.932 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:29.932+0000 7fbcbd053700 1 -- 172.21.15.86:0/2981402434 learned_addr learned my addr 172.21.15.86:0/2981402434 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T21:58:29.932 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:29.932+0000 7fbcbd053700 1 -- 172.21.15.86:0/2981402434 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbca8005040 con 0x7fbcb80f6510 2024-02-19T21:58:29.932 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:29.932+0000 7fbcbd053700 1 --2- 172.21.15.86:0/2981402434 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbcb80f6510 0x7fbcb81067e0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7fbca8005010 tx=0x7fbca800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:58:29.933 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:29.933+0000 7fbcadffb700 1 -- 172.21.15.86:0/2981402434 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbca8009320 con 0x7fbcb80f6510 2024-02-19T21:58:29.934 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:29.933+0000 7fbcbf2b7700 1 -- 172.21.15.86:0/2981402434 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbcb8106ed0 con 0x7fbcb80f6510 2024-02-19T21:58:29.934 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:29.933+0000 7fbcbf2b7700 1 -- 172.21.15.86:0/2981402434 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbcb8107360 con 0x7fbcb80f6510 2024-02-19T21:58:29.934 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:29.933+0000 7fbcadffb700 1 -- 172.21.15.86:0/2981402434 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbca8013380 con 0x7fbcb80f6510 2024-02-19T21:58:29.934 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:29.933+0000 7fbcadffb700 1 -- 172.21.15.86:0/2981402434 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbca8014df0 con 0x7fbcb80f6510 2024-02-19T21:58:29.934 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:29.934+0000 7fbcadffb700 1 -- 172.21.15.86:0/2981402434 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbca8014420 con 0x7fbcb80f6510 2024-02-19T21:58:29.934 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:29.934+0000 7fbcbf2b7700 1 -- 172.21.15.86:0/2981402434 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbcb8045ad0 con 0x7fbcb80f6510 2024-02-19T21:58:29.934 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:29.934+0000 7fbcadffb700 1 --2- 172.21.15.86:0/2981402434 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fbca402ff10 0x7fbca40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:58:29.934 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:29.934+0000 7fbcadffb700 1 -- 172.21.15.86:0/2981402434 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbca80411d0 con 0x7fbcb80f6510 2024-02-19T21:58:29.934 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:29.934+0000 7fbcbc852700 1 --2- 172.21.15.86:0/2981402434 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fbca402ff10 0x7fbca40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:58:29.934 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:29.934+0000 7fbcbc852700 1 --2- 172.21.15.86:0/2981402434 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fbca402ff10 0x7fbca40323d0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7fbcb40096a0 tx=0x7fbcb4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:58:29.937 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:29.939+0000 7fbcadffb700 1 -- 172.21.15.86:0/2981402434 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbca800f990 con 0x7fbcb80f6510 2024-02-19T21:58:30.142 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:30.144+0000 7fbcbf2b7700 1 -- 172.21.15.86:0/2981402434 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbcb8108210 con 0x7fbcb80f6510 2024-02-19T21:58:30.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:30.145+0000 7fbcadffb700 1 -- 172.21.15.86:0/2981402434 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fbca800f7b0 con 0x7fbcb80f6510 2024-02-19T21:58:30.144 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T21:58:30.144 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T21:58:30.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:30.146+0000 7fbcbf2b7700 1 -- 172.21.15.86:0/2981402434 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fbca402ff10 msgr2=0x7fbca40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:58:30.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:30.146+0000 7fbcbf2b7700 1 --2- 172.21.15.86:0/2981402434 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fbca402ff10 0x7fbca40323d0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7fbcb40096a0 tx=0x7fbcb4006b40).stop 2024-02-19T21:58:30.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:30.146+0000 7fbcbf2b7700 1 -- 172.21.15.86:0/2981402434 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbcb80f6510 msgr2=0x7fbcb81067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:58:30.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:30.146+0000 7fbcbf2b7700 1 --2- 172.21.15.86:0/2981402434 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbcb80f6510 0x7fbcb81067e0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7fbca8005010 tx=0x7fbca800b040).stop 2024-02-19T21:58:30.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:30.146+0000 7fbcbf2b7700 1 -- 172.21.15.86:0/2981402434 shutdown_connections 2024-02-19T21:58:30.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:30.146+0000 7fbcbf2b7700 1 --2- 172.21.15.86:0/2981402434 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fbca402ff10 0x7fbca40323d0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:58:30.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:30.146+0000 7fbcbf2b7700 1 --2- 172.21.15.86:0/2981402434 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbcb80f6510 0x7fbcb81067e0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:58:30.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:30.146+0000 7fbcbf2b7700 1 -- 172.21.15.86:0/2981402434 >> 172.21.15.86:0/2981402434 conn(0x7fbcb80f1c70 msgr2=0x7fbcb80f26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:58:30.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:30.146+0000 7fbcbf2b7700 1 -- 172.21.15.86:0/2981402434 shutdown_connections 2024-02-19T21:58:30.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:30.146+0000 7fbcbf2b7700 1 -- 172.21.15.86:0/2981402434 wait complete. 2024-02-19T21:58:30.146 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T21:58:30.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:58:30 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T21:58:29.452907+0000 mgr.smithi053.fyhcdi (mgr.14162) 58 : cluster [DBG] pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:58:30.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:58:30 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:58:30.144761+0000 mon.smithi053 (mon.0) 139 : audit [DBG] from='client.? 172.21.15.86:0/2981402434' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T21:58:31.784 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T21:58:31.785 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T21:58:32.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:58:32 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T21:58:31.453266+0000 mgr.smithi053.fyhcdi (mgr.14162) 59 : cluster [DBG] pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:58:33.573 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:33.573+0000 7f138abe0700 1 -- 172.21.15.86:0/1893526764 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f13840f5410 msgr2=0x7f13840f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:58:33.573 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:33.573+0000 7f138abe0700 1 --2- 172.21.15.86:0/1893526764 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f13840f5410 0x7f13840f5830 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7f1378009fb0 tx=0x7f1378009b30).stop 2024-02-19T21:58:33.573 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:33.573+0000 7f138abe0700 1 -- 172.21.15.86:0/1893526764 shutdown_connections 2024-02-19T21:58:33.573 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:33.573+0000 7f138abe0700 1 --2- 172.21.15.86:0/1893526764 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f13840f5410 0x7f13840f5830 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:58:33.573 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:33.573+0000 7f138abe0700 1 -- 172.21.15.86:0/1893526764 >> 172.21.15.86:0/1893526764 conn(0x7f13840f0b70 msgr2=0x7f13840f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:58:33.573 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:33.573+0000 7f138abe0700 1 -- 172.21.15.86:0/1893526764 shutdown_connections 2024-02-19T21:58:33.573 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:33.573+0000 7f138abe0700 1 -- 172.21.15.86:0/1893526764 wait complete. 2024-02-19T21:58:33.573 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:33.574+0000 7f138abe0700 1 Processor -- start 2024-02-19T21:58:33.573 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:33.574+0000 7f138abe0700 1 -- start start 2024-02-19T21:58:33.573 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:33.574+0000 7f138abe0700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f13840f5410 0x7f1384109510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:58:33.574 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:33.574+0000 7f138abe0700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1384060240 con 0x7f13840f5410 2024-02-19T21:58:33.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:33.574+0000 7f138897c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f13840f5410 0x7f1384109510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:58:33.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:33.574+0000 7f138897c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f13840f5410 0x7f1384109510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:50590/0 (socket says 172.21.15.86:50590) 2024-02-19T21:58:33.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:33.574+0000 7f138897c700 1 -- 172.21.15.86:0/2856846198 learned_addr learned my addr 172.21.15.86:0/2856846198 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T21:58:33.576 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:33.575+0000 7f138897c700 1 -- 172.21.15.86:0/2856846198 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1378005040 con 0x7f13840f5410 2024-02-19T21:58:33.576 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:33.575+0000 7f138897c700 1 --2- 172.21.15.86:0/2856846198 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f13840f5410 0x7f1384109510 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f1378005010 tx=0x7f137800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:58:33.576 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:33.575+0000 7f13817fa700 1 -- 172.21.15.86:0/2856846198 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1378009320 con 0x7f13840f5410 2024-02-19T21:58:33.576 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:33.575+0000 7f138abe0700 1 -- 172.21.15.86:0/2856846198 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1384060440 con 0x7f13840f5410 2024-02-19T21:58:33.576 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:33.575+0000 7f13817fa700 1 -- 172.21.15.86:0/2856846198 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1378013380 con 0x7f13840f5410 2024-02-19T21:58:33.576 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:33.575+0000 7f138abe0700 1 -- 172.21.15.86:0/2856846198 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1384109bc0 con 0x7f13840f5410 2024-02-19T21:58:33.576 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:33.575+0000 7f13817fa700 1 -- 172.21.15.86:0/2856846198 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1378014e50 con 0x7f13840f5410 2024-02-19T21:58:33.576 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:33.576+0000 7f13817fa700 1 -- 172.21.15.86:0/2856846198 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f13780263a0 con 0x7f13840f5410 2024-02-19T21:58:33.576 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:33.576+0000 7f13817fa700 1 --2- 172.21.15.86:0/2856846198 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f137402ff10 0x7f13740323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:58:33.576 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:33.576+0000 7f13817fa700 1 -- 172.21.15.86:0/2856846198 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1378040ff0 con 0x7f13840f5410 2024-02-19T21:58:33.576 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:33.576+0000 7f1383fff700 1 --2- 172.21.15.86:0/2856846198 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f137402ff10 0x7f13740323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:58:33.576 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:33.576+0000 7f138abe0700 1 -- 172.21.15.86:0/2856846198 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1384059100 con 0x7f13840f5410 2024-02-19T21:58:33.576 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:33.576+0000 7f1383fff700 1 --2- 172.21.15.86:0/2856846198 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f137402ff10 0x7f13740323d0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f13700096a0 tx=0x7f1370006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:58:33.580 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:33.581+0000 7f13817fa700 1 -- 172.21.15.86:0/2856846198 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1378026630 con 0x7f13840f5410 2024-02-19T21:58:33.786 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:33.787+0000 7f138abe0700 1 -- 172.21.15.86:0/2856846198 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1384045ad0 con 0x7f13840f5410 2024-02-19T21:58:33.786 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:33.787+0000 7f13817fa700 1 -- 172.21.15.86:0/2856846198 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f137802b020 con 0x7f13840f5410 2024-02-19T21:58:33.787 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T21:58:33.787 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T21:58:33.788 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:33.789+0000 7f138abe0700 1 -- 172.21.15.86:0/2856846198 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f137402ff10 msgr2=0x7f13740323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:58:33.788 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:33.789+0000 7f138abe0700 1 --2- 172.21.15.86:0/2856846198 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f137402ff10 0x7f13740323d0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f13700096a0 tx=0x7f1370006b40).stop 2024-02-19T21:58:33.788 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:33.789+0000 7f138abe0700 1 -- 172.21.15.86:0/2856846198 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f13840f5410 msgr2=0x7f1384109510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:58:33.788 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:33.789+0000 7f138abe0700 1 --2- 172.21.15.86:0/2856846198 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f13840f5410 0x7f1384109510 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f1378005010 tx=0x7f137800b040).stop 2024-02-19T21:58:33.788 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:33.789+0000 7f138abe0700 1 -- 172.21.15.86:0/2856846198 shutdown_connections 2024-02-19T21:58:33.788 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:33.789+0000 7f138abe0700 1 --2- 172.21.15.86:0/2856846198 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f137402ff10 0x7f13740323d0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:58:33.788 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:33.789+0000 7f138abe0700 1 --2- 172.21.15.86:0/2856846198 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f13840f5410 0x7f1384109510 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:58:33.788 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:33.789+0000 7f138abe0700 1 -- 172.21.15.86:0/2856846198 >> 172.21.15.86:0/2856846198 conn(0x7f13840f0b70 msgr2=0x7f13840f15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:58:33.789 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:33.789+0000 7f138abe0700 1 -- 172.21.15.86:0/2856846198 shutdown_connections 2024-02-19T21:58:33.789 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:33.789+0000 7f138abe0700 1 -- 172.21.15.86:0/2856846198 wait complete. 2024-02-19T21:58:33.789 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T21:58:34.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:58:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T21:58:33.453503+0000 mgr.smithi053.fyhcdi (mgr.14162) 60 : cluster [DBG] pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:58:34.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:58:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:58:33.787537+0000 mon.smithi053 (mon.0) 140 : audit [DBG] from='client.? 172.21.15.86:0/2856846198' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T21:58:35.362 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T21:58:35.362 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T21:58:36.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:58:36 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T21:58:35.453781+0000 mgr.smithi053.fyhcdi (mgr.14162) 61 : cluster [DBG] pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:58:36.794 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:58:36 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:58:35.597867+0000 mon.smithi053 (mon.0) 141 : audit [INF] from='mgr.14162 172.21.15.53:0/2860358250' entity='mgr.smithi053.fyhcdi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi053.fyhcdi/mirror_snapshot_schedule"}]: dispatch 2024-02-19T21:58:36.794 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:58:36 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:58:35.601688+0000 mon.smithi053 (mon.0) 142 : audit [INF] from='mgr.14162 172.21.15.53:0/2860358250' entity='mgr.smithi053.fyhcdi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi053.fyhcdi/trash_purge_schedule"}]: dispatch 2024-02-19T21:58:37.138 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:37.138+0000 7ff8897c6700 1 -- 172.21.15.86:0/1779065062 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff8840601e0 msgr2=0x7ff8840f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:58:37.138 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:37.138+0000 7ff8897c6700 1 --2- 172.21.15.86:0/1779065062 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff8840601e0 0x7ff8840f8f20 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7ff874009fb0 tx=0x7ff874009b30).stop 2024-02-19T21:58:37.138 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:37.138+0000 7ff8897c6700 1 -- 172.21.15.86:0/1779065062 shutdown_connections 2024-02-19T21:58:37.138 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:37.138+0000 7ff8897c6700 1 --2- 172.21.15.86:0/1779065062 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff8840601e0 0x7ff8840f8f20 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:58:37.139 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:37.138+0000 7ff8897c6700 1 -- 172.21.15.86:0/1779065062 >> 172.21.15.86:0/1779065062 conn(0x7ff8840f09a0 msgr2=0x7ff8840f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:58:37.139 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:37.138+0000 7ff8897c6700 1 -- 172.21.15.86:0/1779065062 shutdown_connections 2024-02-19T21:58:37.139 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:37.138+0000 7ff8897c6700 1 -- 172.21.15.86:0/1779065062 wait complete. 2024-02-19T21:58:37.139 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:37.138+0000 7ff8897c6700 1 Processor -- start 2024-02-19T21:58:37.139 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:37.139+0000 7ff8897c6700 1 -- start start 2024-02-19T21:58:37.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:37.139+0000 7ff8897c6700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff8840601e0 0x7ff884106600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:58:37.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:37.139+0000 7ff8897c6700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff884106af0 con 0x7ff8840601e0 2024-02-19T21:58:37.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:37.139+0000 7ff883fff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff8840601e0 0x7ff884106600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:58:37.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:37.139+0000 7ff883fff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff8840601e0 0x7ff884106600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:50606/0 (socket says 172.21.15.86:50606) 2024-02-19T21:58:37.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:37.139+0000 7ff883fff700 1 -- 172.21.15.86:0/1417227200 learned_addr learned my addr 172.21.15.86:0/1417227200 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T21:58:37.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:37.139+0000 7ff883fff700 1 -- 172.21.15.86:0/1417227200 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff874005040 con 0x7ff8840601e0 2024-02-19T21:58:37.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:37.139+0000 7ff883fff700 1 --2- 172.21.15.86:0/1417227200 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff8840601e0 0x7ff884106600 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7ff874005010 tx=0x7ff87400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:58:37.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:37.140+0000 7ff880ff9700 1 -- 172.21.15.86:0/1417227200 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff874009320 con 0x7ff8840601e0 2024-02-19T21:58:37.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:37.140+0000 7ff8897c6700 1 -- 172.21.15.86:0/1417227200 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff884106cf0 con 0x7ff8840601e0 2024-02-19T21:58:37.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:37.140+0000 7ff880ff9700 1 -- 172.21.15.86:0/1417227200 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff874013380 con 0x7ff8840601e0 2024-02-19T21:58:37.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:37.140+0000 7ff880ff9700 1 -- 172.21.15.86:0/1417227200 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff874014e50 con 0x7ff8840601e0 2024-02-19T21:58:37.142 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:37.140+0000 7ff8897c6700 1 -- 172.21.15.86:0/1417227200 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff884107180 con 0x7ff8840601e0 2024-02-19T21:58:37.142 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:37.141+0000 7ff880ff9700 1 -- 172.21.15.86:0/1417227200 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff874014420 con 0x7ff8840601e0 2024-02-19T21:58:37.142 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:37.141+0000 7ff880ff9700 1 --2- 172.21.15.86:0/1417227200 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7ff86c02ff10 0x7ff86c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:58:37.142 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:37.141+0000 7ff880ff9700 1 -- 172.21.15.86:0/1417227200 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff874040f70 con 0x7ff8840601e0 2024-02-19T21:58:37.142 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:37.141+0000 7ff8897c6700 1 -- 172.21.15.86:0/1417227200 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff884045ad0 con 0x7ff8840601e0 2024-02-19T21:58:37.142 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:37.141+0000 7ff8837fe700 1 --2- 172.21.15.86:0/1417227200 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7ff86c02ff10 0x7ff86c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:58:37.142 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:37.142+0000 7ff8837fe700 1 --2- 172.21.15.86:0/1417227200 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7ff86c02ff10 0x7ff86c0323d0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7ff8780096a0 tx=0x7ff878006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:58:37.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:37.146+0000 7ff880ff9700 1 -- 172.21.15.86:0/1417227200 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff8740095f0 con 0x7ff8840601e0 2024-02-19T21:58:37.349 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:37.350+0000 7ff8897c6700 1 -- 172.21.15.86:0/1417227200 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff884107ed0 con 0x7ff8840601e0 2024-02-19T21:58:37.350 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:37.350+0000 7ff880ff9700 1 -- 172.21.15.86:0/1417227200 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff87401db70 con 0x7ff8840601e0 2024-02-19T21:58:37.350 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T21:58:37.350 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T21:58:37.352 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:37.352+0000 7ff8897c6700 1 -- 172.21.15.86:0/1417227200 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7ff86c02ff10 msgr2=0x7ff86c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:58:37.352 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:37.352+0000 7ff8897c6700 1 --2- 172.21.15.86:0/1417227200 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7ff86c02ff10 0x7ff86c0323d0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7ff8780096a0 tx=0x7ff878006b40).stop 2024-02-19T21:58:37.352 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:37.352+0000 7ff8897c6700 1 -- 172.21.15.86:0/1417227200 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff8840601e0 msgr2=0x7ff884106600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:58:37.352 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:37.352+0000 7ff8897c6700 1 --2- 172.21.15.86:0/1417227200 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff8840601e0 0x7ff884106600 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7ff874005010 tx=0x7ff87400b040).stop 2024-02-19T21:58:37.352 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:37.352+0000 7ff8897c6700 1 -- 172.21.15.86:0/1417227200 shutdown_connections 2024-02-19T21:58:37.352 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:37.352+0000 7ff8897c6700 1 --2- 172.21.15.86:0/1417227200 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7ff86c02ff10 0x7ff86c0323d0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:58:37.352 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:37.352+0000 7ff8897c6700 1 --2- 172.21.15.86:0/1417227200 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff8840601e0 0x7ff884106600 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:58:37.352 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:37.352+0000 7ff8897c6700 1 -- 172.21.15.86:0/1417227200 >> 172.21.15.86:0/1417227200 conn(0x7ff8840f09a0 msgr2=0x7ff8840f13d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:58:37.352 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:37.352+0000 7ff8897c6700 1 -- 172.21.15.86:0/1417227200 shutdown_connections 2024-02-19T21:58:37.352 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:37.352+0000 7ff8897c6700 1 -- 172.21.15.86:0/1417227200 wait complete. 2024-02-19T21:58:37.353 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T21:58:37.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:58:37 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:58:37.351649+0000 mon.smithi053 (mon.0) 143 : audit [DBG] from='client.? 172.21.15.86:0/1417227200' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T21:58:38.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:58:38 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T21:58:37.454194+0000 mgr.smithi053.fyhcdi (mgr.14162) 62 : cluster [DBG] pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:58:38.970 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T21:58:38.971 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T21:58:40.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:58:40 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T21:58:39.454438+0000 mgr.smithi053.fyhcdi (mgr.14162) 63 : cluster [DBG] pgmap v56: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:58:41.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:41.006+0000 7fda0acdb700 1 -- 172.21.15.86:0/2031210182 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fda040f51a0 msgr2=0x7fda040f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:58:41.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:41.006+0000 7fda0acdb700 1 --2- 172.21.15.86:0/2031210182 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fda040f51a0 0x7fda040f55c0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7fda00009fb0 tx=0x7fda00009b30).stop 2024-02-19T21:58:41.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:41.006+0000 7fda0acdb700 1 -- 172.21.15.86:0/2031210182 shutdown_connections 2024-02-19T21:58:41.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:41.006+0000 7fda0acdb700 1 --2- 172.21.15.86:0/2031210182 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fda040f51a0 0x7fda040f55c0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:58:41.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:41.006+0000 7fda0acdb700 1 -- 172.21.15.86:0/2031210182 >> 172.21.15.86:0/2031210182 conn(0x7fda040f09a0 msgr2=0x7fda040f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:58:41.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:41.006+0000 7fda0acdb700 1 -- 172.21.15.86:0/2031210182 shutdown_connections 2024-02-19T21:58:41.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:41.006+0000 7fda0acdb700 1 -- 172.21.15.86:0/2031210182 wait complete. 2024-02-19T21:58:41.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:41.006+0000 7fda0acdb700 1 Processor -- start 2024-02-19T21:58:41.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:41.007+0000 7fda0acdb700 1 -- start start 2024-02-19T21:58:41.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:41.007+0000 7fda0acdb700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fda040f51a0 0x7fda041065d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:58:41.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:41.007+0000 7fda0acdb700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda04106ac0 con 0x7fda040f51a0 2024-02-19T21:58:41.009 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:41.007+0000 7fda09cd9700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fda040f51a0 0x7fda041065d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:58:41.009 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:41.007+0000 7fda09cd9700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fda040f51a0 0x7fda041065d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:41768/0 (socket says 172.21.15.86:41768) 2024-02-19T21:58:41.009 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:41.007+0000 7fda09cd9700 1 -- 172.21.15.86:0/1464288040 learned_addr learned my addr 172.21.15.86:0/1464288040 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T21:58:41.009 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:41.007+0000 7fda09cd9700 1 -- 172.21.15.86:0/1464288040 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda00005040 con 0x7fda040f51a0 2024-02-19T21:58:41.009 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:41.007+0000 7fda09cd9700 1 --2- 172.21.15.86:0/1464288040 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fda040f51a0 0x7fda041065d0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7fda000089f0 tx=0x7fda0000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:58:41.009 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:41.007+0000 7fd9fa7fc700 1 -- 172.21.15.86:0/1464288040 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fda00009320 con 0x7fda040f51a0 2024-02-19T21:58:41.009 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:41.008+0000 7fda0acdb700 1 -- 172.21.15.86:0/1464288040 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda04106cc0 con 0x7fda040f51a0 2024-02-19T21:58:41.009 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:41.008+0000 7fd9fa7fc700 1 -- 172.21.15.86:0/1464288040 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fda00013380 con 0x7fda040f51a0 2024-02-19T21:58:41.009 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:41.008+0000 7fd9fa7fc700 1 -- 172.21.15.86:0/1464288040 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fda00014e20 con 0x7fda040f51a0 2024-02-19T21:58:41.009 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:41.008+0000 7fda0acdb700 1 -- 172.21.15.86:0/1464288040 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda04107150 con 0x7fda040f51a0 2024-02-19T21:58:41.010 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:41.008+0000 7fd9fa7fc700 1 -- 172.21.15.86:0/1464288040 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fda000095f0 con 0x7fda040f51a0 2024-02-19T21:58:41.010 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:41.008+0000 7fda0acdb700 1 -- 172.21.15.86:0/1464288040 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda04045ad0 con 0x7fda040f51a0 2024-02-19T21:58:41.010 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:41.009+0000 7fd9fa7fc700 1 --2- 172.21.15.86:0/1464288040 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fd9f002ff10 0x7fd9f00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:58:41.010 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:41.009+0000 7fd9fa7fc700 1 -- 172.21.15.86:0/1464288040 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fda00040f80 con 0x7fda040f51a0 2024-02-19T21:58:41.011 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:41.009+0000 7fda094d8700 1 --2- 172.21.15.86:0/1464288040 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fd9f002ff10 0x7fd9f00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:58:41.011 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:41.009+0000 7fda094d8700 1 --2- 172.21.15.86:0/1464288040 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fd9f002ff10 0x7fd9f00323d0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7fd9f40096a0 tx=0x7fd9f4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:58:41.013 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:41.013+0000 7fd9fa7fc700 1 -- 172.21.15.86:0/1464288040 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fda00014420 con 0x7fda040f51a0 2024-02-19T21:58:41.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:41.220+0000 7fda0acdb700 1 -- 172.21.15.86:0/1464288040 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fda04108770 con 0x7fda040f51a0 2024-02-19T21:58:41.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:41.221+0000 7fd9fa7fc700 1 -- 172.21.15.86:0/1464288040 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fda00010230 con 0x7fda040f51a0 2024-02-19T21:58:41.221 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T21:58:41.221 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T21:58:41.223 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:41.223+0000 7fda0acdb700 1 -- 172.21.15.86:0/1464288040 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fd9f002ff10 msgr2=0x7fd9f00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:58:41.223 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:41.223+0000 7fda0acdb700 1 --2- 172.21.15.86:0/1464288040 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fd9f002ff10 0x7fd9f00323d0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7fd9f40096a0 tx=0x7fd9f4006b40).stop 2024-02-19T21:58:41.223 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:41.223+0000 7fda0acdb700 1 -- 172.21.15.86:0/1464288040 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fda040f51a0 msgr2=0x7fda041065d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:58:41.224 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:41.223+0000 7fda0acdb700 1 --2- 172.21.15.86:0/1464288040 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fda040f51a0 0x7fda041065d0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7fda000089f0 tx=0x7fda0000b040).stop 2024-02-19T21:58:41.224 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:41.223+0000 7fda0acdb700 1 -- 172.21.15.86:0/1464288040 shutdown_connections 2024-02-19T21:58:41.224 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:41.223+0000 7fda0acdb700 1 --2- 172.21.15.86:0/1464288040 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fd9f002ff10 0x7fd9f00323d0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:58:41.224 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:41.223+0000 7fda0acdb700 1 --2- 172.21.15.86:0/1464288040 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fda040f51a0 0x7fda041065d0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:58:41.224 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:41.223+0000 7fda0acdb700 1 -- 172.21.15.86:0/1464288040 >> 172.21.15.86:0/1464288040 conn(0x7fda040f09a0 msgr2=0x7fda0405fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:58:41.224 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:41.223+0000 7fda0acdb700 1 -- 172.21.15.86:0/1464288040 shutdown_connections 2024-02-19T21:58:41.224 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:41.223+0000 7fda0acdb700 1 -- 172.21.15.86:0/1464288040 wait complete. 2024-02-19T21:58:41.224 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T21:58:41.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:58:41 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:58:41.222567+0000 mon.smithi053 (mon.0) 144 : audit [DBG] from='client.? 172.21.15.86:0/1464288040' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T21:58:42.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:58:42 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T21:58:41.454685+0000 mgr.smithi053.fyhcdi (mgr.14162) 64 : cluster [DBG] pgmap v57: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:58:42.808 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T21:58:42.808 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T21:58:44.590 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:44.588+0000 7f1be4359700 1 -- 172.21.15.86:0/1086544083 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1bdc0f5410 msgr2=0x7f1bdc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:58:44.590 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:44.588+0000 7f1be4359700 1 --2- 172.21.15.86:0/1086544083 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1bdc0f5410 0x7f1bdc0f5830 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7f1bcc009fb0 tx=0x7f1bcc009b30).stop 2024-02-19T21:58:44.590 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:44.588+0000 7f1be4359700 1 -- 172.21.15.86:0/1086544083 shutdown_connections 2024-02-19T21:58:44.590 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:44.588+0000 7f1be4359700 1 --2- 172.21.15.86:0/1086544083 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1bdc0f5410 0x7f1bdc0f5830 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:58:44.590 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:44.588+0000 7f1be4359700 1 -- 172.21.15.86:0/1086544083 >> 172.21.15.86:0/1086544083 conn(0x7f1bdc0f0b70 msgr2=0x7f1bdc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:58:44.590 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:44.589+0000 7f1be4359700 1 -- 172.21.15.86:0/1086544083 shutdown_connections 2024-02-19T21:58:44.590 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:44.589+0000 7f1be4359700 1 -- 172.21.15.86:0/1086544083 wait complete. 2024-02-19T21:58:44.590 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:44.589+0000 7f1be4359700 1 Processor -- start 2024-02-19T21:58:44.590 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:44.589+0000 7f1be4359700 1 -- start start 2024-02-19T21:58:44.591 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:44.589+0000 7f1be4359700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1bdc0f5410 0x7f1bdc1067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:58:44.591 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:44.589+0000 7f1be4359700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bdc106cd0 con 0x7f1bdc0f5410 2024-02-19T21:58:44.591 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:44.589+0000 7f1be20f5700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1bdc0f5410 0x7f1bdc1067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:58:44.591 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:44.589+0000 7f1be20f5700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1bdc0f5410 0x7f1bdc1067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:41780/0 (socket says 172.21.15.86:41780) 2024-02-19T21:58:44.591 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:44.589+0000 7f1be20f5700 1 -- 172.21.15.86:0/775057403 learned_addr learned my addr 172.21.15.86:0/775057403 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T21:58:44.591 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:44.590+0000 7f1be20f5700 1 -- 172.21.15.86:0/775057403 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bcc005040 con 0x7f1bdc0f5410 2024-02-19T21:58:44.591 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:44.590+0000 7f1be20f5700 1 --2- 172.21.15.86:0/775057403 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1bdc0f5410 0x7f1bdc1067e0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f1bcc005010 tx=0x7f1bcc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:58:44.591 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:44.590+0000 7f1bd2ffd700 1 -- 172.21.15.86:0/775057403 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1bcc009320 con 0x7f1bdc0f5410 2024-02-19T21:58:44.591 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:44.590+0000 7f1bd2ffd700 1 -- 172.21.15.86:0/775057403 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1bcc013380 con 0x7f1bdc0f5410 2024-02-19T21:58:44.591 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:44.590+0000 7f1be4359700 1 -- 172.21.15.86:0/775057403 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bdc106ed0 con 0x7f1bdc0f5410 2024-02-19T21:58:44.591 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:44.590+0000 7f1bd2ffd700 1 -- 172.21.15.86:0/775057403 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1bcc014e50 con 0x7f1bdc0f5410 2024-02-19T21:58:44.591 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:44.590+0000 7f1be4359700 1 -- 172.21.15.86:0/775057403 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bdc107360 con 0x7f1bdc0f5410 2024-02-19T21:58:44.592 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:44.591+0000 7f1bd2ffd700 1 -- 172.21.15.86:0/775057403 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1bcc0095f0 con 0x7f1bdc0f5410 2024-02-19T21:58:44.592 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:44.591+0000 7f1be4359700 1 -- 172.21.15.86:0/775057403 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1bdc045ad0 con 0x7f1bdc0f5410 2024-02-19T21:58:44.592 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:44.591+0000 7f1bd2ffd700 1 --2- 172.21.15.86:0/775057403 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f1bc802ff10 0x7f1bc80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:58:44.592 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:44.591+0000 7f1bd2ffd700 1 -- 172.21.15.86:0/775057403 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1bcc040e90 con 0x7f1bdc0f5410 2024-02-19T21:58:44.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:44.591+0000 7f1be18f4700 1 --2- 172.21.15.86:0/775057403 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f1bc802ff10 0x7f1bc80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:58:44.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:44.591+0000 7f1be18f4700 1 --2- 172.21.15.86:0/775057403 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f1bc802ff10 0x7f1bc80323d0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f1bd80096a0 tx=0x7f1bd8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:58:44.595 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:44.596+0000 7f1bd2ffd700 1 -- 172.21.15.86:0/775057403 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1bcc014420 con 0x7f1bdc0f5410 2024-02-19T21:58:44.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:58:44 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T21:58:43.455067+0000 mgr.smithi053.fyhcdi (mgr.14162) 65 : cluster [DBG] pgmap v58: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:58:44.800 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:44.800+0000 7f1be4359700 1 -- 172.21.15.86:0/775057403 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1bdc0fafa0 con 0x7f1bdc0f5410 2024-02-19T21:58:44.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:44.801+0000 7f1bd2ffd700 1 -- 172.21.15.86:0/775057403 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1bcc010230 con 0x7f1bdc0f5410 2024-02-19T21:58:44.801 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T21:58:44.801 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T21:58:44.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:44.802+0000 7f1be4359700 1 -- 172.21.15.86:0/775057403 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f1bc802ff10 msgr2=0x7f1bc80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:58:44.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:44.802+0000 7f1be4359700 1 --2- 172.21.15.86:0/775057403 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f1bc802ff10 0x7f1bc80323d0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f1bd80096a0 tx=0x7f1bd8006b40).stop 2024-02-19T21:58:44.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:44.802+0000 7f1be4359700 1 -- 172.21.15.86:0/775057403 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1bdc0f5410 msgr2=0x7f1bdc1067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:58:44.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:44.802+0000 7f1be4359700 1 --2- 172.21.15.86:0/775057403 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1bdc0f5410 0x7f1bdc1067e0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f1bcc005010 tx=0x7f1bcc00b040).stop 2024-02-19T21:58:44.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:44.802+0000 7f1be4359700 1 -- 172.21.15.86:0/775057403 shutdown_connections 2024-02-19T21:58:44.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:44.802+0000 7f1be4359700 1 --2- 172.21.15.86:0/775057403 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f1bc802ff10 0x7f1bc80323d0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:58:44.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:44.802+0000 7f1be4359700 1 --2- 172.21.15.86:0/775057403 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1bdc0f5410 0x7f1bdc1067e0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:58:44.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:44.802+0000 7f1be4359700 1 -- 172.21.15.86:0/775057403 >> 172.21.15.86:0/775057403 conn(0x7f1bdc0f0b70 msgr2=0x7f1bdc05fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:58:44.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:44.803+0000 7f1be4359700 1 -- 172.21.15.86:0/775057403 shutdown_connections 2024-02-19T21:58:44.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:44.803+0000 7f1be4359700 1 -- 172.21.15.86:0/775057403 wait complete. 2024-02-19T21:58:44.803 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T21:58:45.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:58:45 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:58:44.802193+0000 mon.smithi053 (mon.0) 145 : audit [DBG] from='client.? 172.21.15.86:0/775057403' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T21:58:46.429 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T21:58:46.430 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T21:58:46.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:58:46 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T21:58:45.455532+0000 mgr.smithi053.fyhcdi (mgr.14162) 66 : cluster [DBG] pgmap v59: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:58:48.182 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:48.181+0000 7fc413fff700 1 -- 172.21.15.86:0/2555708032 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc414060140 msgr2=0x7fc4140f8fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:58:48.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:48.181+0000 7fc413fff700 1 --2- 172.21.15.86:0/2555708032 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc414060140 0x7fc4140f8fc0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7fc404009fb0 tx=0x7fc404009b30).stop 2024-02-19T21:58:48.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:48.181+0000 7fc413fff700 1 -- 172.21.15.86:0/2555708032 shutdown_connections 2024-02-19T21:58:48.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:48.181+0000 7fc413fff700 1 --2- 172.21.15.86:0/2555708032 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc414060140 0x7fc4140f8fc0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:58:48.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:48.181+0000 7fc413fff700 1 -- 172.21.15.86:0/2555708032 >> 172.21.15.86:0/2555708032 conn(0x7fc4140f09a0 msgr2=0x7fc4140f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:58:48.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:48.181+0000 7fc413fff700 1 -- 172.21.15.86:0/2555708032 shutdown_connections 2024-02-19T21:58:48.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:48.181+0000 7fc413fff700 1 -- 172.21.15.86:0/2555708032 wait complete. 2024-02-19T21:58:48.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:48.181+0000 7fc413fff700 1 Processor -- start 2024-02-19T21:58:48.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:48.181+0000 7fc413fff700 1 -- start start 2024-02-19T21:58:48.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:48.182+0000 7fc413fff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc414060140 0x7fc4140f6a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:58:48.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:48.182+0000 7fc413fff700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4140f4ec0 con 0x7fc414060140 2024-02-19T21:58:48.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:48.182+0000 7fc412ffd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc414060140 0x7fc4140f6a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:58:48.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:48.182+0000 7fc412ffd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc414060140 0x7fc4140f6a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:41784/0 (socket says 172.21.15.86:41784) 2024-02-19T21:58:48.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:48.182+0000 7fc412ffd700 1 -- 172.21.15.86:0/3627053336 learned_addr learned my addr 172.21.15.86:0/3627053336 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T21:58:48.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:48.182+0000 7fc412ffd700 1 -- 172.21.15.86:0/3627053336 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc404005040 con 0x7fc414060140 2024-02-19T21:58:48.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:48.182+0000 7fc412ffd700 1 --2- 172.21.15.86:0/3627053336 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc414060140 0x7fc4140f6a40 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7fc404005010 tx=0x7fc40400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:58:48.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:48.182+0000 7fc3fb7fe700 1 -- 172.21.15.86:0/3627053336 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc404009320 con 0x7fc414060140 2024-02-19T21:58:48.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:48.182+0000 7fc3fb7fe700 1 -- 172.21.15.86:0/3627053336 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc404013380 con 0x7fc414060140 2024-02-19T21:58:48.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:48.182+0000 7fc413fff700 1 -- 172.21.15.86:0/3627053336 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4140f5060 con 0x7fc414060140 2024-02-19T21:58:48.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:48.182+0000 7fc3fb7fe700 1 -- 172.21.15.86:0/3627053336 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc404014e50 con 0x7fc414060140 2024-02-19T21:58:48.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:48.182+0000 7fc413fff700 1 -- 172.21.15.86:0/3627053336 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4140f54f0 con 0x7fc414060140 2024-02-19T21:58:48.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:48.183+0000 7fc3fb7fe700 1 -- 172.21.15.86:0/3627053336 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc4040095f0 con 0x7fc414060140 2024-02-19T21:58:48.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:48.183+0000 7fc413fff700 1 -- 172.21.15.86:0/3627053336 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc414059100 con 0x7fc414060140 2024-02-19T21:58:48.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:48.183+0000 7fc3fb7fe700 1 --2- 172.21.15.86:0/3627053336 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fc3fc02ff10 0x7fc3fc0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:58:48.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:48.183+0000 7fc3fb7fe700 1 -- 172.21.15.86:0/3627053336 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc404040fa0 con 0x7fc414060140 2024-02-19T21:58:48.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:48.184+0000 7fc4127fc700 1 --2- 172.21.15.86:0/3627053336 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fc3fc02ff10 0x7fc3fc0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:58:48.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:48.184+0000 7fc4127fc700 1 --2- 172.21.15.86:0/3627053336 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fc3fc02ff10 0x7fc3fc0323d0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7fc4080096a0 tx=0x7fc408006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:58:48.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:48.188+0000 7fc3fb7fe700 1 -- 172.21.15.86:0/3627053336 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc404014420 con 0x7fc414060140 2024-02-19T21:58:48.394 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:48.394+0000 7fc413fff700 1 -- 172.21.15.86:0/3627053336 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc4140fbe20 con 0x7fc414060140 2024-02-19T21:58:48.394 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:48.395+0000 7fc3fb7fe700 1 -- 172.21.15.86:0/3627053336 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc404010230 con 0x7fc414060140 2024-02-19T21:58:48.395 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T21:58:48.395 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T21:58:48.396 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:48.396+0000 7fc413fff700 1 -- 172.21.15.86:0/3627053336 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fc3fc02ff10 msgr2=0x7fc3fc0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:58:48.396 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:48.396+0000 7fc413fff700 1 --2- 172.21.15.86:0/3627053336 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fc3fc02ff10 0x7fc3fc0323d0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7fc4080096a0 tx=0x7fc408006b40).stop 2024-02-19T21:58:48.396 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:48.396+0000 7fc413fff700 1 -- 172.21.15.86:0/3627053336 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc414060140 msgr2=0x7fc4140f6a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:58:48.396 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:48.396+0000 7fc413fff700 1 --2- 172.21.15.86:0/3627053336 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc414060140 0x7fc4140f6a40 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7fc404005010 tx=0x7fc40400b040).stop 2024-02-19T21:58:48.396 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:48.396+0000 7fc413fff700 1 -- 172.21.15.86:0/3627053336 shutdown_connections 2024-02-19T21:58:48.397 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:48.396+0000 7fc413fff700 1 --2- 172.21.15.86:0/3627053336 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fc3fc02ff10 0x7fc3fc0323d0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:58:48.397 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:48.396+0000 7fc413fff700 1 --2- 172.21.15.86:0/3627053336 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc414060140 0x7fc4140f6a40 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:58:48.397 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:48.396+0000 7fc413fff700 1 -- 172.21.15.86:0/3627053336 >> 172.21.15.86:0/3627053336 conn(0x7fc4140f09a0 msgr2=0x7fc4140f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:58:48.397 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:48.396+0000 7fc413fff700 1 -- 172.21.15.86:0/3627053336 shutdown_connections 2024-02-19T21:58:48.397 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:48.396+0000 7fc413fff700 1 -- 172.21.15.86:0/3627053336 wait complete. 2024-02-19T21:58:48.397 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T21:58:48.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:58:48 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T21:58:47.455880+0000 mgr.smithi053.fyhcdi (mgr.14162) 67 : cluster [DBG] pgmap v60: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:58:48.794 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:58:48 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:58:48.396144+0000 mon.smithi053 (mon.0) 146 : audit [DBG] from='client.? 172.21.15.86:0/3627053336' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T21:58:50.066 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T21:58:50.067 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T21:58:50.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:58:50 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T21:58:49.456186+0000 mgr.smithi053.fyhcdi (mgr.14162) 68 : cluster [DBG] pgmap v61: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:58:51.813 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:51.812+0000 7ff5f31a6700 1 -- 172.21.15.86:0/3283666576 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff5ec0f6510 msgr2=0x7ff5ec0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:58:51.813 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:51.812+0000 7ff5f31a6700 1 --2- 172.21.15.86:0/3283666576 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff5ec0f6510 0x7ff5ec0f6930 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7ff5e0009fb0 tx=0x7ff5e0009b30).stop 2024-02-19T21:58:51.813 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:51.812+0000 7ff5f31a6700 1 -- 172.21.15.86:0/3283666576 shutdown_connections 2024-02-19T21:58:51.813 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:51.812+0000 7ff5f31a6700 1 --2- 172.21.15.86:0/3283666576 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff5ec0f6510 0x7ff5ec0f6930 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:58:51.813 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:51.812+0000 7ff5f31a6700 1 -- 172.21.15.86:0/3283666576 >> 172.21.15.86:0/3283666576 conn(0x7ff5ec0f1c70 msgr2=0x7ff5ec0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:58:51.813 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:51.812+0000 7ff5f31a6700 1 -- 172.21.15.86:0/3283666576 shutdown_connections 2024-02-19T21:58:51.813 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:51.812+0000 7ff5f31a6700 1 -- 172.21.15.86:0/3283666576 wait complete. 2024-02-19T21:58:51.813 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:51.812+0000 7ff5f31a6700 1 Processor -- start 2024-02-19T21:58:51.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:51.812+0000 7ff5f31a6700 1 -- start start 2024-02-19T21:58:51.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:51.813+0000 7ff5f31a6700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff5ec0f6510 0x7ff5ec06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:58:51.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:51.813+0000 7ff5f31a6700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5ec069e20 con 0x7ff5ec0f6510 2024-02-19T21:58:51.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:51.813+0000 7ff5f0f42700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff5ec0f6510 0x7ff5ec06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:58:51.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:51.813+0000 7ff5f0f42700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff5ec0f6510 0x7ff5ec06b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:42654/0 (socket says 172.21.15.86:42654) 2024-02-19T21:58:51.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:51.813+0000 7ff5f0f42700 1 -- 172.21.15.86:0/2350848173 learned_addr learned my addr 172.21.15.86:0/2350848173 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T21:58:51.815 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:51.813+0000 7ff5f0f42700 1 -- 172.21.15.86:0/2350848173 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5e0005040 con 0x7ff5ec0f6510 2024-02-19T21:58:51.816 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:51.814+0000 7ff5f0f42700 1 --2- 172.21.15.86:0/2350848173 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff5ec0f6510 0x7ff5ec06b9a0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7ff5e00089f0 tx=0x7ff5e000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:58:51.816 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:51.814+0000 7ff5e97fa700 1 -- 172.21.15.86:0/2350848173 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff5e0009320 con 0x7ff5ec0f6510 2024-02-19T21:58:51.816 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:51.814+0000 7ff5e97fa700 1 -- 172.21.15.86:0/2350848173 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff5e0013380 con 0x7ff5ec0f6510 2024-02-19T21:58:51.816 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:51.814+0000 7ff5f31a6700 1 -- 172.21.15.86:0/2350848173 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5ec069fc0 con 0x7ff5ec0f6510 2024-02-19T21:58:51.816 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:51.814+0000 7ff5e97fa700 1 -- 172.21.15.86:0/2350848173 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff5e001ed80 con 0x7ff5ec0f6510 2024-02-19T21:58:51.816 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:51.814+0000 7ff5f31a6700 1 -- 172.21.15.86:0/2350848173 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5ec06a450 con 0x7ff5ec0f6510 2024-02-19T21:58:51.816 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:51.815+0000 7ff5e97fa700 1 -- 172.21.15.86:0/2350848173 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff5e001e420 con 0x7ff5ec0f6510 2024-02-19T21:58:51.816 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:51.815+0000 7ff5f31a6700 1 -- 172.21.15.86:0/2350848173 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5ec059100 con 0x7ff5ec0f6510 2024-02-19T21:58:51.816 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:51.815+0000 7ff5e97fa700 1 --2- 172.21.15.86:0/2350848173 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7ff5dc02ff60 0x7ff5dc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:58:51.816 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:51.815+0000 7ff5e97fa700 1 -- 172.21.15.86:0/2350848173 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff5e0041090 con 0x7ff5ec0f6510 2024-02-19T21:58:51.816 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:51.815+0000 7ff5ebfff700 1 --2- 172.21.15.86:0/2350848173 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7ff5dc02ff60 0x7ff5dc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:58:51.816 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:51.815+0000 7ff5ebfff700 1 --2- 172.21.15.86:0/2350848173 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7ff5dc02ff60 0x7ff5dc032420 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7ff5d80096a0 tx=0x7ff5d8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:58:51.819 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:51.820+0000 7ff5e97fa700 1 -- 172.21.15.86:0/2350848173 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff5e00095f0 con 0x7ff5ec0f6510 2024-02-19T21:58:52.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:52.024+0000 7ff5f31a6700 1 -- 172.21.15.86:0/2350848173 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff5ec0fc0a0 con 0x7ff5ec0f6510 2024-02-19T21:58:52.024 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:52.024+0000 7ff5e97fa700 1 -- 172.21.15.86:0/2350848173 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff5e0004490 con 0x7ff5ec0f6510 2024-02-19T21:58:52.024 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T21:58:52.025 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T21:58:52.026 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:52.026+0000 7ff5f31a6700 1 -- 172.21.15.86:0/2350848173 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7ff5dc02ff60 msgr2=0x7ff5dc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:58:52.026 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:52.026+0000 7ff5f31a6700 1 --2- 172.21.15.86:0/2350848173 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7ff5dc02ff60 0x7ff5dc032420 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7ff5d80096a0 tx=0x7ff5d8006b40).stop 2024-02-19T21:58:52.026 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:52.026+0000 7ff5f31a6700 1 -- 172.21.15.86:0/2350848173 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff5ec0f6510 msgr2=0x7ff5ec06b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:58:52.026 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:52.026+0000 7ff5f31a6700 1 --2- 172.21.15.86:0/2350848173 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff5ec0f6510 0x7ff5ec06b9a0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7ff5e00089f0 tx=0x7ff5e000b040).stop 2024-02-19T21:58:52.026 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:52.026+0000 7ff5f31a6700 1 -- 172.21.15.86:0/2350848173 shutdown_connections 2024-02-19T21:58:52.026 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:52.026+0000 7ff5f31a6700 1 --2- 172.21.15.86:0/2350848173 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7ff5dc02ff60 0x7ff5dc032420 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:58:52.026 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:52.026+0000 7ff5f31a6700 1 --2- 172.21.15.86:0/2350848173 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff5ec0f6510 0x7ff5ec06b9a0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:58:52.026 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:52.026+0000 7ff5f31a6700 1 -- 172.21.15.86:0/2350848173 >> 172.21.15.86:0/2350848173 conn(0x7ff5ec0f1c70 msgr2=0x7ff5ec0f26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:58:52.026 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:52.026+0000 7ff5f31a6700 1 -- 172.21.15.86:0/2350848173 shutdown_connections 2024-02-19T21:58:52.026 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:52.026+0000 7ff5f31a6700 1 -- 172.21.15.86:0/2350848173 wait complete. 2024-02-19T21:58:52.027 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T21:58:52.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:58:52 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T21:58:51.456479+0000 mgr.smithi053.fyhcdi (mgr.14162) 69 : cluster [DBG] pgmap v62: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:58:52.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:58:52 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:58:52.025485+0000 mon.smithi053 (mon.0) 147 : audit [DBG] from='client.? 172.21.15.86:0/2350848173' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T21:58:53.610 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T21:58:53.611 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T21:58:54.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:58:54 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T21:58:53.456715+0000 mgr.smithi053.fyhcdi (mgr.14162) 70 : cluster [DBG] pgmap v63: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:58:55.488 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:55.487+0000 7f82f2682700 1 -- 172.21.15.86:0/3910151879 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f82ec0601e0 msgr2=0x7f82ec0f9190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:58:55.488 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:55.487+0000 7f82f2682700 1 --2- 172.21.15.86:0/3910151879 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f82ec0601e0 0x7f82ec0f9190 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7f82dc009fb0 tx=0x7f82dc009b30).stop 2024-02-19T21:58:55.488 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:55.487+0000 7f82f2682700 1 -- 172.21.15.86:0/3910151879 shutdown_connections 2024-02-19T21:58:55.488 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:55.487+0000 7f82f2682700 1 --2- 172.21.15.86:0/3910151879 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f82ec0601e0 0x7f82ec0f9190 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:58:55.488 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:55.487+0000 7f82f2682700 1 -- 172.21.15.86:0/3910151879 >> 172.21.15.86:0/3910151879 conn(0x7f82ec0f0b70 msgr2=0x7f82ec0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:58:55.488 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:55.487+0000 7f82f2682700 1 -- 172.21.15.86:0/3910151879 shutdown_connections 2024-02-19T21:58:55.488 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:55.487+0000 7f82f2682700 1 -- 172.21.15.86:0/3910151879 wait complete. 2024-02-19T21:58:55.488 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:55.488+0000 7f82f2682700 1 Processor -- start 2024-02-19T21:58:55.488 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:55.488+0000 7f82f2682700 1 -- start start 2024-02-19T21:58:55.489 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:55.488+0000 7f82f2682700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f82ec0601e0 0x7f82ec0f6c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:58:55.489 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:55.488+0000 7f82f2682700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82ec0f5090 con 0x7f82ec0601e0 2024-02-19T21:58:55.490 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:55.488+0000 7f82ebfff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f82ec0601e0 0x7f82ec0f6c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:58:55.490 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:55.488+0000 7f82ebfff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f82ec0601e0 0x7f82ec0f6c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:42668/0 (socket says 172.21.15.86:42668) 2024-02-19T21:58:55.490 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:55.488+0000 7f82ebfff700 1 -- 172.21.15.86:0/2210254052 learned_addr learned my addr 172.21.15.86:0/2210254052 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T21:58:55.491 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:55.489+0000 7f82ebfff700 1 -- 172.21.15.86:0/2210254052 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82dc005040 con 0x7f82ec0601e0 2024-02-19T21:58:55.491 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:55.489+0000 7f82ebfff700 1 --2- 172.21.15.86:0/2210254052 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f82ec0601e0 0x7f82ec0f6c10 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f82dc007a10 tx=0x7f82dc009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:58:55.491 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:55.489+0000 7f82e8ff9700 1 -- 172.21.15.86:0/2210254052 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f82dc018070 con 0x7f82ec0601e0 2024-02-19T21:58:55.491 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:55.489+0000 7f82e8ff9700 1 -- 172.21.15.86:0/2210254052 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f82dc013410 con 0x7f82ec0601e0 2024-02-19T21:58:55.491 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:55.489+0000 7f82f2682700 1 -- 172.21.15.86:0/2210254052 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82ec0f5230 con 0x7f82ec0601e0 2024-02-19T21:58:55.491 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:55.489+0000 7f82e8ff9700 1 -- 172.21.15.86:0/2210254052 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f82dc01c3c0 con 0x7f82ec0601e0 2024-02-19T21:58:55.491 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:55.489+0000 7f82f2682700 1 -- 172.21.15.86:0/2210254052 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82ec0f56c0 con 0x7f82ec0601e0 2024-02-19T21:58:55.491 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:55.490+0000 7f82e8ff9700 1 -- 172.21.15.86:0/2210254052 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f82dc01c680 con 0x7f82ec0601e0 2024-02-19T21:58:55.491 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:55.490+0000 7f82e8ff9700 1 --2- 172.21.15.86:0/2210254052 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f82d402ff60 0x7f82d4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:58:55.491 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:55.490+0000 7f82f2682700 1 -- 172.21.15.86:0/2210254052 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82ec059100 con 0x7f82ec0601e0 2024-02-19T21:58:55.491 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:55.490+0000 7f82e8ff9700 1 -- 172.21.15.86:0/2210254052 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f82dc0414d0 con 0x7f82ec0601e0 2024-02-19T21:58:55.491 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:55.490+0000 7f82eb7fe700 1 --2- 172.21.15.86:0/2210254052 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f82d402ff60 0x7f82d4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:58:55.491 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:55.491+0000 7f82eb7fe700 1 --2- 172.21.15.86:0/2210254052 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f82d402ff60 0x7f82d4032420 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f82e00096a0 tx=0x7f82e0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:58:55.495 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:55.495+0000 7f82e8ff9700 1 -- 172.21.15.86:0/2210254052 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f82dc01ca80 con 0x7f82ec0601e0 2024-02-19T21:58:55.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:55.699+0000 7f82f2682700 1 -- 172.21.15.86:0/2210254052 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f82ec045ad0 con 0x7f82ec0601e0 2024-02-19T21:58:55.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:55.700+0000 7f82e8ff9700 1 -- 172.21.15.86:0/2210254052 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f82dc01bb60 con 0x7f82ec0601e0 2024-02-19T21:58:55.700 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T21:58:55.700 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T21:58:55.702 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:55.702+0000 7f82f2682700 1 -- 172.21.15.86:0/2210254052 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f82d402ff60 msgr2=0x7f82d4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:58:55.702 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:55.702+0000 7f82f2682700 1 --2- 172.21.15.86:0/2210254052 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f82d402ff60 0x7f82d4032420 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f82e00096a0 tx=0x7f82e0006b40).stop 2024-02-19T21:58:55.702 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:55.702+0000 7f82f2682700 1 -- 172.21.15.86:0/2210254052 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f82ec0601e0 msgr2=0x7f82ec0f6c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:58:55.702 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:55.702+0000 7f82f2682700 1 --2- 172.21.15.86:0/2210254052 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f82ec0601e0 0x7f82ec0f6c10 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f82dc007a10 tx=0x7f82dc009b30).stop 2024-02-19T21:58:55.702 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:55.702+0000 7f82f2682700 1 -- 172.21.15.86:0/2210254052 shutdown_connections 2024-02-19T21:58:55.702 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:55.702+0000 7f82f2682700 1 --2- 172.21.15.86:0/2210254052 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f82d402ff60 0x7f82d4032420 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:58:55.702 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:55.702+0000 7f82f2682700 1 --2- 172.21.15.86:0/2210254052 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f82ec0601e0 0x7f82ec0f6c10 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:58:55.702 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:55.702+0000 7f82f2682700 1 -- 172.21.15.86:0/2210254052 >> 172.21.15.86:0/2210254052 conn(0x7f82ec0f0b70 msgr2=0x7f82ec0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:58:55.702 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:55.702+0000 7f82f2682700 1 -- 172.21.15.86:0/2210254052 shutdown_connections 2024-02-19T21:58:55.702 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:55.702+0000 7f82f2682700 1 -- 172.21.15.86:0/2210254052 wait complete. 2024-02-19T21:58:55.703 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T21:58:56.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:58:56 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T21:58:55.456978+0000 mgr.smithi053.fyhcdi (mgr.14162) 71 : cluster [DBG] pgmap v64: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:58:56.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:58:56 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:58:55.701248+0000 mon.smithi053 (mon.0) 148 : audit [DBG] from='client.? 172.21.15.86:0/2210254052' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T21:58:57.401 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T21:58:57.401 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T21:58:58.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:58:58 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T21:58:57.457322+0000 mgr.smithi053.fyhcdi (mgr.14162) 72 : cluster [DBG] pgmap v65: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:58:59.298 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:59.297+0000 7fa432e8f700 1 -- 172.21.15.86:0/1123162541 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa42c0f51a0 msgr2=0x7fa42c0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:58:59.298 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:59.297+0000 7fa432e8f700 1 --2- 172.21.15.86:0/1123162541 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa42c0f51a0 0x7fa42c0f55c0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7fa41c009fb0 tx=0x7fa41c009b30).stop 2024-02-19T21:58:59.298 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:59.297+0000 7fa432e8f700 1 -- 172.21.15.86:0/1123162541 shutdown_connections 2024-02-19T21:58:59.298 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:59.297+0000 7fa432e8f700 1 --2- 172.21.15.86:0/1123162541 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa42c0f51a0 0x7fa42c0f55c0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:58:59.298 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:59.297+0000 7fa432e8f700 1 -- 172.21.15.86:0/1123162541 >> 172.21.15.86:0/1123162541 conn(0x7fa42c0f09a0 msgr2=0x7fa42c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:58:59.298 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:59.297+0000 7fa432e8f700 1 -- 172.21.15.86:0/1123162541 shutdown_connections 2024-02-19T21:58:59.298 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:59.297+0000 7fa432e8f700 1 -- 172.21.15.86:0/1123162541 wait complete. 2024-02-19T21:58:59.298 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:59.297+0000 7fa432e8f700 1 Processor -- start 2024-02-19T21:58:59.299 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:59.298+0000 7fa432e8f700 1 -- start start 2024-02-19T21:58:59.299 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:59.298+0000 7fa432e8f700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa42c0f51a0 0x7fa42c109290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:58:59.299 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:59.298+0000 7fa432e8f700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa42c060280 con 0x7fa42c0f51a0 2024-02-19T21:58:59.299 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:59.298+0000 7fa431e8d700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa42c0f51a0 0x7fa42c109290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:58:59.299 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:59.298+0000 7fa431e8d700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa42c0f51a0 0x7fa42c109290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:42686/0 (socket says 172.21.15.86:42686) 2024-02-19T21:58:59.299 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:59.298+0000 7fa431e8d700 1 -- 172.21.15.86:0/2391532270 learned_addr learned my addr 172.21.15.86:0/2391532270 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T21:58:59.299 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:59.298+0000 7fa431e8d700 1 -- 172.21.15.86:0/2391532270 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa41c005040 con 0x7fa42c0f51a0 2024-02-19T21:58:59.299 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:59.298+0000 7fa431e8d700 1 --2- 172.21.15.86:0/2391532270 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa42c0f51a0 0x7fa42c109290 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7fa41c0089f0 tx=0x7fa41c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:58:59.300 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:59.299+0000 7fa4227fc700 1 -- 172.21.15.86:0/2391532270 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa41c009320 con 0x7fa42c0f51a0 2024-02-19T21:58:59.300 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:59.299+0000 7fa432e8f700 1 -- 172.21.15.86:0/2391532270 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa42c060480 con 0x7fa42c0f51a0 2024-02-19T21:58:59.300 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:59.299+0000 7fa4227fc700 1 -- 172.21.15.86:0/2391532270 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa41c013380 con 0x7fa42c0f51a0 2024-02-19T21:58:59.301 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:59.299+0000 7fa4227fc700 1 -- 172.21.15.86:0/2391532270 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa41c014e20 con 0x7fa42c0f51a0 2024-02-19T21:58:59.301 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:59.299+0000 7fa432e8f700 1 -- 172.21.15.86:0/2391532270 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa42c1099c0 con 0x7fa42c0f51a0 2024-02-19T21:58:59.301 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:59.300+0000 7fa432e8f700 1 -- 172.21.15.86:0/2391532270 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa42c045ad0 con 0x7fa42c0f51a0 2024-02-19T21:58:59.301 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:59.300+0000 7fa4227fc700 1 -- 172.21.15.86:0/2391532270 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa41c0095f0 con 0x7fa42c0f51a0 2024-02-19T21:58:59.301 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:59.300+0000 7fa4227fc700 1 --2- 172.21.15.86:0/2391532270 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fa41802ff10 0x7fa4180323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:58:59.301 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:59.300+0000 7fa4227fc700 1 -- 172.21.15.86:0/2391532270 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa41c040ec0 con 0x7fa42c0f51a0 2024-02-19T21:58:59.301 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:59.300+0000 7fa43168c700 1 --2- 172.21.15.86:0/2391532270 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fa41802ff10 0x7fa4180323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:58:59.301 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:59.300+0000 7fa43168c700 1 --2- 172.21.15.86:0/2391532270 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fa41802ff10 0x7fa4180323d0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7fa4280096a0 tx=0x7fa428006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:58:59.304 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:59.304+0000 7fa4227fc700 1 -- 172.21.15.86:0/2391532270 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa41c014420 con 0x7fa42c0f51a0 2024-02-19T21:58:59.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:59.510+0000 7fa432e8f700 1 -- 172.21.15.86:0/2391532270 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa42c0fad30 con 0x7fa42c0f51a0 2024-02-19T21:58:59.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:59.510+0000 7fa4227fc700 1 -- 172.21.15.86:0/2391532270 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa41c010230 con 0x7fa42c0f51a0 2024-02-19T21:58:59.510 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T21:58:59.510 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T21:58:59.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:59.512+0000 7fa432e8f700 1 -- 172.21.15.86:0/2391532270 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fa41802ff10 msgr2=0x7fa4180323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:58:59.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:59.512+0000 7fa432e8f700 1 --2- 172.21.15.86:0/2391532270 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fa41802ff10 0x7fa4180323d0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7fa4280096a0 tx=0x7fa428006b40).stop 2024-02-19T21:58:59.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:59.512+0000 7fa432e8f700 1 -- 172.21.15.86:0/2391532270 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa42c0f51a0 msgr2=0x7fa42c109290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:58:59.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:59.512+0000 7fa432e8f700 1 --2- 172.21.15.86:0/2391532270 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa42c0f51a0 0x7fa42c109290 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7fa41c0089f0 tx=0x7fa41c00b040).stop 2024-02-19T21:58:59.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:59.512+0000 7fa432e8f700 1 -- 172.21.15.86:0/2391532270 shutdown_connections 2024-02-19T21:58:59.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:59.512+0000 7fa432e8f700 1 --2- 172.21.15.86:0/2391532270 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fa41802ff10 0x7fa4180323d0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:58:59.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:59.512+0000 7fa432e8f700 1 --2- 172.21.15.86:0/2391532270 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa42c0f51a0 0x7fa42c109290 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:58:59.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:59.512+0000 7fa432e8f700 1 -- 172.21.15.86:0/2391532270 >> 172.21.15.86:0/2391532270 conn(0x7fa42c0f09a0 msgr2=0x7fa42c0f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:58:59.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:59.512+0000 7fa432e8f700 1 -- 172.21.15.86:0/2391532270 shutdown_connections 2024-02-19T21:58:59.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:58:59.512+0000 7fa432e8f700 1 -- 172.21.15.86:0/2391532270 wait complete. 2024-02-19T21:58:59.513 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T21:59:00.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:59:00 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T21:58:59.457583+0000 mgr.smithi053.fyhcdi (mgr.14162) 73 : cluster [DBG] pgmap v66: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:59:00.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:59:00 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:58:59.511792+0000 mon.smithi053 (mon.0) 149 : audit [DBG] from='client.? 172.21.15.86:0/2391532270' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T21:59:01.070 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T21:59:01.070 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T21:59:02.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:59:02 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T21:59:01.457901+0000 mgr.smithi053.fyhcdi (mgr.14162) 74 : cluster [DBG] pgmap v67: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:59:02.812 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:02.812+0000 7f5066d7e700 1 -- 172.21.15.86:0/3964629988 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f50600f5410 msgr2=0x7f50600f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:59:02.812 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:02.812+0000 7f5066d7e700 1 --2- 172.21.15.86:0/3964629988 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f50600f5410 0x7f50600f5830 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f5054009fb0 tx=0x7f5054009b30).stop 2024-02-19T21:59:02.812 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:02.812+0000 7f5066d7e700 1 -- 172.21.15.86:0/3964629988 shutdown_connections 2024-02-19T21:59:02.812 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:02.812+0000 7f5066d7e700 1 --2- 172.21.15.86:0/3964629988 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f50600f5410 0x7f50600f5830 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:59:02.813 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:02.812+0000 7f5066d7e700 1 -- 172.21.15.86:0/3964629988 >> 172.21.15.86:0/3964629988 conn(0x7f50600f0b70 msgr2=0x7f50600f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:59:02.813 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:02.812+0000 7f5066d7e700 1 -- 172.21.15.86:0/3964629988 shutdown_connections 2024-02-19T21:59:02.813 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:02.812+0000 7f5066d7e700 1 -- 172.21.15.86:0/3964629988 wait complete. 2024-02-19T21:59:02.813 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:02.813+0000 7f5066d7e700 1 Processor -- start 2024-02-19T21:59:02.813 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:02.813+0000 7f5066d7e700 1 -- start start 2024-02-19T21:59:02.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:02.813+0000 7f5066d7e700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f50600f5410 0x7f50601056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:59:02.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:02.813+0000 7f5066d7e700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5060105bd0 con 0x7f50600f5410 2024-02-19T21:59:02.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:02.813+0000 7f5064b1a700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f50600f5410 0x7f50601056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:59:02.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:02.813+0000 7f5064b1a700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f50600f5410 0x7f50601056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:41496/0 (socket says 172.21.15.86:41496) 2024-02-19T21:59:02.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:02.813+0000 7f5064b1a700 1 -- 172.21.15.86:0/285080355 learned_addr learned my addr 172.21.15.86:0/285080355 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T21:59:02.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:02.814+0000 7f5064b1a700 1 -- 172.21.15.86:0/285080355 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5054005040 con 0x7f50600f5410 2024-02-19T21:59:02.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:02.814+0000 7f5064b1a700 1 --2- 172.21.15.86:0/285080355 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f50600f5410 0x7f50601056e0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f5054005010 tx=0x7f505400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:59:02.815 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:02.814+0000 7f505d7fa700 1 -- 172.21.15.86:0/285080355 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5054009320 con 0x7f50600f5410 2024-02-19T21:59:02.815 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:02.814+0000 7f5066d7e700 1 -- 172.21.15.86:0/285080355 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5060105dd0 con 0x7f50600f5410 2024-02-19T21:59:02.815 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:02.814+0000 7f5066d7e700 1 -- 172.21.15.86:0/285080355 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5060106260 con 0x7f50600f5410 2024-02-19T21:59:02.815 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:02.814+0000 7f505d7fa700 1 -- 172.21.15.86:0/285080355 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5054013380 con 0x7f50600f5410 2024-02-19T21:59:02.815 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:02.814+0000 7f505d7fa700 1 -- 172.21.15.86:0/285080355 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5054014df0 con 0x7f50600f5410 2024-02-19T21:59:02.816 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:02.815+0000 7f505d7fa700 1 -- 172.21.15.86:0/285080355 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5054014420 con 0x7f50600f5410 2024-02-19T21:59:02.816 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:02.815+0000 7f5066d7e700 1 -- 172.21.15.86:0/285080355 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5040004fa0 con 0x7f50600f5410 2024-02-19T21:59:02.816 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:02.815+0000 7f505d7fa700 1 --2- 172.21.15.86:0/285080355 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f505002ff10 0x7f50500323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:59:02.816 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:02.815+0000 7f505d7fa700 1 -- 172.21.15.86:0/285080355 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5054041180 con 0x7f50600f5410 2024-02-19T21:59:02.816 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:02.815+0000 7f505ffff700 1 --2- 172.21.15.86:0/285080355 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f505002ff10 0x7f50500323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:59:02.816 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:02.816+0000 7f505ffff700 1 --2- 172.21.15.86:0/285080355 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f505002ff10 0x7f50500323d0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f504c0096a0 tx=0x7f504c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:59:02.819 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:02.820+0000 7f505d7fa700 1 -- 172.21.15.86:0/285080355 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5054012b20 con 0x7f50600f5410 2024-02-19T21:59:03.026 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:03.026+0000 7f5066d7e700 1 -- 172.21.15.86:0/285080355 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5040005670 con 0x7f50600f5410 2024-02-19T21:59:03.026 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:03.027+0000 7f505d7fa700 1 -- 172.21.15.86:0/285080355 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5054012b20 con 0x7f50600f5410 2024-02-19T21:59:03.027 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T21:59:03.027 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T21:59:03.028 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:03.028+0000 7f5066d7e700 1 -- 172.21.15.86:0/285080355 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f505002ff10 msgr2=0x7f50500323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:59:03.028 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:03.028+0000 7f5066d7e700 1 --2- 172.21.15.86:0/285080355 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f505002ff10 0x7f50500323d0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f504c0096a0 tx=0x7f504c006b40).stop 2024-02-19T21:59:03.028 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:03.028+0000 7f5066d7e700 1 -- 172.21.15.86:0/285080355 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f50600f5410 msgr2=0x7f50601056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:59:03.028 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:03.028+0000 7f5066d7e700 1 --2- 172.21.15.86:0/285080355 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f50600f5410 0x7f50601056e0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f5054005010 tx=0x7f505400b040).stop 2024-02-19T21:59:03.028 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:03.028+0000 7f5066d7e700 1 -- 172.21.15.86:0/285080355 shutdown_connections 2024-02-19T21:59:03.028 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:03.028+0000 7f5066d7e700 1 --2- 172.21.15.86:0/285080355 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f505002ff10 0x7f50500323d0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:59:03.028 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:03.028+0000 7f5066d7e700 1 --2- 172.21.15.86:0/285080355 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f50600f5410 0x7f50601056e0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:59:03.029 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:03.028+0000 7f5066d7e700 1 -- 172.21.15.86:0/285080355 >> 172.21.15.86:0/285080355 conn(0x7f50600f0b70 msgr2=0x7f50600f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:59:03.029 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:03.028+0000 7f5066d7e700 1 -- 172.21.15.86:0/285080355 shutdown_connections 2024-02-19T21:59:03.029 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:03.028+0000 7f5066d7e700 1 -- 172.21.15.86:0/285080355 wait complete. 2024-02-19T21:59:03.029 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T21:59:03.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:59:03 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:59:03.027913+0000 mon.smithi053 (mon.0) 150 : audit [DBG] from='client.? 172.21.15.86:0/285080355' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T21:59:04.643 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T21:59:04.643 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T21:59:04.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:59:04 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T21:59:03.458231+0000 mgr.smithi053.fyhcdi (mgr.14162) 75 : cluster [DBG] pgmap v68: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:59:06.413 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:06.410+0000 7f7a1fb13700 1 -- 172.21.15.86:0/4129550370 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7a180f5410 msgr2=0x7f7a180f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:59:06.413 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:06.410+0000 7f7a1fb13700 1 --2- 172.21.15.86:0/4129550370 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7a180f5410 0x7f7a180f5830 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7f7a08009fb0 tx=0x7f7a08009b30).stop 2024-02-19T21:59:06.413 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:06.410+0000 7f7a1fb13700 1 -- 172.21.15.86:0/4129550370 shutdown_connections 2024-02-19T21:59:06.413 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:06.410+0000 7f7a1fb13700 1 --2- 172.21.15.86:0/4129550370 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7a180f5410 0x7f7a180f5830 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:59:06.413 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:06.410+0000 7f7a1fb13700 1 -- 172.21.15.86:0/4129550370 >> 172.21.15.86:0/4129550370 conn(0x7f7a180f0b70 msgr2=0x7f7a180f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:59:06.414 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:06.410+0000 7f7a1fb13700 1 -- 172.21.15.86:0/4129550370 shutdown_connections 2024-02-19T21:59:06.414 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:06.410+0000 7f7a1fb13700 1 -- 172.21.15.86:0/4129550370 wait complete. 2024-02-19T21:59:06.414 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:06.410+0000 7f7a1fb13700 1 Processor -- start 2024-02-19T21:59:06.414 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:06.410+0000 7f7a1fb13700 1 -- start start 2024-02-19T21:59:06.414 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:06.410+0000 7f7a1fb13700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7a180f5410 0x7f7a181056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:59:06.414 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:06.410+0000 7f7a1fb13700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a18105bd0 con 0x7f7a180f5410 2024-02-19T21:59:06.414 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:06.410+0000 7f7a1d8af700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7a180f5410 0x7f7a181056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:59:06.414 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:06.411+0000 7f7a1d8af700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7a180f5410 0x7f7a181056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:41510/0 (socket says 172.21.15.86:41510) 2024-02-19T21:59:06.414 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:06.411+0000 7f7a1d8af700 1 -- 172.21.15.86:0/1765333334 learned_addr learned my addr 172.21.15.86:0/1765333334 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T21:59:06.414 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:06.411+0000 7f7a1d8af700 1 -- 172.21.15.86:0/1765333334 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a08005040 con 0x7f7a180f5410 2024-02-19T21:59:06.414 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:06.411+0000 7f7a1d8af700 1 --2- 172.21.15.86:0/1765333334 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7a180f5410 0x7f7a181056e0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f7a08005010 tx=0x7f7a0800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:59:06.414 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:06.411+0000 7f7a0e7fc700 1 -- 172.21.15.86:0/1765333334 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7a08009320 con 0x7f7a180f5410 2024-02-19T21:59:06.414 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:06.411+0000 7f7a1fb13700 1 -- 172.21.15.86:0/1765333334 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a18105dd0 con 0x7f7a180f5410 2024-02-19T21:59:06.414 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:06.411+0000 7f7a0e7fc700 1 -- 172.21.15.86:0/1765333334 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7a08013380 con 0x7f7a180f5410 2024-02-19T21:59:06.414 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:06.411+0000 7f7a0e7fc700 1 -- 172.21.15.86:0/1765333334 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7a08014e50 con 0x7f7a180f5410 2024-02-19T21:59:06.415 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:06.411+0000 7f7a1fb13700 1 -- 172.21.15.86:0/1765333334 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a18106260 con 0x7f7a180f5410 2024-02-19T21:59:06.415 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:06.412+0000 7f7a0e7fc700 1 -- 172.21.15.86:0/1765333334 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7a080095f0 con 0x7f7a180f5410 2024-02-19T21:59:06.415 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:06.412+0000 7f7a1fb13700 1 -- 172.21.15.86:0/1765333334 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a18045ad0 con 0x7f7a180f5410 2024-02-19T21:59:06.415 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:06.412+0000 7f7a0e7fc700 1 --2- 172.21.15.86:0/1765333334 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f7a0402ff10 0x7f7a040323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:59:06.415 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:06.412+0000 7f7a0e7fc700 1 -- 172.21.15.86:0/1765333334 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7a08042c70 con 0x7f7a180f5410 2024-02-19T21:59:06.415 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:06.412+0000 7f7a1d0ae700 1 --2- 172.21.15.86:0/1765333334 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f7a0402ff10 0x7f7a040323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:59:06.415 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:06.413+0000 7f7a1d0ae700 1 --2- 172.21.15.86:0/1765333334 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f7a0402ff10 0x7f7a040323d0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f7a140096a0 tx=0x7f7a14006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:59:06.416 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:06.417+0000 7f7a0e7fc700 1 -- 172.21.15.86:0/1765333334 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7a08014420 con 0x7f7a180f5410 2024-02-19T21:59:06.618 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:06.619+0000 7f7a1fb13700 1 -- 172.21.15.86:0/1765333334 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7a181070b0 con 0x7f7a180f5410 2024-02-19T21:59:06.619 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:06.619+0000 7f7a0e7fc700 1 -- 172.21.15.86:0/1765333334 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7a08011290 con 0x7f7a180f5410 2024-02-19T21:59:06.619 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T21:59:06.619 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T21:59:06.621 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:06.621+0000 7f7a1fb13700 1 -- 172.21.15.86:0/1765333334 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f7a0402ff10 msgr2=0x7f7a040323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:59:06.621 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:06.621+0000 7f7a1fb13700 1 --2- 172.21.15.86:0/1765333334 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f7a0402ff10 0x7f7a040323d0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f7a140096a0 tx=0x7f7a14006b40).stop 2024-02-19T21:59:06.621 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:06.621+0000 7f7a1fb13700 1 -- 172.21.15.86:0/1765333334 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7a180f5410 msgr2=0x7f7a181056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:59:06.621 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:06.621+0000 7f7a1fb13700 1 --2- 172.21.15.86:0/1765333334 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7a180f5410 0x7f7a181056e0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f7a08005010 tx=0x7f7a0800b040).stop 2024-02-19T21:59:06.621 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:06.621+0000 7f7a1fb13700 1 -- 172.21.15.86:0/1765333334 shutdown_connections 2024-02-19T21:59:06.621 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:06.621+0000 7f7a1fb13700 1 --2- 172.21.15.86:0/1765333334 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f7a0402ff10 0x7f7a040323d0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:59:06.621 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:06.621+0000 7f7a1fb13700 1 --2- 172.21.15.86:0/1765333334 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7a180f5410 0x7f7a181056e0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:59:06.621 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:06.621+0000 7f7a1fb13700 1 -- 172.21.15.86:0/1765333334 >> 172.21.15.86:0/1765333334 conn(0x7f7a180f0b70 msgr2=0x7f7a180f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:59:06.621 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:06.621+0000 7f7a1fb13700 1 -- 172.21.15.86:0/1765333334 shutdown_connections 2024-02-19T21:59:06.621 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:06.621+0000 7f7a1fb13700 1 -- 172.21.15.86:0/1765333334 wait complete. 2024-02-19T21:59:06.622 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T21:59:06.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:59:06 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T21:59:05.458502+0000 mgr.smithi053.fyhcdi (mgr.14162) 76 : cluster [DBG] pgmap v69: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:59:07.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:59:07 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:59:06.620412+0000 mon.smithi053 (mon.0) 151 : audit [DBG] from='client.? 172.21.15.86:0/1765333334' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T21:59:08.255 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T21:59:08.255 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T21:59:08.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:59:08 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T21:59:07.458749+0000 mgr.smithi053.fyhcdi (mgr.14162) 77 : cluster [DBG] pgmap v70: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:59:10.032 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:10.031+0000 7f2eda228700 1 -- 172.21.15.86:0/1286154209 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2ed40f51a0 msgr2=0x7f2ed40f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:59:10.032 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:10.031+0000 7f2eda228700 1 --2- 172.21.15.86:0/1286154209 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2ed40f51a0 0x7f2ed40f55c0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7f2ed0009fb0 tx=0x7f2ed0009b30).stop 2024-02-19T21:59:10.032 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:10.031+0000 7f2eda228700 1 -- 172.21.15.86:0/1286154209 shutdown_connections 2024-02-19T21:59:10.032 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:10.031+0000 7f2eda228700 1 --2- 172.21.15.86:0/1286154209 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2ed40f51a0 0x7f2ed40f55c0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:59:10.033 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:10.031+0000 7f2eda228700 1 -- 172.21.15.86:0/1286154209 >> 172.21.15.86:0/1286154209 conn(0x7f2ed40f09a0 msgr2=0x7f2ed40f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:59:10.033 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:10.031+0000 7f2eda228700 1 -- 172.21.15.86:0/1286154209 shutdown_connections 2024-02-19T21:59:10.033 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:10.031+0000 7f2eda228700 1 -- 172.21.15.86:0/1286154209 wait complete. 2024-02-19T21:59:10.033 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:10.031+0000 7f2eda228700 1 Processor -- start 2024-02-19T21:59:10.033 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:10.031+0000 7f2eda228700 1 -- start start 2024-02-19T21:59:10.033 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:10.031+0000 7f2eda228700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2ed40f51a0 0x7f2ed40f2f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:59:10.033 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:10.031+0000 7f2eda228700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ed40f13a0 con 0x7f2ed40f51a0 2024-02-19T21:59:10.033 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:10.032+0000 7f2ed9226700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2ed40f51a0 0x7f2ed40f2f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:59:10.033 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:10.032+0000 7f2ed9226700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2ed40f51a0 0x7f2ed40f2f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:37902/0 (socket says 172.21.15.86:37902) 2024-02-19T21:59:10.033 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:10.032+0000 7f2ed9226700 1 -- 172.21.15.86:0/1992628354 learned_addr learned my addr 172.21.15.86:0/1992628354 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T21:59:10.033 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:10.032+0000 7f2ed9226700 1 -- 172.21.15.86:0/1992628354 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ed0005040 con 0x7f2ed40f51a0 2024-02-19T21:59:10.033 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:10.032+0000 7f2ed9226700 1 --2- 172.21.15.86:0/1992628354 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2ed40f51a0 0x7f2ed40f2f20 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f2ed00089f0 tx=0x7f2ed000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:59:10.033 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:10.032+0000 7f2ec9ffb700 1 -- 172.21.15.86:0/1992628354 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2ed0009320 con 0x7f2ed40f51a0 2024-02-19T21:59:10.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:10.032+0000 7f2eda228700 1 -- 172.21.15.86:0/1992628354 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ed40f1540 con 0x7f2ed40f51a0 2024-02-19T21:59:10.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:10.032+0000 7f2ec9ffb700 1 -- 172.21.15.86:0/1992628354 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2ed0013380 con 0x7f2ed40f51a0 2024-02-19T21:59:10.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:10.032+0000 7f2ec9ffb700 1 -- 172.21.15.86:0/1992628354 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2ed0014e20 con 0x7f2ed40f51a0 2024-02-19T21:59:10.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:10.032+0000 7f2eda228700 1 -- 172.21.15.86:0/1992628354 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ed40f19d0 con 0x7f2ed40f51a0 2024-02-19T21:59:10.035 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:10.033+0000 7f2ec9ffb700 1 -- 172.21.15.86:0/1992628354 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2ed00095f0 con 0x7f2ed40f51a0 2024-02-19T21:59:10.035 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:10.033+0000 7f2eda228700 1 -- 172.21.15.86:0/1992628354 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ed4059100 con 0x7f2ed40f51a0 2024-02-19T21:59:10.035 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:10.033+0000 7f2ec9ffb700 1 --2- 172.21.15.86:0/1992628354 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f2ec002ff10 0x7f2ec00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:59:10.035 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:10.033+0000 7f2ec9ffb700 1 -- 172.21.15.86:0/1992628354 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2ed0040ec0 con 0x7f2ed40f51a0 2024-02-19T21:59:10.035 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:10.034+0000 7f2ed8a25700 1 --2- 172.21.15.86:0/1992628354 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f2ec002ff10 0x7f2ec00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:59:10.035 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:10.034+0000 7f2ed8a25700 1 --2- 172.21.15.86:0/1992628354 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f2ec002ff10 0x7f2ec00323d0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f2ec40096a0 tx=0x7f2ec4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:59:10.038 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:10.038+0000 7f2ec9ffb700 1 -- 172.21.15.86:0/1992628354 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2ed0014420 con 0x7f2ed40f51a0 2024-02-19T21:59:10.243 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:10.243+0000 7f2eda228700 1 -- 172.21.15.86:0/1992628354 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2ed4045ad0 con 0x7f2ed40f51a0 2024-02-19T21:59:10.244 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:10.244+0000 7f2ec9ffb700 1 -- 172.21.15.86:0/1992628354 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2ed0010230 con 0x7f2ed40f51a0 2024-02-19T21:59:10.244 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T21:59:10.244 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T21:59:10.245 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:10.246+0000 7f2eda228700 1 -- 172.21.15.86:0/1992628354 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f2ec002ff10 msgr2=0x7f2ec00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:59:10.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:10.246+0000 7f2eda228700 1 --2- 172.21.15.86:0/1992628354 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f2ec002ff10 0x7f2ec00323d0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f2ec40096a0 tx=0x7f2ec4006b40).stop 2024-02-19T21:59:10.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:10.246+0000 7f2eda228700 1 -- 172.21.15.86:0/1992628354 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2ed40f51a0 msgr2=0x7f2ed40f2f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:59:10.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:10.246+0000 7f2eda228700 1 --2- 172.21.15.86:0/1992628354 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2ed40f51a0 0x7f2ed40f2f20 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f2ed00089f0 tx=0x7f2ed000b040).stop 2024-02-19T21:59:10.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:10.246+0000 7f2eda228700 1 -- 172.21.15.86:0/1992628354 shutdown_connections 2024-02-19T21:59:10.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:10.246+0000 7f2eda228700 1 --2- 172.21.15.86:0/1992628354 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f2ec002ff10 0x7f2ec00323d0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:59:10.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:10.246+0000 7f2eda228700 1 --2- 172.21.15.86:0/1992628354 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2ed40f51a0 0x7f2ed40f2f20 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:59:10.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:10.246+0000 7f2eda228700 1 -- 172.21.15.86:0/1992628354 >> 172.21.15.86:0/1992628354 conn(0x7f2ed40f09a0 msgr2=0x7f2ed405fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:59:10.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:10.246+0000 7f2eda228700 1 -- 172.21.15.86:0/1992628354 shutdown_connections 2024-02-19T21:59:10.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:10.246+0000 7f2eda228700 1 -- 172.21.15.86:0/1992628354 wait complete. 2024-02-19T21:59:10.247 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T21:59:10.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:59:10 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T21:59:09.458964+0000 mgr.smithi053.fyhcdi (mgr.14162) 78 : cluster [DBG] pgmap v71: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:59:10.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:59:10 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:59:10.245306+0000 mon.smithi053 (mon.0) 152 : audit [DBG] from='client.? 172.21.15.86:0/1992628354' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T21:59:12.065 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T21:59:12.066 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T21:59:12.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:59:12 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T21:59:11.459192+0000 mgr.smithi053.fyhcdi (mgr.14162) 79 : cluster [DBG] pgmap v72: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:59:13.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:13.793+0000 7fbc57fff700 1 -- 172.21.15.86:0/3721839245 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbc580f62a0 msgr2=0x7fbc580f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:59:13.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:13.793+0000 7fbc57fff700 1 --2- 172.21.15.86:0/3721839245 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbc580f62a0 0x7fbc580f66c0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7fbc48009fb0 tx=0x7fbc48009b30).stop 2024-02-19T21:59:13.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:13.793+0000 7fbc57fff700 1 -- 172.21.15.86:0/3721839245 shutdown_connections 2024-02-19T21:59:13.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:13.793+0000 7fbc57fff700 1 --2- 172.21.15.86:0/3721839245 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbc580f62a0 0x7fbc580f66c0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:59:13.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:13.793+0000 7fbc57fff700 1 -- 172.21.15.86:0/3721839245 >> 172.21.15.86:0/3721839245 conn(0x7fbc580f1aa0 msgr2=0x7fbc580f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:59:13.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:13.793+0000 7fbc57fff700 1 -- 172.21.15.86:0/3721839245 shutdown_connections 2024-02-19T21:59:13.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:13.793+0000 7fbc57fff700 1 -- 172.21.15.86:0/3721839245 wait complete. 2024-02-19T21:59:13.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:13.794+0000 7fbc57fff700 1 Processor -- start 2024-02-19T21:59:13.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:13.794+0000 7fbc57fff700 1 -- start start 2024-02-19T21:59:13.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:13.794+0000 7fbc57fff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbc580f62a0 0x7fbc581065d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:59:13.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:13.794+0000 7fbc57fff700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc58106ac0 con 0x7fbc580f62a0 2024-02-19T21:59:13.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:13.794+0000 7fbc56ffd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbc580f62a0 0x7fbc581065d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:59:13.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:13.794+0000 7fbc56ffd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbc580f62a0 0x7fbc581065d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:37918/0 (socket says 172.21.15.86:37918) 2024-02-19T21:59:13.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:13.794+0000 7fbc56ffd700 1 -- 172.21.15.86:0/3836821108 learned_addr learned my addr 172.21.15.86:0/3836821108 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T21:59:13.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:13.794+0000 7fbc56ffd700 1 -- 172.21.15.86:0/3836821108 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc48005040 con 0x7fbc580f62a0 2024-02-19T21:59:13.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:13.795+0000 7fbc56ffd700 1 --2- 172.21.15.86:0/3836821108 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbc580f62a0 0x7fbc581065d0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7fbc480089f0 tx=0x7fbc4800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:59:13.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:13.795+0000 7fbc3f7fe700 1 -- 172.21.15.86:0/3836821108 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbc48009320 con 0x7fbc580f62a0 2024-02-19T21:59:13.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:13.795+0000 7fbc3f7fe700 1 -- 172.21.15.86:0/3836821108 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbc48013380 con 0x7fbc580f62a0 2024-02-19T21:59:13.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:13.795+0000 7fbc57fff700 1 -- 172.21.15.86:0/3836821108 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc58106cc0 con 0x7fbc580f62a0 2024-02-19T21:59:13.805 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:13.795+0000 7fbc3f7fe700 1 -- 172.21.15.86:0/3836821108 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbc48014e20 con 0x7fbc580f62a0 2024-02-19T21:59:13.805 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:13.795+0000 7fbc57fff700 1 -- 172.21.15.86:0/3836821108 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc58107150 con 0x7fbc580f62a0 2024-02-19T21:59:13.805 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:13.796+0000 7fbc3f7fe700 1 -- 172.21.15.86:0/3836821108 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbc480095f0 con 0x7fbc580f62a0 2024-02-19T21:59:13.805 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:13.796+0000 7fbc57fff700 1 -- 172.21.15.86:0/3836821108 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc58045ad0 con 0x7fbc580f62a0 2024-02-19T21:59:13.805 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:13.796+0000 7fbc3f7fe700 1 --2- 172.21.15.86:0/3836821108 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fbc4002ff10 0x7fbc400323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:59:13.805 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:13.796+0000 7fbc3f7fe700 1 -- 172.21.15.86:0/3836821108 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbc48040f50 con 0x7fbc580f62a0 2024-02-19T21:59:13.805 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:13.796+0000 7fbc567fc700 1 --2- 172.21.15.86:0/3836821108 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fbc4002ff10 0x7fbc400323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:59:13.805 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:13.796+0000 7fbc567fc700 1 --2- 172.21.15.86:0/3836821108 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fbc4002ff10 0x7fbc400323d0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7fbc4c0096a0 tx=0x7fbc4c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:59:13.805 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:13.801+0000 7fbc3f7fe700 1 -- 172.21.15.86:0/3836821108 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbc48014420 con 0x7fbc580f62a0 2024-02-19T21:59:14.004 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:14.004+0000 7fbc57fff700 1 -- 172.21.15.86:0/3836821108 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbc58108000 con 0x7fbc580f62a0 2024-02-19T21:59:14.004 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:14.006+0000 7fbc3f7fe700 1 -- 172.21.15.86:0/3836821108 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fbc4801d980 con 0x7fbc580f62a0 2024-02-19T21:59:14.005 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T21:59:14.005 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T21:59:14.006 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:14.006+0000 7fbc57fff700 1 -- 172.21.15.86:0/3836821108 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fbc4002ff10 msgr2=0x7fbc400323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:59:14.006 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:14.006+0000 7fbc57fff700 1 --2- 172.21.15.86:0/3836821108 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fbc4002ff10 0x7fbc400323d0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7fbc4c0096a0 tx=0x7fbc4c006b40).stop 2024-02-19T21:59:14.006 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:14.006+0000 7fbc57fff700 1 -- 172.21.15.86:0/3836821108 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbc580f62a0 msgr2=0x7fbc581065d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:59:14.006 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:14.006+0000 7fbc57fff700 1 --2- 172.21.15.86:0/3836821108 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbc580f62a0 0x7fbc581065d0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7fbc480089f0 tx=0x7fbc4800b040).stop 2024-02-19T21:59:14.007 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:14.006+0000 7fbc57fff700 1 -- 172.21.15.86:0/3836821108 shutdown_connections 2024-02-19T21:59:14.007 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:14.006+0000 7fbc57fff700 1 --2- 172.21.15.86:0/3836821108 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fbc4002ff10 0x7fbc400323d0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:59:14.007 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:14.006+0000 7fbc57fff700 1 --2- 172.21.15.86:0/3836821108 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbc580f62a0 0x7fbc581065d0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:59:14.007 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:14.006+0000 7fbc57fff700 1 -- 172.21.15.86:0/3836821108 >> 172.21.15.86:0/3836821108 conn(0x7fbc580f1aa0 msgr2=0x7fbc580f2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:59:14.007 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:14.006+0000 7fbc57fff700 1 -- 172.21.15.86:0/3836821108 shutdown_connections 2024-02-19T21:59:14.007 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:14.006+0000 7fbc57fff700 1 -- 172.21.15.86:0/3836821108 wait complete. 2024-02-19T21:59:14.007 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T21:59:14.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:59:14 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T21:59:13.459460+0000 mgr.smithi053.fyhcdi (mgr.14162) 80 : cluster [DBG] pgmap v73: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:59:14.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:59:14 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:59:14.005837+0000 mon.smithi053 (mon.0) 153 : audit [DBG] from='client.? 172.21.15.86:0/3836821108' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T21:59:15.674 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T21:59:15.674 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T21:59:16.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:59:16 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T21:59:15.459789+0000 mgr.smithi053.fyhcdi (mgr.14162) 81 : cluster [DBG] pgmap v74: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:59:17.480 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:17.477+0000 7fe72862a700 1 -- 172.21.15.86:0/2499870955 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe7200f5410 msgr2=0x7fe7200f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:59:17.480 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:17.477+0000 7fe72862a700 1 --2- 172.21.15.86:0/2499870955 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe7200f5410 0x7fe7200f5830 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7fe71c009fb0 tx=0x7fe71c009b30).stop 2024-02-19T21:59:17.480 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:17.477+0000 7fe72862a700 1 -- 172.21.15.86:0/2499870955 shutdown_connections 2024-02-19T21:59:17.480 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:17.477+0000 7fe72862a700 1 --2- 172.21.15.86:0/2499870955 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe7200f5410 0x7fe7200f5830 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:59:17.480 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:17.477+0000 7fe72862a700 1 -- 172.21.15.86:0/2499870955 >> 172.21.15.86:0/2499870955 conn(0x7fe7200f0b70 msgr2=0x7fe7200f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:59:17.480 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:17.477+0000 7fe72862a700 1 -- 172.21.15.86:0/2499870955 shutdown_connections 2024-02-19T21:59:17.480 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:17.477+0000 7fe72862a700 1 -- 172.21.15.86:0/2499870955 wait complete. 2024-02-19T21:59:17.480 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:17.477+0000 7fe72862a700 1 Processor -- start 2024-02-19T21:59:17.481 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:17.478+0000 7fe72862a700 1 -- start start 2024-02-19T21:59:17.481 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:17.478+0000 7fe72862a700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe7200f5410 0x7fe7201056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:59:17.481 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:17.478+0000 7fe72862a700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe720105bd0 con 0x7fe7200f5410 2024-02-19T21:59:17.481 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:17.478+0000 7fe7263c6700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe7200f5410 0x7fe7201056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:59:17.481 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:17.478+0000 7fe7263c6700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe7200f5410 0x7fe7201056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:37928/0 (socket says 172.21.15.86:37928) 2024-02-19T21:59:17.481 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:17.478+0000 7fe7263c6700 1 -- 172.21.15.86:0/4005375864 learned_addr learned my addr 172.21.15.86:0/4005375864 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T21:59:17.481 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:17.478+0000 7fe7263c6700 1 -- 172.21.15.86:0/4005375864 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe71c005040 con 0x7fe7200f5410 2024-02-19T21:59:17.481 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:17.478+0000 7fe7263c6700 1 --2- 172.21.15.86:0/4005375864 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe7200f5410 0x7fe7201056e0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7fe71c005010 tx=0x7fe71c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:59:17.481 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:17.479+0000 7fe716ffd700 1 -- 172.21.15.86:0/4005375864 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe71c009320 con 0x7fe7200f5410 2024-02-19T21:59:17.481 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:17.479+0000 7fe716ffd700 1 -- 172.21.15.86:0/4005375864 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe71c013380 con 0x7fe7200f5410 2024-02-19T21:59:17.482 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:17.479+0000 7fe72862a700 1 -- 172.21.15.86:0/4005375864 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe720105dd0 con 0x7fe7200f5410 2024-02-19T21:59:17.482 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:17.479+0000 7fe716ffd700 1 -- 172.21.15.86:0/4005375864 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe71c014e50 con 0x7fe7200f5410 2024-02-19T21:59:17.482 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:17.479+0000 7fe72862a700 1 -- 172.21.15.86:0/4005375864 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe720106260 con 0x7fe7200f5410 2024-02-19T21:59:17.482 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:17.479+0000 7fe716ffd700 1 -- 172.21.15.86:0/4005375864 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe71c0095f0 con 0x7fe7200f5410 2024-02-19T21:59:17.482 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:17.480+0000 7fe72862a700 1 -- 172.21.15.86:0/4005375864 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe720045ad0 con 0x7fe7200f5410 2024-02-19T21:59:17.482 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:17.480+0000 7fe716ffd700 1 --2- 172.21.15.86:0/4005375864 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fe70c02ff10 0x7fe70c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:59:17.482 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:17.480+0000 7fe716ffd700 1 -- 172.21.15.86:0/4005375864 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe71c040f10 con 0x7fe7200f5410 2024-02-19T21:59:17.482 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:17.480+0000 7fe725bc5700 1 --2- 172.21.15.86:0/4005375864 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fe70c02ff10 0x7fe70c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:59:17.482 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:17.480+0000 7fe725bc5700 1 --2- 172.21.15.86:0/4005375864 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fe70c02ff10 0x7fe70c0323d0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7fe7100096a0 tx=0x7fe710006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:59:17.483 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:17.484+0000 7fe716ffd700 1 -- 172.21.15.86:0/4005375864 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe71c014420 con 0x7fe7200f5410 2024-02-19T21:59:17.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:17.690+0000 7fe72862a700 1 -- 172.21.15.86:0/4005375864 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe720107110 con 0x7fe7200f5410 2024-02-19T21:59:17.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:17.691+0000 7fe716ffd700 1 -- 172.21.15.86:0/4005375864 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe71c010230 con 0x7fe7200f5410 2024-02-19T21:59:17.690 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T21:59:17.690 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T21:59:17.691 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:17.692+0000 7fe72862a700 1 -- 172.21.15.86:0/4005375864 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fe70c02ff10 msgr2=0x7fe70c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:59:17.691 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:17.692+0000 7fe72862a700 1 --2- 172.21.15.86:0/4005375864 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fe70c02ff10 0x7fe70c0323d0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7fe7100096a0 tx=0x7fe710006b40).stop 2024-02-19T21:59:17.691 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:17.692+0000 7fe72862a700 1 -- 172.21.15.86:0/4005375864 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe7200f5410 msgr2=0x7fe7201056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:59:17.691 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:17.692+0000 7fe72862a700 1 --2- 172.21.15.86:0/4005375864 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe7200f5410 0x7fe7201056e0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7fe71c005010 tx=0x7fe71c00b040).stop 2024-02-19T21:59:17.691 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:17.692+0000 7fe72862a700 1 -- 172.21.15.86:0/4005375864 shutdown_connections 2024-02-19T21:59:17.692 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:17.692+0000 7fe72862a700 1 --2- 172.21.15.86:0/4005375864 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fe70c02ff10 0x7fe70c0323d0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:59:17.692 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:17.692+0000 7fe72862a700 1 --2- 172.21.15.86:0/4005375864 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe7200f5410 0x7fe7201056e0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:59:17.692 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:17.692+0000 7fe72862a700 1 -- 172.21.15.86:0/4005375864 >> 172.21.15.86:0/4005375864 conn(0x7fe7200f0b70 msgr2=0x7fe7200f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:59:17.692 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:17.692+0000 7fe72862a700 1 -- 172.21.15.86:0/4005375864 shutdown_connections 2024-02-19T21:59:17.692 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:17.692+0000 7fe72862a700 1 -- 172.21.15.86:0/4005375864 wait complete. 2024-02-19T21:59:17.692 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T21:59:18.794 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:59:18 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T21:59:17.460250+0000 mgr.smithi053.fyhcdi (mgr.14162) 82 : cluster [DBG] pgmap v75: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:59:18.794 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:59:18 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:59:17.690889+0000 mon.smithi053 (mon.0) 154 : audit [DBG] from='client.? 172.21.15.86:0/4005375864' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T21:59:19.297 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T21:59:19.297 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T21:59:20.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:59:20 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T21:59:19.460551+0000 mgr.smithi053.fyhcdi (mgr.14162) 83 : cluster [DBG] pgmap v76: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:59:21.193 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:21.193+0000 7f8857815700 1 -- 172.21.15.86:0/996048270 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f88500f6510 msgr2=0x7f88500f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:59:21.193 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:21.193+0000 7f8857815700 1 --2- 172.21.15.86:0/996048270 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f88500f6510 0x7f88500f6930 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7f8840009fb0 tx=0x7f8840009b30).stop 2024-02-19T21:59:21.193 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:21.194+0000 7f8857815700 1 -- 172.21.15.86:0/996048270 shutdown_connections 2024-02-19T21:59:21.193 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:21.194+0000 7f8857815700 1 --2- 172.21.15.86:0/996048270 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f88500f6510 0x7f88500f6930 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:59:21.194 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:21.194+0000 7f8857815700 1 -- 172.21.15.86:0/996048270 >> 172.21.15.86:0/996048270 conn(0x7f88500f1c70 msgr2=0x7f88500f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:59:21.194 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:21.194+0000 7f8857815700 1 -- 172.21.15.86:0/996048270 shutdown_connections 2024-02-19T21:59:21.194 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:21.194+0000 7f8857815700 1 -- 172.21.15.86:0/996048270 wait complete. 2024-02-19T21:59:21.194 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:21.194+0000 7f8857815700 1 Processor -- start 2024-02-19T21:59:21.194 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:21.194+0000 7f8857815700 1 -- start start 2024-02-19T21:59:21.194 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:21.194+0000 7f8857815700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f88500f6510 0x7f88501067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:59:21.194 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:21.194+0000 7f8857815700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8850106cd0 con 0x7f88500f6510 2024-02-19T21:59:21.194 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:21.194+0000 7f88555b1700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f88500f6510 0x7f88501067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:59:21.194 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:21.195+0000 7f88555b1700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f88500f6510 0x7f88501067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:43574/0 (socket says 172.21.15.86:43574) 2024-02-19T21:59:21.194 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:21.195+0000 7f88555b1700 1 -- 172.21.15.86:0/2474515906 learned_addr learned my addr 172.21.15.86:0/2474515906 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T21:59:21.196 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:21.195+0000 7f88555b1700 1 -- 172.21.15.86:0/2474515906 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8840005040 con 0x7f88500f6510 2024-02-19T21:59:21.196 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:21.195+0000 7f88555b1700 1 --2- 172.21.15.86:0/2474515906 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f88500f6510 0x7f88501067e0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f8840005010 tx=0x7f884000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:59:21.196 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:21.195+0000 7f8845ffb700 1 -- 172.21.15.86:0/2474515906 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8840009320 con 0x7f88500f6510 2024-02-19T21:59:21.196 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:21.195+0000 7f8857815700 1 -- 172.21.15.86:0/2474515906 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8850106ed0 con 0x7f88500f6510 2024-02-19T21:59:21.196 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:21.195+0000 7f8845ffb700 1 -- 172.21.15.86:0/2474515906 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8840013380 con 0x7f88500f6510 2024-02-19T21:59:21.196 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:21.195+0000 7f8845ffb700 1 -- 172.21.15.86:0/2474515906 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8840014e50 con 0x7f88500f6510 2024-02-19T21:59:21.196 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:21.195+0000 7f8857815700 1 -- 172.21.15.86:0/2474515906 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8850107360 con 0x7f88500f6510 2024-02-19T21:59:21.196 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:21.196+0000 7f8845ffb700 1 -- 172.21.15.86:0/2474515906 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f88400095f0 con 0x7f88500f6510 2024-02-19T21:59:21.196 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:21.196+0000 7f8857815700 1 -- 172.21.15.86:0/2474515906 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8850045ad0 con 0x7f88500f6510 2024-02-19T21:59:21.196 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:21.196+0000 7f8845ffb700 1 --2- 172.21.15.86:0/2474515906 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f883c02ff10 0x7f883c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:59:21.196 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:21.196+0000 7f8845ffb700 1 -- 172.21.15.86:0/2474515906 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8840040f80 con 0x7f88500f6510 2024-02-19T21:59:21.196 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:21.197+0000 7f8854db0700 1 --2- 172.21.15.86:0/2474515906 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f883c02ff10 0x7f883c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:59:21.197 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:21.197+0000 7f8854db0700 1 --2- 172.21.15.86:0/2474515906 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f883c02ff10 0x7f883c0323d0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f884c0096a0 tx=0x7f884c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:59:21.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:21.201+0000 7f8845ffb700 1 -- 172.21.15.86:0/2474515906 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8840014420 con 0x7f88500f6510 2024-02-19T21:59:21.405 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:21.405+0000 7f8857815700 1 -- 172.21.15.86:0/2474515906 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8850108210 con 0x7f88500f6510 2024-02-19T21:59:21.406 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:21.406+0000 7f8845ffb700 1 -- 172.21.15.86:0/2474515906 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8840010230 con 0x7f88500f6510 2024-02-19T21:59:21.406 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T21:59:21.406 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T21:59:21.407 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:21.408+0000 7f8857815700 1 -- 172.21.15.86:0/2474515906 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f883c02ff10 msgr2=0x7f883c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:59:21.408 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:21.408+0000 7f8857815700 1 --2- 172.21.15.86:0/2474515906 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f883c02ff10 0x7f883c0323d0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f884c0096a0 tx=0x7f884c006b40).stop 2024-02-19T21:59:21.408 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:21.408+0000 7f8857815700 1 -- 172.21.15.86:0/2474515906 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f88500f6510 msgr2=0x7f88501067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:59:21.408 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:21.408+0000 7f8857815700 1 --2- 172.21.15.86:0/2474515906 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f88500f6510 0x7f88501067e0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f8840005010 tx=0x7f884000b040).stop 2024-02-19T21:59:21.408 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:21.408+0000 7f8857815700 1 -- 172.21.15.86:0/2474515906 shutdown_connections 2024-02-19T21:59:21.408 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:21.408+0000 7f8857815700 1 --2- 172.21.15.86:0/2474515906 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f883c02ff10 0x7f883c0323d0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:59:21.408 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:21.408+0000 7f8857815700 1 --2- 172.21.15.86:0/2474515906 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f88500f6510 0x7f88501067e0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:59:21.408 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:21.408+0000 7f8857815700 1 -- 172.21.15.86:0/2474515906 >> 172.21.15.86:0/2474515906 conn(0x7f88500f1c70 msgr2=0x7f88500f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:59:21.408 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:21.408+0000 7f8857815700 1 -- 172.21.15.86:0/2474515906 shutdown_connections 2024-02-19T21:59:21.408 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:21.408+0000 7f8857815700 1 -- 172.21.15.86:0/2474515906 wait complete. 2024-02-19T21:59:21.409 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T21:59:21.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:59:21 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:59:21.407412+0000 mon.smithi053 (mon.0) 155 : audit [DBG] from='client.? 172.21.15.86:0/2474515906' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T21:59:22.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:59:22 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T21:59:21.460790+0000 mgr.smithi053.fyhcdi (mgr.14162) 84 : cluster [DBG] pgmap v77: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:59:23.065 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T21:59:23.065 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T21:59:24.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:59:24 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T21:59:23.461108+0000 mgr.smithi053.fyhcdi (mgr.14162) 85 : cluster [DBG] pgmap v78: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:59:24.885 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:24.884+0000 7fc8101b4700 1 -- 172.21.15.86:0/4056059640 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc8080f6510 msgr2=0x7fc8080f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:59:24.885 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:24.884+0000 7fc8101b4700 1 --2- 172.21.15.86:0/4056059640 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc8080f6510 0x7fc8080f6930 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7fc804009fb0 tx=0x7fc804009b30).stop 2024-02-19T21:59:24.885 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:24.884+0000 7fc8101b4700 1 -- 172.21.15.86:0/4056059640 shutdown_connections 2024-02-19T21:59:24.886 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:24.884+0000 7fc8101b4700 1 --2- 172.21.15.86:0/4056059640 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc8080f6510 0x7fc8080f6930 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:59:24.886 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:24.884+0000 7fc8101b4700 1 -- 172.21.15.86:0/4056059640 >> 172.21.15.86:0/4056059640 conn(0x7fc8080f1c70 msgr2=0x7fc8080f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:59:24.886 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:24.884+0000 7fc8101b4700 1 -- 172.21.15.86:0/4056059640 shutdown_connections 2024-02-19T21:59:24.886 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:24.884+0000 7fc8101b4700 1 -- 172.21.15.86:0/4056059640 wait complete. 2024-02-19T21:59:24.886 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:24.884+0000 7fc8101b4700 1 Processor -- start 2024-02-19T21:59:24.886 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:24.884+0000 7fc8101b4700 1 -- start start 2024-02-19T21:59:24.886 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:24.885+0000 7fc8101b4700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc8080f6510 0x7fc808108a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:59:24.886 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:24.885+0000 7fc8101b4700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc808108ef0 con 0x7fc8080f6510 2024-02-19T21:59:24.886 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:24.885+0000 7fc80df50700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc8080f6510 0x7fc808108a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:59:24.886 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:24.885+0000 7fc80df50700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc8080f6510 0x7fc808108a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:43596/0 (socket says 172.21.15.86:43596) 2024-02-19T21:59:24.886 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:24.885+0000 7fc80df50700 1 -- 172.21.15.86:0/2366632626 learned_addr learned my addr 172.21.15.86:0/2366632626 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T21:59:24.886 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:24.885+0000 7fc80df50700 1 -- 172.21.15.86:0/2366632626 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc804005040 con 0x7fc8080f6510 2024-02-19T21:59:24.886 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:24.885+0000 7fc80df50700 1 --2- 172.21.15.86:0/2366632626 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc8080f6510 0x7fc808108a00 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7fc8040089f0 tx=0x7fc80400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:59:24.886 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:24.885+0000 7fc7fe7fc700 1 -- 172.21.15.86:0/2366632626 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc804009320 con 0x7fc8080f6510 2024-02-19T21:59:24.887 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:24.885+0000 7fc8101b4700 1 -- 172.21.15.86:0/2366632626 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8081090f0 con 0x7fc8080f6510 2024-02-19T21:59:24.887 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:24.885+0000 7fc7fe7fc700 1 -- 172.21.15.86:0/2366632626 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc804013380 con 0x7fc8080f6510 2024-02-19T21:59:24.887 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:24.885+0000 7fc7fe7fc700 1 -- 172.21.15.86:0/2366632626 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc80401ed80 con 0x7fc8080f6510 2024-02-19T21:59:24.887 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:24.885+0000 7fc8101b4700 1 -- 172.21.15.86:0/2366632626 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc808109580 con 0x7fc8080f6510 2024-02-19T21:59:24.888 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:24.886+0000 7fc7fe7fc700 1 -- 172.21.15.86:0/2366632626 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc80401e420 con 0x7fc8080f6510 2024-02-19T21:59:24.888 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:24.886+0000 7fc8101b4700 1 -- 172.21.15.86:0/2366632626 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc808045ad0 con 0x7fc8080f6510 2024-02-19T21:59:24.888 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:24.886+0000 7fc7fe7fc700 1 --2- 172.21.15.86:0/2366632626 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fc7f402ff60 0x7fc7f4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:59:24.888 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:24.886+0000 7fc7fe7fc700 1 -- 172.21.15.86:0/2366632626 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc804041110 con 0x7fc8080f6510 2024-02-19T21:59:24.888 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:24.887+0000 7fc80d74f700 1 --2- 172.21.15.86:0/2366632626 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fc7f402ff60 0x7fc7f4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:59:24.888 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:24.887+0000 7fc80d74f700 1 --2- 172.21.15.86:0/2366632626 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fc7f402ff60 0x7fc7f4032420 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7fc7f80096a0 tx=0x7fc7f8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:59:24.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:24.891+0000 7fc7fe7fc700 1 -- 172.21.15.86:0/2366632626 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc8040095f0 con 0x7fc8080f6510 2024-02-19T21:59:25.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:25.096+0000 7fc8101b4700 1 -- 172.21.15.86:0/2366632626 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc8080fc0a0 con 0x7fc8080f6510 2024-02-19T21:59:25.097 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:25.097+0000 7fc7fe7fc700 1 -- 172.21.15.86:0/2366632626 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc804004490 con 0x7fc8080f6510 2024-02-19T21:59:25.097 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T21:59:25.097 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T21:59:25.099 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:25.098+0000 7fc8101b4700 1 -- 172.21.15.86:0/2366632626 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fc7f402ff60 msgr2=0x7fc7f4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:59:25.099 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:25.098+0000 7fc8101b4700 1 --2- 172.21.15.86:0/2366632626 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fc7f402ff60 0x7fc7f4032420 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7fc7f80096a0 tx=0x7fc7f8006b40).stop 2024-02-19T21:59:25.099 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:25.098+0000 7fc8101b4700 1 -- 172.21.15.86:0/2366632626 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc8080f6510 msgr2=0x7fc808108a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:59:25.099 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:25.098+0000 7fc8101b4700 1 --2- 172.21.15.86:0/2366632626 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc8080f6510 0x7fc808108a00 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7fc8040089f0 tx=0x7fc80400b040).stop 2024-02-19T21:59:25.099 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:25.098+0000 7fc8101b4700 1 -- 172.21.15.86:0/2366632626 shutdown_connections 2024-02-19T21:59:25.099 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:25.098+0000 7fc8101b4700 1 --2- 172.21.15.86:0/2366632626 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fc7f402ff60 0x7fc7f4032420 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:59:25.099 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:25.098+0000 7fc8101b4700 1 --2- 172.21.15.86:0/2366632626 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc8080f6510 0x7fc808108a00 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:59:25.099 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:25.098+0000 7fc8101b4700 1 -- 172.21.15.86:0/2366632626 >> 172.21.15.86:0/2366632626 conn(0x7fc8080f1c70 msgr2=0x7fc8080f2690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:59:25.099 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:25.098+0000 7fc8101b4700 1 -- 172.21.15.86:0/2366632626 shutdown_connections 2024-02-19T21:59:25.099 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:25.098+0000 7fc8101b4700 1 -- 172.21.15.86:0/2366632626 wait complete. 2024-02-19T21:59:25.099 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T21:59:25.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:59:25 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:59:25.098201+0000 mon.smithi053 (mon.0) 156 : audit [DBG] from='client.? 172.21.15.86:0/2366632626' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T21:59:26.709 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T21:59:26.709 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T21:59:26.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:59:26 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T21:59:25.461451+0000 mgr.smithi053.fyhcdi (mgr.14162) 86 : cluster [DBG] pgmap v79: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:59:28.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:28.557+0000 7fa269973700 1 -- 172.21.15.86:0/1740046723 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa2640f6510 msgr2=0x7fa2640f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:59:28.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:28.557+0000 7fa269973700 1 --2- 172.21.15.86:0/1740046723 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa2640f6510 0x7fa2640f6930 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7fa24c009fb0 tx=0x7fa24c009b30).stop 2024-02-19T21:59:28.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:28.557+0000 7fa269973700 1 -- 172.21.15.86:0/1740046723 shutdown_connections 2024-02-19T21:59:28.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:28.557+0000 7fa269973700 1 --2- 172.21.15.86:0/1740046723 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa2640f6510 0x7fa2640f6930 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:59:28.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:28.557+0000 7fa269973700 1 -- 172.21.15.86:0/1740046723 >> 172.21.15.86:0/1740046723 conn(0x7fa2640f1c70 msgr2=0x7fa2640f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:59:28.560 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:28.557+0000 7fa269973700 1 -- 172.21.15.86:0/1740046723 shutdown_connections 2024-02-19T21:59:28.560 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:28.557+0000 7fa269973700 1 -- 172.21.15.86:0/1740046723 wait complete. 2024-02-19T21:59:28.560 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:28.558+0000 7fa269973700 1 Processor -- start 2024-02-19T21:59:28.560 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:28.558+0000 7fa269973700 1 -- start start 2024-02-19T21:59:28.560 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:28.558+0000 7fa269973700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa2640f6510 0x7fa2641067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:59:28.560 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:28.558+0000 7fa269973700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa264106cd0 con 0x7fa2640f6510 2024-02-19T21:59:28.560 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:28.558+0000 7fa262ffd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa2640f6510 0x7fa2641067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:59:28.560 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:28.558+0000 7fa262ffd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa2640f6510 0x7fa2641067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:43610/0 (socket says 172.21.15.86:43610) 2024-02-19T21:59:28.560 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:28.558+0000 7fa262ffd700 1 -- 172.21.15.86:0/4270915745 learned_addr learned my addr 172.21.15.86:0/4270915745 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T21:59:28.560 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:28.559+0000 7fa262ffd700 1 -- 172.21.15.86:0/4270915745 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa24c005040 con 0x7fa2640f6510 2024-02-19T21:59:28.560 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:28.559+0000 7fa262ffd700 1 --2- 172.21.15.86:0/4270915745 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa2640f6510 0x7fa2641067e0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7fa24c005010 tx=0x7fa24c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:59:28.560 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:28.559+0000 7fa25b7fe700 1 -- 172.21.15.86:0/4270915745 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa24c009320 con 0x7fa2640f6510 2024-02-19T21:59:28.560 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:28.559+0000 7fa269973700 1 -- 172.21.15.86:0/4270915745 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa264106ed0 con 0x7fa2640f6510 2024-02-19T21:59:28.560 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:28.559+0000 7fa25b7fe700 1 -- 172.21.15.86:0/4270915745 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa24c013380 con 0x7fa2640f6510 2024-02-19T21:59:28.561 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:28.559+0000 7fa25b7fe700 1 -- 172.21.15.86:0/4270915745 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa24c014e50 con 0x7fa2640f6510 2024-02-19T21:59:28.561 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:28.559+0000 7fa269973700 1 -- 172.21.15.86:0/4270915745 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa264107360 con 0x7fa2640f6510 2024-02-19T21:59:28.562 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:28.560+0000 7fa25b7fe700 1 -- 172.21.15.86:0/4270915745 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa24c014420 con 0x7fa2640f6510 2024-02-19T21:59:28.562 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:28.560+0000 7fa269973700 1 -- 172.21.15.86:0/4270915745 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa264045ad0 con 0x7fa2640f6510 2024-02-19T21:59:28.562 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:28.560+0000 7fa25b7fe700 1 --2- 172.21.15.86:0/4270915745 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fa25002ff10 0x7fa2500323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:59:28.562 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:28.560+0000 7fa25b7fe700 1 -- 172.21.15.86:0/4270915745 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa24c040e60 con 0x7fa2640f6510 2024-02-19T21:59:28.562 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:28.560+0000 7fa2627fc700 1 --2- 172.21.15.86:0/4270915745 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fa25002ff10 0x7fa2500323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:59:28.562 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:28.561+0000 7fa2627fc700 1 --2- 172.21.15.86:0/4270915745 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fa25002ff10 0x7fa2500323d0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7fa2540096a0 tx=0x7fa254006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:59:28.564 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:28.565+0000 7fa25b7fe700 1 -- 172.21.15.86:0/4270915745 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa24c0095f0 con 0x7fa2640f6510 2024-02-19T21:59:28.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:28.771+0000 7fa269973700 1 -- 172.21.15.86:0/4270915745 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa2641081b0 con 0x7fa2640f6510 2024-02-19T21:59:28.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:28.771+0000 7fa25b7fe700 1 -- 172.21.15.86:0/4270915745 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa24c01db70 con 0x7fa2640f6510 2024-02-19T21:59:28.771 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T21:59:28.772 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T21:59:28.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:28.773+0000 7fa269973700 1 -- 172.21.15.86:0/4270915745 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fa25002ff10 msgr2=0x7fa2500323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:59:28.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:28.773+0000 7fa269973700 1 --2- 172.21.15.86:0/4270915745 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fa25002ff10 0x7fa2500323d0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7fa2540096a0 tx=0x7fa254006b40).stop 2024-02-19T21:59:28.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:28.773+0000 7fa269973700 1 -- 172.21.15.86:0/4270915745 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa2640f6510 msgr2=0x7fa2641067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:59:28.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:28.773+0000 7fa269973700 1 --2- 172.21.15.86:0/4270915745 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa2640f6510 0x7fa2641067e0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7fa24c005010 tx=0x7fa24c00b040).stop 2024-02-19T21:59:28.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:28.773+0000 7fa269973700 1 -- 172.21.15.86:0/4270915745 shutdown_connections 2024-02-19T21:59:28.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:28.773+0000 7fa269973700 1 --2- 172.21.15.86:0/4270915745 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fa25002ff10 0x7fa2500323d0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:59:28.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:28.773+0000 7fa269973700 1 --2- 172.21.15.86:0/4270915745 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa2640f6510 0x7fa2641067e0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:59:28.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:28.773+0000 7fa269973700 1 -- 172.21.15.86:0/4270915745 >> 172.21.15.86:0/4270915745 conn(0x7fa2640f1c70 msgr2=0x7fa2640f26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:59:28.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:28.773+0000 7fa269973700 1 -- 172.21.15.86:0/4270915745 shutdown_connections 2024-02-19T21:59:28.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:28.773+0000 7fa269973700 1 -- 172.21.15.86:0/4270915745 wait complete. 2024-02-19T21:59:28.774 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T21:59:28.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:59:28 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T21:59:27.461756+0000 mgr.smithi053.fyhcdi (mgr.14162) 87 : cluster [DBG] pgmap v80: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:59:29.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:59:29 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:59:28.772759+0000 mon.smithi053 (mon.0) 157 : audit [DBG] from='client.? 172.21.15.86:0/4270915745' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T21:59:30.409 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T21:59:30.410 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T21:59:30.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:59:30 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T21:59:29.462000+0000 mgr.smithi053.fyhcdi (mgr.14162) 88 : cluster [DBG] pgmap v81: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:59:32.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:32.257+0000 7f1518793700 1 -- 172.21.15.86:0/2223560929 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f15100f5410 msgr2=0x7f15100f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:59:32.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:32.257+0000 7f1518793700 1 --2- 172.21.15.86:0/2223560929 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f15100f5410 0x7f15100f5830 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f150c009fb0 tx=0x7f150c009b30).stop 2024-02-19T21:59:32.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:32.257+0000 7f1518793700 1 -- 172.21.15.86:0/2223560929 shutdown_connections 2024-02-19T21:59:32.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:32.257+0000 7f1518793700 1 --2- 172.21.15.86:0/2223560929 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f15100f5410 0x7f15100f5830 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:59:32.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:32.257+0000 7f1518793700 1 -- 172.21.15.86:0/2223560929 >> 172.21.15.86:0/2223560929 conn(0x7f15100f0b70 msgr2=0x7f15100f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:59:32.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:32.257+0000 7f1518793700 1 -- 172.21.15.86:0/2223560929 shutdown_connections 2024-02-19T21:59:32.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:32.257+0000 7f1518793700 1 -- 172.21.15.86:0/2223560929 wait complete. 2024-02-19T21:59:32.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:32.258+0000 7f1518793700 1 Processor -- start 2024-02-19T21:59:32.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:32.258+0000 7f1518793700 1 -- start start 2024-02-19T21:59:32.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:32.258+0000 7f1518793700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f15100f5410 0x7f15101094e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:59:32.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:32.258+0000 7f1518793700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1510060240 con 0x7f15100f5410 2024-02-19T21:59:32.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:32.258+0000 7f151652f700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f15100f5410 0x7f15101094e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:59:32.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:32.258+0000 7f151652f700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f15100f5410 0x7f15101094e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:58782/0 (socket says 172.21.15.86:58782) 2024-02-19T21:59:32.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:32.258+0000 7f151652f700 1 -- 172.21.15.86:0/1286037534 learned_addr learned my addr 172.21.15.86:0/1286037534 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T21:59:32.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:32.259+0000 7f151652f700 1 -- 172.21.15.86:0/1286037534 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f150c005040 con 0x7f15100f5410 2024-02-19T21:59:32.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:32.259+0000 7f151652f700 1 --2- 172.21.15.86:0/1286037534 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f15100f5410 0x7f15101094e0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f150c005010 tx=0x7f150c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:59:32.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:32.259+0000 7f1506ffd700 1 -- 172.21.15.86:0/1286037534 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f150c009320 con 0x7f15100f5410 2024-02-19T21:59:32.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:32.259+0000 7f1518793700 1 -- 172.21.15.86:0/1286037534 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1510060440 con 0x7f15100f5410 2024-02-19T21:59:32.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:32.259+0000 7f1506ffd700 1 -- 172.21.15.86:0/1286037534 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f150c013380 con 0x7f15100f5410 2024-02-19T21:59:32.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:32.259+0000 7f1506ffd700 1 -- 172.21.15.86:0/1286037534 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f150c014e50 con 0x7f15100f5410 2024-02-19T21:59:32.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:32.259+0000 7f1518793700 1 -- 172.21.15.86:0/1286037534 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1510109b90 con 0x7f15100f5410 2024-02-19T21:59:32.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:32.260+0000 7f1506ffd700 1 -- 172.21.15.86:0/1286037534 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f150c0263a0 con 0x7f15100f5410 2024-02-19T21:59:32.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:32.260+0000 7f1506ffd700 1 --2- 172.21.15.86:0/1286037534 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f14fc02ff10 0x7f14fc0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:59:32.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:32.260+0000 7f1506ffd700 1 -- 172.21.15.86:0/1286037534 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f150c040fa0 con 0x7f15100f5410 2024-02-19T21:59:32.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:32.260+0000 7f1518793700 1 -- 172.21.15.86:0/1286037534 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1510059100 con 0x7f15100f5410 2024-02-19T21:59:32.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:32.260+0000 7f1515d2e700 1 --2- 172.21.15.86:0/1286037534 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f14fc02ff10 0x7f14fc0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:59:32.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:32.260+0000 7f1515d2e700 1 --2- 172.21.15.86:0/1286037534 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f14fc02ff10 0x7f14fc0323d0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f15000096a0 tx=0x7f1500006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:59:32.265 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:32.265+0000 7f1506ffd700 1 -- 172.21.15.86:0/1286037534 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f150c012920 con 0x7f15100f5410 2024-02-19T21:59:32.469 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:32.470+0000 7f1518793700 1 -- 172.21.15.86:0/1286037534 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f15100fafa0 con 0x7f15100f5410 2024-02-19T21:59:32.471 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:32.471+0000 7f1506ffd700 1 -- 172.21.15.86:0/1286037534 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f150c02b020 con 0x7f15100f5410 2024-02-19T21:59:32.471 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T21:59:32.472 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T21:59:32.473 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:32.473+0000 7f1518793700 1 -- 172.21.15.86:0/1286037534 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f14fc02ff10 msgr2=0x7f14fc0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:59:32.473 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:32.473+0000 7f1518793700 1 --2- 172.21.15.86:0/1286037534 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f14fc02ff10 0x7f14fc0323d0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f15000096a0 tx=0x7f1500006b40).stop 2024-02-19T21:59:32.473 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:32.473+0000 7f1518793700 1 -- 172.21.15.86:0/1286037534 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f15100f5410 msgr2=0x7f15101094e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:59:32.473 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:32.473+0000 7f1518793700 1 --2- 172.21.15.86:0/1286037534 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f15100f5410 0x7f15101094e0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f150c005010 tx=0x7f150c00b040).stop 2024-02-19T21:59:32.473 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:32.473+0000 7f1518793700 1 -- 172.21.15.86:0/1286037534 shutdown_connections 2024-02-19T21:59:32.473 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:32.473+0000 7f1518793700 1 --2- 172.21.15.86:0/1286037534 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f14fc02ff10 0x7f14fc0323d0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:59:32.473 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:32.473+0000 7f1518793700 1 --2- 172.21.15.86:0/1286037534 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f15100f5410 0x7f15101094e0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:59:32.473 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:32.473+0000 7f1518793700 1 -- 172.21.15.86:0/1286037534 >> 172.21.15.86:0/1286037534 conn(0x7f15100f0b70 msgr2=0x7f15100f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:59:32.473 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:32.473+0000 7f1518793700 1 -- 172.21.15.86:0/1286037534 shutdown_connections 2024-02-19T21:59:32.473 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:32.473+0000 7f1518793700 1 -- 172.21.15.86:0/1286037534 wait complete. 2024-02-19T21:59:32.474 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T21:59:32.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:59:32 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T21:59:31.462314+0000 mgr.smithi053.fyhcdi (mgr.14162) 89 : cluster [DBG] pgmap v82: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:59:33.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:59:33 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:59:32.472587+0000 mon.smithi053 (mon.0) 158 : audit [DBG] from='client.? 172.21.15.86:0/1286037534' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T21:59:34.127 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T21:59:34.127 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T21:59:34.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:59:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T21:59:33.462595+0000 mgr.smithi053.fyhcdi (mgr.14162) 90 : cluster [DBG] pgmap v83: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:59:36.045 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:36.043+0000 7f45808a6700 1 -- 172.21.15.86:0/1411810532 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f457c0f5410 msgr2=0x7f457c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:59:36.045 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:36.043+0000 7f45808a6700 1 --2- 172.21.15.86:0/1411810532 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f457c0f5410 0x7f457c0f5830 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f4564009fb0 tx=0x7f4564009b30).stop 2024-02-19T21:59:36.046 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:36.043+0000 7f45808a6700 1 -- 172.21.15.86:0/1411810532 shutdown_connections 2024-02-19T21:59:36.046 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:36.044+0000 7f45808a6700 1 --2- 172.21.15.86:0/1411810532 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f457c0f5410 0x7f457c0f5830 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:59:36.046 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:36.044+0000 7f45808a6700 1 -- 172.21.15.86:0/1411810532 >> 172.21.15.86:0/1411810532 conn(0x7f457c0f0b70 msgr2=0x7f457c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:59:36.046 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:36.044+0000 7f45808a6700 1 -- 172.21.15.86:0/1411810532 shutdown_connections 2024-02-19T21:59:36.046 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:36.044+0000 7f45808a6700 1 -- 172.21.15.86:0/1411810532 wait complete. 2024-02-19T21:59:36.046 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:36.044+0000 7f45808a6700 1 Processor -- start 2024-02-19T21:59:36.046 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:36.044+0000 7f45808a6700 1 -- start start 2024-02-19T21:59:36.046 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:36.044+0000 7f45808a6700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f457c0f5410 0x7f457c0fd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:59:36.046 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:36.044+0000 7f45808a6700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f457c0fb960 con 0x7f457c0f5410 2024-02-19T21:59:36.046 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:36.045+0000 7f457a59c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f457c0f5410 0x7f457c0fd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:59:36.046 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:36.045+0000 7f457a59c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f457c0f5410 0x7f457c0fd4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:58802/0 (socket says 172.21.15.86:58802) 2024-02-19T21:59:36.046 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:36.045+0000 7f457a59c700 1 -- 172.21.15.86:0/1381659421 learned_addr learned my addr 172.21.15.86:0/1381659421 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T21:59:36.046 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:36.045+0000 7f457a59c700 1 -- 172.21.15.86:0/1381659421 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4564005040 con 0x7f457c0f5410 2024-02-19T21:59:36.046 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:36.045+0000 7f457a59c700 1 --2- 172.21.15.86:0/1381659421 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f457c0f5410 0x7f457c0fd4e0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f4564005010 tx=0x7f456400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:59:36.047 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:36.045+0000 7f4572ffd700 1 -- 172.21.15.86:0/1381659421 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4564009320 con 0x7f457c0f5410 2024-02-19T21:59:36.047 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:36.045+0000 7f45808a6700 1 -- 172.21.15.86:0/1381659421 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f457c0fbb00 con 0x7f457c0f5410 2024-02-19T21:59:36.047 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:36.045+0000 7f4572ffd700 1 -- 172.21.15.86:0/1381659421 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4564013380 con 0x7f457c0f5410 2024-02-19T21:59:36.047 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:36.045+0000 7f4572ffd700 1 -- 172.21.15.86:0/1381659421 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4564014e50 con 0x7f457c0f5410 2024-02-19T21:59:36.047 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:36.045+0000 7f45808a6700 1 -- 172.21.15.86:0/1381659421 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f457c0fbf90 con 0x7f457c0f5410 2024-02-19T21:59:36.048 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:36.046+0000 7f45808a6700 1 -- 172.21.15.86:0/1381659421 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f457c059100 con 0x7f457c0f5410 2024-02-19T21:59:36.048 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:36.046+0000 7f4572ffd700 1 -- 172.21.15.86:0/1381659421 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f45640095f0 con 0x7f457c0f5410 2024-02-19T21:59:36.048 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:36.046+0000 7f4572ffd700 1 --2- 172.21.15.86:0/1381659421 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f456802ff10 0x7f45680323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:59:36.048 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:36.046+0000 7f4572ffd700 1 -- 172.21.15.86:0/1381659421 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4564040e70 con 0x7f457c0f5410 2024-02-19T21:59:36.048 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:36.046+0000 7f4579d9b700 1 --2- 172.21.15.86:0/1381659421 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f456802ff10 0x7f45680323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:59:36.048 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:36.047+0000 7f4579d9b700 1 --2- 172.21.15.86:0/1381659421 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f456802ff10 0x7f45680323d0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f456c0096a0 tx=0x7f456c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:59:36.050 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:36.051+0000 7f4572ffd700 1 -- 172.21.15.86:0/1381659421 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4564014420 con 0x7f457c0f5410 2024-02-19T21:59:36.254 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:36.254+0000 7f45808a6700 1 -- 172.21.15.86:0/1381659421 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f457c045ad0 con 0x7f457c0f5410 2024-02-19T21:59:36.255 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:36.255+0000 7f4572ffd700 1 -- 172.21.15.86:0/1381659421 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4564010230 con 0x7f457c0f5410 2024-02-19T21:59:36.255 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T21:59:36.255 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T21:59:36.257 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:36.257+0000 7f45808a6700 1 -- 172.21.15.86:0/1381659421 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f456802ff10 msgr2=0x7f45680323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:59:36.257 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:36.257+0000 7f45808a6700 1 --2- 172.21.15.86:0/1381659421 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f456802ff10 0x7f45680323d0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f456c0096a0 tx=0x7f456c006b40).stop 2024-02-19T21:59:36.257 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:36.257+0000 7f45808a6700 1 -- 172.21.15.86:0/1381659421 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f457c0f5410 msgr2=0x7f457c0fd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:59:36.257 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:36.257+0000 7f45808a6700 1 --2- 172.21.15.86:0/1381659421 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f457c0f5410 0x7f457c0fd4e0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f4564005010 tx=0x7f456400b040).stop 2024-02-19T21:59:36.257 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:36.257+0000 7f45808a6700 1 -- 172.21.15.86:0/1381659421 shutdown_connections 2024-02-19T21:59:36.257 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:36.257+0000 7f45808a6700 1 --2- 172.21.15.86:0/1381659421 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f456802ff10 0x7f45680323d0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:59:36.257 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:36.257+0000 7f45808a6700 1 --2- 172.21.15.86:0/1381659421 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f457c0f5410 0x7f457c0fd4e0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:59:36.257 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:36.257+0000 7f45808a6700 1 -- 172.21.15.86:0/1381659421 >> 172.21.15.86:0/1381659421 conn(0x7f457c0f0b70 msgr2=0x7f457c0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:59:36.257 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:36.257+0000 7f45808a6700 1 -- 172.21.15.86:0/1381659421 shutdown_connections 2024-02-19T21:59:36.257 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:36.257+0000 7f45808a6700 1 -- 172.21.15.86:0/1381659421 wait complete. 2024-02-19T21:59:36.258 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T21:59:36.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:59:36 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T21:59:35.462854+0000 mgr.smithi053.fyhcdi (mgr.14162) 91 : cluster [DBG] pgmap v84: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:59:36.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:59:36 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:59:35.599001+0000 mon.smithi053 (mon.0) 159 : audit [INF] from='mgr.14162 172.21.15.53:0/2860358250' entity='mgr.smithi053.fyhcdi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi053.fyhcdi/mirror_snapshot_schedule"}]: dispatch 2024-02-19T21:59:36.794 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:59:36 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:59:35.602630+0000 mon.smithi053 (mon.0) 160 : audit [INF] from='mgr.14162 172.21.15.53:0/2860358250' entity='mgr.smithi053.fyhcdi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi053.fyhcdi/trash_purge_schedule"}]: dispatch 2024-02-19T21:59:36.794 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:59:36 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:59:36.256268+0000 mon.smithi053 (mon.0) 161 : audit [DBG] from='client.? 172.21.15.86:0/1381659421' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T21:59:37.900 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T21:59:37.900 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T21:59:38.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:59:38 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T21:59:37.463231+0000 mgr.smithi053.fyhcdi (mgr.14162) 92 : cluster [DBG] pgmap v85: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:59:39.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:39.551+0000 7f5e3b59e700 1 -- 172.21.15.86:0/3465462371 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5e3c0f62a0 msgr2=0x7f5e3c0f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:59:39.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:39.551+0000 7f5e3b59e700 1 --2- 172.21.15.86:0/3465462371 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5e3c0f62a0 0x7f5e3c0f66c0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f5e30009fb0 tx=0x7f5e30009b30).stop 2024-02-19T21:59:39.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:39.551+0000 7f5e3b59e700 1 -- 172.21.15.86:0/3465462371 shutdown_connections 2024-02-19T21:59:39.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:39.551+0000 7f5e3b59e700 1 --2- 172.21.15.86:0/3465462371 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5e3c0f62a0 0x7f5e3c0f66c0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:59:39.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:39.551+0000 7f5e3b59e700 1 -- 172.21.15.86:0/3465462371 >> 172.21.15.86:0/3465462371 conn(0x7f5e3c0f1aa0 msgr2=0x7f5e3c0f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:59:39.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:39.551+0000 7f5e3b59e700 1 -- 172.21.15.86:0/3465462371 shutdown_connections 2024-02-19T21:59:39.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:39.551+0000 7f5e3b59e700 1 -- 172.21.15.86:0/3465462371 wait complete. 2024-02-19T21:59:39.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:39.552+0000 7f5e3b59e700 1 Processor -- start 2024-02-19T21:59:39.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:39.552+0000 7f5e3b59e700 1 -- start start 2024-02-19T21:59:39.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:39.552+0000 7f5e3b59e700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5e3c0f62a0 0x7f5e3c1087e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:59:39.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:39.552+0000 7f5e3b59e700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e3c108cd0 con 0x7f5e3c0f62a0 2024-02-19T21:59:39.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:39.552+0000 7f5e3a59c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5e3c0f62a0 0x7f5e3c1087e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:59:39.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:39.552+0000 7f5e3a59c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5e3c0f62a0 0x7f5e3c1087e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:50640/0 (socket says 172.21.15.86:50640) 2024-02-19T21:59:39.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:39.552+0000 7f5e3a59c700 1 -- 172.21.15.86:0/750279678 learned_addr learned my addr 172.21.15.86:0/750279678 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T21:59:39.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:39.552+0000 7f5e3a59c700 1 -- 172.21.15.86:0/750279678 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e30005040 con 0x7f5e3c0f62a0 2024-02-19T21:59:39.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:39.552+0000 7f5e3a59c700 1 --2- 172.21.15.86:0/750279678 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5e3c0f62a0 0x7f5e3c1087e0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f5e30009830 tx=0x7f5e3000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:59:39.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:39.553+0000 7f5e2affd700 1 -- 172.21.15.86:0/750279678 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5e30009320 con 0x7f5e3c0f62a0 2024-02-19T21:59:39.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:39.553+0000 7f5e3b59e700 1 -- 172.21.15.86:0/750279678 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e3c108ed0 con 0x7f5e3c0f62a0 2024-02-19T21:59:39.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:39.553+0000 7f5e2affd700 1 -- 172.21.15.86:0/750279678 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5e30013380 con 0x7f5e3c0f62a0 2024-02-19T21:59:39.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:39.553+0000 7f5e2affd700 1 -- 172.21.15.86:0/750279678 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5e30014420 con 0x7f5e3c0f62a0 2024-02-19T21:59:39.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:39.553+0000 7f5e3b59e700 1 -- 172.21.15.86:0/750279678 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e3c109360 con 0x7f5e3c0f62a0 2024-02-19T21:59:39.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:39.554+0000 7f5e2affd700 1 -- 172.21.15.86:0/750279678 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5e3001e420 con 0x7f5e3c0f62a0 2024-02-19T21:59:39.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:39.554+0000 7f5e3b59e700 1 -- 172.21.15.86:0/750279678 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e3c045ad0 con 0x7f5e3c0f62a0 2024-02-19T21:59:39.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:39.554+0000 7f5e2affd700 1 --2- 172.21.15.86:0/750279678 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f5e2402ff60 0x7f5e24032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:59:39.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:39.554+0000 7f5e2affd700 1 -- 172.21.15.86:0/750279678 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5e30040e20 con 0x7f5e3c0f62a0 2024-02-19T21:59:39.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:39.554+0000 7f5e39d9b700 1 --2- 172.21.15.86:0/750279678 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f5e2402ff60 0x7f5e24032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:59:39.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:39.554+0000 7f5e39d9b700 1 --2- 172.21.15.86:0/750279678 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f5e2402ff60 0x7f5e24032420 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f5e2c0096a0 tx=0x7f5e2c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:59:39.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:39.558+0000 7f5e2affd700 1 -- 172.21.15.86:0/750279678 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5e3001c930 con 0x7f5e3c0f62a0 2024-02-19T21:59:39.764 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:39.764+0000 7f5e3b59e700 1 -- 172.21.15.86:0/750279678 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5e3c0fbe30 con 0x7f5e3c0f62a0 2024-02-19T21:59:39.764 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:39.764+0000 7f5e2affd700 1 -- 172.21.15.86:0/750279678 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5e3002c2d0 con 0x7f5e3c0f62a0 2024-02-19T21:59:39.764 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T21:59:39.764 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T21:59:39.766 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:39.766+0000 7f5e3b59e700 1 -- 172.21.15.86:0/750279678 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f5e2402ff60 msgr2=0x7f5e24032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:59:39.766 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:39.766+0000 7f5e3b59e700 1 --2- 172.21.15.86:0/750279678 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f5e2402ff60 0x7f5e24032420 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f5e2c0096a0 tx=0x7f5e2c006b40).stop 2024-02-19T21:59:39.766 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:39.766+0000 7f5e3b59e700 1 -- 172.21.15.86:0/750279678 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5e3c0f62a0 msgr2=0x7f5e3c1087e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:59:39.766 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:39.766+0000 7f5e3b59e700 1 --2- 172.21.15.86:0/750279678 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5e3c0f62a0 0x7f5e3c1087e0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f5e30009830 tx=0x7f5e3000b040).stop 2024-02-19T21:59:39.766 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:39.766+0000 7f5e3b59e700 1 -- 172.21.15.86:0/750279678 shutdown_connections 2024-02-19T21:59:39.766 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:39.766+0000 7f5e3b59e700 1 --2- 172.21.15.86:0/750279678 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f5e2402ff60 0x7f5e24032420 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:59:39.766 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:39.766+0000 7f5e3b59e700 1 --2- 172.21.15.86:0/750279678 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5e3c0f62a0 0x7f5e3c1087e0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:59:39.766 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:39.766+0000 7f5e3b59e700 1 -- 172.21.15.86:0/750279678 >> 172.21.15.86:0/750279678 conn(0x7f5e3c0f1aa0 msgr2=0x7f5e3c0f2460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:59:39.766 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:39.766+0000 7f5e3b59e700 1 -- 172.21.15.86:0/750279678 shutdown_connections 2024-02-19T21:59:39.766 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:39.766+0000 7f5e3b59e700 1 -- 172.21.15.86:0/750279678 wait complete. 2024-02-19T21:59:39.767 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T21:59:40.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:59:40 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T21:59:39.463607+0000 mgr.smithi053.fyhcdi (mgr.14162) 93 : cluster [DBG] pgmap v86: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:59:40.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:59:40 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:59:39.765717+0000 mon.smithi053 (mon.0) 162 : audit [DBG] from='client.? 172.21.15.86:0/750279678' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T21:59:41.471 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T21:59:41.472 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T21:59:42.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:59:42 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T21:59:41.464005+0000 mgr.smithi053.fyhcdi (mgr.14162) 94 : cluster [DBG] pgmap v87: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:59:43.124 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:43.121+0000 7fde0381c700 1 -- 172.21.15.86:0/1903428399 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fddfc0f5410 msgr2=0x7fddfc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:59:43.125 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:43.121+0000 7fde0381c700 1 --2- 172.21.15.86:0/1903428399 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fddfc0f5410 0x7fddfc0f5830 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7fddec009fb0 tx=0x7fddec009b30).stop 2024-02-19T21:59:43.125 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:43.121+0000 7fde0381c700 1 -- 172.21.15.86:0/1903428399 shutdown_connections 2024-02-19T21:59:43.125 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:43.121+0000 7fde0381c700 1 --2- 172.21.15.86:0/1903428399 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fddfc0f5410 0x7fddfc0f5830 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:59:43.125 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:43.121+0000 7fde0381c700 1 -- 172.21.15.86:0/1903428399 >> 172.21.15.86:0/1903428399 conn(0x7fddfc0f0b70 msgr2=0x7fddfc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:59:43.125 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:43.121+0000 7fde0381c700 1 -- 172.21.15.86:0/1903428399 shutdown_connections 2024-02-19T21:59:43.125 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:43.121+0000 7fde0381c700 1 -- 172.21.15.86:0/1903428399 wait complete. 2024-02-19T21:59:43.126 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:43.121+0000 7fde0381c700 1 Processor -- start 2024-02-19T21:59:43.126 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:43.121+0000 7fde0381c700 1 -- start start 2024-02-19T21:59:43.126 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:43.121+0000 7fde0381c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fddfc0f5410 0x7fddfc1067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:59:43.126 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:43.121+0000 7fde0381c700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddfc106cd0 con 0x7fddfc0f5410 2024-02-19T21:59:43.126 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:43.122+0000 7fde015b8700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fddfc0f5410 0x7fddfc1067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:59:43.126 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:43.122+0000 7fde015b8700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fddfc0f5410 0x7fddfc1067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:50662/0 (socket says 172.21.15.86:50662) 2024-02-19T21:59:43.126 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:43.122+0000 7fde015b8700 1 -- 172.21.15.86:0/619241599 learned_addr learned my addr 172.21.15.86:0/619241599 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T21:59:43.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:43.122+0000 7fde015b8700 1 -- 172.21.15.86:0/619241599 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddec005040 con 0x7fddfc0f5410 2024-02-19T21:59:43.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:43.122+0000 7fde015b8700 1 --2- 172.21.15.86:0/619241599 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fddfc0f5410 0x7fddfc1067e0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7fddec005010 tx=0x7fddec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:59:43.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:43.122+0000 7fddf1ffb700 1 -- 172.21.15.86:0/619241599 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fddec009320 con 0x7fddfc0f5410 2024-02-19T21:59:43.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:43.122+0000 7fde0381c700 1 -- 172.21.15.86:0/619241599 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddfc106ed0 con 0x7fddfc0f5410 2024-02-19T21:59:43.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:43.122+0000 7fddf1ffb700 1 -- 172.21.15.86:0/619241599 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fddec013380 con 0x7fddfc0f5410 2024-02-19T21:59:43.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:43.122+0000 7fddf1ffb700 1 -- 172.21.15.86:0/619241599 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fddec014e50 con 0x7fddfc0f5410 2024-02-19T21:59:43.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:43.122+0000 7fde0381c700 1 -- 172.21.15.86:0/619241599 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fddfc107360 con 0x7fddfc0f5410 2024-02-19T21:59:43.128 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:43.123+0000 7fde0381c700 1 -- 172.21.15.86:0/619241599 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fddfc045ad0 con 0x7fddfc0f5410 2024-02-19T21:59:43.128 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:43.123+0000 7fddf1ffb700 1 -- 172.21.15.86:0/619241599 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fddec0095f0 con 0x7fddfc0f5410 2024-02-19T21:59:43.128 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:43.123+0000 7fddf1ffb700 1 --2- 172.21.15.86:0/619241599 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fdde802ff10 0x7fdde80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:59:43.128 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:43.123+0000 7fddf1ffb700 1 -- 172.21.15.86:0/619241599 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fddec040e90 con 0x7fddfc0f5410 2024-02-19T21:59:43.128 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:43.124+0000 7fde00db7700 1 --2- 172.21.15.86:0/619241599 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fdde802ff10 0x7fdde80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:59:43.128 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:43.124+0000 7fde00db7700 1 --2- 172.21.15.86:0/619241599 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fdde802ff10 0x7fdde80323d0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7fddf80096a0 tx=0x7fddf8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:59:43.129 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:43.128+0000 7fddf1ffb700 1 -- 172.21.15.86:0/619241599 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fddec014420 con 0x7fddfc0f5410 2024-02-19T21:59:43.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:43.332+0000 7fde0381c700 1 -- 172.21.15.86:0/619241599 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fddfc0fafa0 con 0x7fddfc0f5410 2024-02-19T21:59:43.333 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:43.333+0000 7fddf1ffb700 1 -- 172.21.15.86:0/619241599 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fddec010230 con 0x7fddfc0f5410 2024-02-19T21:59:43.334 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T21:59:43.334 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T21:59:43.335 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:43.334+0000 7fde0381c700 1 -- 172.21.15.86:0/619241599 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fdde802ff10 msgr2=0x7fdde80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:59:43.336 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:43.334+0000 7fde0381c700 1 --2- 172.21.15.86:0/619241599 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fdde802ff10 0x7fdde80323d0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7fddf80096a0 tx=0x7fddf8006b40).stop 2024-02-19T21:59:43.336 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:43.334+0000 7fde0381c700 1 -- 172.21.15.86:0/619241599 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fddfc0f5410 msgr2=0x7fddfc1067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:59:43.336 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:43.334+0000 7fde0381c700 1 --2- 172.21.15.86:0/619241599 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fddfc0f5410 0x7fddfc1067e0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7fddec005010 tx=0x7fddec00b040).stop 2024-02-19T21:59:43.336 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:43.335+0000 7fde0381c700 1 -- 172.21.15.86:0/619241599 shutdown_connections 2024-02-19T21:59:43.336 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:43.335+0000 7fde0381c700 1 --2- 172.21.15.86:0/619241599 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fdde802ff10 0x7fdde80323d0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:59:43.336 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:43.335+0000 7fde0381c700 1 --2- 172.21.15.86:0/619241599 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fddfc0f5410 0x7fddfc1067e0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:59:43.337 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:43.335+0000 7fde0381c700 1 -- 172.21.15.86:0/619241599 >> 172.21.15.86:0/619241599 conn(0x7fddfc0f0b70 msgr2=0x7fddfc05fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:59:43.337 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:43.335+0000 7fde0381c700 1 -- 172.21.15.86:0/619241599 shutdown_connections 2024-02-19T21:59:43.337 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:43.335+0000 7fde0381c700 1 -- 172.21.15.86:0/619241599 wait complete. 2024-02-19T21:59:43.337 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T21:59:43.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:59:43 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:59:43.334320+0000 mon.smithi053 (mon.0) 163 : audit [DBG] from='client.? 172.21.15.86:0/619241599' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T21:59:44.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:59:44 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T21:59:43.464246+0000 mgr.smithi053.fyhcdi (mgr.14162) 95 : cluster [DBG] pgmap v88: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:59:44.967 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T21:59:44.968 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T21:59:46.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:59:46 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T21:59:45.464589+0000 mgr.smithi053.fyhcdi (mgr.14162) 96 : cluster [DBG] pgmap v89: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:59:46.873 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:46.872+0000 7f3888f9c700 1 -- 172.21.15.86:0/349598168 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f38840f5410 msgr2=0x7f38840f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:59:46.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:46.872+0000 7f3888f9c700 1 --2- 172.21.15.86:0/349598168 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f38840f5410 0x7f38840f5830 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7f3878009fb0 tx=0x7f3878009b30).stop 2024-02-19T21:59:46.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:46.872+0000 7f3888f9c700 1 -- 172.21.15.86:0/349598168 shutdown_connections 2024-02-19T21:59:46.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:46.872+0000 7f3888f9c700 1 --2- 172.21.15.86:0/349598168 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f38840f5410 0x7f38840f5830 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:59:46.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:46.872+0000 7f3888f9c700 1 -- 172.21.15.86:0/349598168 >> 172.21.15.86:0/349598168 conn(0x7f38840f0b70 msgr2=0x7f38840f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:59:46.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:46.872+0000 7f3888f9c700 1 -- 172.21.15.86:0/349598168 shutdown_connections 2024-02-19T21:59:46.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:46.872+0000 7f3888f9c700 1 -- 172.21.15.86:0/349598168 wait complete. 2024-02-19T21:59:46.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:46.872+0000 7f3888f9c700 1 Processor -- start 2024-02-19T21:59:46.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:46.873+0000 7f3888f9c700 1 -- start start 2024-02-19T21:59:46.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:46.873+0000 7f3888f9c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f38840f5410 0x7f3884105710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:59:46.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:46.873+0000 7f3888f9c700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3884105c00 con 0x7f38840f5410 2024-02-19T21:59:46.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:46.873+0000 7f388259c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f38840f5410 0x7f3884105710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:59:46.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:46.873+0000 7f388259c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f38840f5410 0x7f3884105710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:50686/0 (socket says 172.21.15.86:50686) 2024-02-19T21:59:46.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:46.873+0000 7f388259c700 1 -- 172.21.15.86:0/3896351669 learned_addr learned my addr 172.21.15.86:0/3896351669 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T21:59:46.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:46.873+0000 7f388259c700 1 -- 172.21.15.86:0/3896351669 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3878005040 con 0x7f38840f5410 2024-02-19T21:59:46.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:46.873+0000 7f388259c700 1 --2- 172.21.15.86:0/3896351669 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f38840f5410 0x7f3884105710 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f3878005010 tx=0x7f387800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:59:46.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:46.873+0000 7f3872ffd700 1 -- 172.21.15.86:0/3896351669 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3878009320 con 0x7f38840f5410 2024-02-19T21:59:46.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:46.874+0000 7f3888f9c700 1 -- 172.21.15.86:0/3896351669 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3884105e00 con 0x7f38840f5410 2024-02-19T21:59:46.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:46.874+0000 7f3872ffd700 1 -- 172.21.15.86:0/3896351669 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3878013380 con 0x7f38840f5410 2024-02-19T21:59:46.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:46.874+0000 7f3872ffd700 1 -- 172.21.15.86:0/3896351669 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3878014e50 con 0x7f38840f5410 2024-02-19T21:59:46.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:46.874+0000 7f3888f9c700 1 -- 172.21.15.86:0/3896351669 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3884106290 con 0x7f38840f5410 2024-02-19T21:59:46.876 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:46.874+0000 7f3872ffd700 1 -- 172.21.15.86:0/3896351669 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f38780095f0 con 0x7f38840f5410 2024-02-19T21:59:46.876 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:46.874+0000 7f3888f9c700 1 -- 172.21.15.86:0/3896351669 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3884045ad0 con 0x7f38840f5410 2024-02-19T21:59:46.876 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:46.874+0000 7f3872ffd700 1 --2- 172.21.15.86:0/3896351669 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f386c02ff10 0x7f386c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:59:46.876 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:46.875+0000 7f3872ffd700 1 -- 172.21.15.86:0/3896351669 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3878040ef0 con 0x7f38840f5410 2024-02-19T21:59:46.876 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:46.875+0000 7f3881d9b700 1 --2- 172.21.15.86:0/3896351669 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f386c02ff10 0x7f386c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:59:46.876 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:46.875+0000 7f3881d9b700 1 --2- 172.21.15.86:0/3896351669 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f386c02ff10 0x7f386c0323d0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7f38740096a0 tx=0x7f3874006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:59:46.879 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:46.879+0000 7f3872ffd700 1 -- 172.21.15.86:0/3896351669 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3878014420 con 0x7f38840f5410 2024-02-19T21:59:47.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:47.083+0000 7f3888f9c700 1 -- 172.21.15.86:0/3896351669 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3884107140 con 0x7f38840f5410 2024-02-19T21:59:47.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:47.083+0000 7f3872ffd700 1 -- 172.21.15.86:0/3896351669 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3878010230 con 0x7f38840f5410 2024-02-19T21:59:47.083 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T21:59:47.084 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T21:59:47.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:47.085+0000 7f3888f9c700 1 -- 172.21.15.86:0/3896351669 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f386c02ff10 msgr2=0x7f386c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:59:47.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:47.085+0000 7f3888f9c700 1 --2- 172.21.15.86:0/3896351669 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f386c02ff10 0x7f386c0323d0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7f38740096a0 tx=0x7f3874006b40).stop 2024-02-19T21:59:47.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:47.085+0000 7f3888f9c700 1 -- 172.21.15.86:0/3896351669 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f38840f5410 msgr2=0x7f3884105710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:59:47.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:47.085+0000 7f3888f9c700 1 --2- 172.21.15.86:0/3896351669 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f38840f5410 0x7f3884105710 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f3878005010 tx=0x7f387800b040).stop 2024-02-19T21:59:47.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:47.085+0000 7f3888f9c700 1 -- 172.21.15.86:0/3896351669 shutdown_connections 2024-02-19T21:59:47.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:47.085+0000 7f3888f9c700 1 --2- 172.21.15.86:0/3896351669 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f386c02ff10 0x7f386c0323d0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:59:47.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:47.085+0000 7f3888f9c700 1 --2- 172.21.15.86:0/3896351669 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f38840f5410 0x7f3884105710 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:59:47.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:47.085+0000 7f3888f9c700 1 -- 172.21.15.86:0/3896351669 >> 172.21.15.86:0/3896351669 conn(0x7f38840f0b70 msgr2=0x7f38840f15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:59:47.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:47.085+0000 7f3888f9c700 1 -- 172.21.15.86:0/3896351669 shutdown_connections 2024-02-19T21:59:47.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:47.085+0000 7f3888f9c700 1 -- 172.21.15.86:0/3896351669 wait complete. 2024-02-19T21:59:47.086 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T21:59:47.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:59:47 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:59:47.084651+0000 mon.smithi053 (mon.0) 164 : audit [DBG] from='client.? 172.21.15.86:0/3896351669' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T21:59:48.712 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T21:59:48.712 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T21:59:48.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:59:48 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T21:59:47.464903+0000 mgr.smithi053.fyhcdi (mgr.14162) 97 : cluster [DBG] pgmap v90: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:59:50.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:50.681+0000 7f3d8fb45700 1 -- 172.21.15.86:0/3771625465 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3d880f5410 msgr2=0x7f3d880f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:59:50.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:50.681+0000 7f3d8fb45700 1 --2- 172.21.15.86:0/3771625465 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3d880f5410 0x7f3d880f5830 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7f3d78009fb0 tx=0x7f3d78009b30).stop 2024-02-19T21:59:50.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:50.681+0000 7f3d8fb45700 1 -- 172.21.15.86:0/3771625465 shutdown_connections 2024-02-19T21:59:50.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:50.681+0000 7f3d8fb45700 1 --2- 172.21.15.86:0/3771625465 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3d880f5410 0x7f3d880f5830 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:59:50.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:50.681+0000 7f3d8fb45700 1 -- 172.21.15.86:0/3771625465 >> 172.21.15.86:0/3771625465 conn(0x7f3d880f0b70 msgr2=0x7f3d880f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:59:50.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:50.681+0000 7f3d8fb45700 1 -- 172.21.15.86:0/3771625465 shutdown_connections 2024-02-19T21:59:50.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:50.681+0000 7f3d8fb45700 1 -- 172.21.15.86:0/3771625465 wait complete. 2024-02-19T21:59:50.684 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:50.681+0000 7f3d8fb45700 1 Processor -- start 2024-02-19T21:59:50.684 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:50.682+0000 7f3d8fb45700 1 -- start start 2024-02-19T21:59:50.684 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:50.682+0000 7f3d8fb45700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3d880f5410 0x7f3d88105720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:59:50.684 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:50.682+0000 7f3d8fb45700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d88105c10 con 0x7f3d880f5410 2024-02-19T21:59:50.684 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:50.682+0000 7f3d8d8e1700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3d880f5410 0x7f3d88105720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:59:50.684 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:50.682+0000 7f3d8d8e1700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3d880f5410 0x7f3d88105720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:34460/0 (socket says 172.21.15.86:34460) 2024-02-19T21:59:50.684 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:50.682+0000 7f3d8d8e1700 1 -- 172.21.15.86:0/3035470396 learned_addr learned my addr 172.21.15.86:0/3035470396 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T21:59:50.684 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:50.682+0000 7f3d8d8e1700 1 -- 172.21.15.86:0/3035470396 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d78005040 con 0x7f3d880f5410 2024-02-19T21:59:50.684 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:50.683+0000 7f3d8d8e1700 1 --2- 172.21.15.86:0/3035470396 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3d880f5410 0x7f3d88105720 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f3d780089f0 tx=0x7f3d7800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:59:50.684 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:50.683+0000 7f3d7e7fc700 1 -- 172.21.15.86:0/3035470396 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3d78009320 con 0x7f3d880f5410 2024-02-19T21:59:50.685 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:50.683+0000 7f3d7e7fc700 1 -- 172.21.15.86:0/3035470396 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3d78013380 con 0x7f3d880f5410 2024-02-19T21:59:50.685 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:50.683+0000 7f3d7e7fc700 1 -- 172.21.15.86:0/3035470396 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3d7801ed80 con 0x7f3d880f5410 2024-02-19T21:59:50.685 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:50.683+0000 7f3d8fb45700 1 -- 172.21.15.86:0/3035470396 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d88105e10 con 0x7f3d880f5410 2024-02-19T21:59:50.685 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:50.683+0000 7f3d8fb45700 1 -- 172.21.15.86:0/3035470396 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d881062a0 con 0x7f3d880f5410 2024-02-19T21:59:50.685 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:50.684+0000 7f3d7e7fc700 1 -- 172.21.15.86:0/3035470396 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3d7801e420 con 0x7f3d880f5410 2024-02-19T21:59:50.685 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:50.684+0000 7f3d8fb45700 1 -- 172.21.15.86:0/3035470396 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d88045ad0 con 0x7f3d880f5410 2024-02-19T21:59:50.685 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:50.684+0000 7f3d7e7fc700 1 --2- 172.21.15.86:0/3035470396 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f3d7402ff60 0x7f3d74032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:59:50.685 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:50.684+0000 7f3d7e7fc700 1 -- 172.21.15.86:0/3035470396 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3d78040fa0 con 0x7f3d880f5410 2024-02-19T21:59:50.685 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:50.684+0000 7f3d8d0e0700 1 --2- 172.21.15.86:0/3035470396 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f3d7402ff60 0x7f3d74032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:59:50.685 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:50.684+0000 7f3d8d0e0700 1 --2- 172.21.15.86:0/3035470396 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f3d7402ff60 0x7f3d74032420 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7f3d840096a0 tx=0x7f3d84006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:59:50.688 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:50.688+0000 7f3d7e7fc700 1 -- 172.21.15.86:0/3035470396 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3d780095f0 con 0x7f3d880f5410 2024-02-19T21:59:50.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:59:50 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T21:59:49.465190+0000 mgr.smithi053.fyhcdi (mgr.14162) 98 : cluster [DBG] pgmap v91: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:59:50.893 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:50.893+0000 7f3d8fb45700 1 -- 172.21.15.86:0/3035470396 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3d88107150 con 0x7f3d880f5410 2024-02-19T21:59:50.893 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:50.893+0000 7f3d7e7fc700 1 -- 172.21.15.86:0/3035470396 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3d7801cb10 con 0x7f3d880f5410 2024-02-19T21:59:50.893 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T21:59:50.894 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T21:59:50.895 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:50.895+0000 7f3d8fb45700 1 -- 172.21.15.86:0/3035470396 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f3d7402ff60 msgr2=0x7f3d74032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:59:50.895 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:50.895+0000 7f3d8fb45700 1 --2- 172.21.15.86:0/3035470396 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f3d7402ff60 0x7f3d74032420 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7f3d840096a0 tx=0x7f3d84006b40).stop 2024-02-19T21:59:50.895 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:50.895+0000 7f3d8fb45700 1 -- 172.21.15.86:0/3035470396 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3d880f5410 msgr2=0x7f3d88105720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:59:50.895 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:50.895+0000 7f3d8fb45700 1 --2- 172.21.15.86:0/3035470396 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3d880f5410 0x7f3d88105720 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f3d780089f0 tx=0x7f3d7800b040).stop 2024-02-19T21:59:50.895 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:50.895+0000 7f3d8fb45700 1 -- 172.21.15.86:0/3035470396 shutdown_connections 2024-02-19T21:59:50.895 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:50.895+0000 7f3d8fb45700 1 --2- 172.21.15.86:0/3035470396 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f3d7402ff60 0x7f3d74032420 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:59:50.895 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:50.895+0000 7f3d8fb45700 1 --2- 172.21.15.86:0/3035470396 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3d880f5410 0x7f3d88105720 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:59:50.895 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:50.895+0000 7f3d8fb45700 1 -- 172.21.15.86:0/3035470396 >> 172.21.15.86:0/3035470396 conn(0x7f3d880f0b70 msgr2=0x7f3d880f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:59:50.895 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:50.895+0000 7f3d8fb45700 1 -- 172.21.15.86:0/3035470396 shutdown_connections 2024-02-19T21:59:50.896 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:50.895+0000 7f3d8fb45700 1 -- 172.21.15.86:0/3035470396 wait complete. 2024-02-19T21:59:50.896 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T21:59:51.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:59:51 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:59:50.894533+0000 mon.smithi053 (mon.0) 165 : audit [DBG] from='client.? 172.21.15.86:0/3035470396' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T21:59:52.561 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T21:59:52.561 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T21:59:52.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:59:52 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T21:59:51.465422+0000 mgr.smithi053.fyhcdi (mgr.14162) 99 : cluster [DBG] pgmap v92: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:59:54.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:54.329+0000 7f5ee7145700 1 -- 172.21.15.86:0/3153059667 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5ee00f6510 msgr2=0x7f5ee00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:59:54.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:54.329+0000 7f5ee7145700 1 --2- 172.21.15.86:0/3153059667 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5ee00f6510 0x7f5ee00f6930 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f5ed4009fb0 tx=0x7f5ed4009b30).stop 2024-02-19T21:59:54.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:54.330+0000 7f5ee7145700 1 -- 172.21.15.86:0/3153059667 shutdown_connections 2024-02-19T21:59:54.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:54.330+0000 7f5ee7145700 1 --2- 172.21.15.86:0/3153059667 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5ee00f6510 0x7f5ee00f6930 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:59:54.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:54.330+0000 7f5ee7145700 1 -- 172.21.15.86:0/3153059667 >> 172.21.15.86:0/3153059667 conn(0x7f5ee00f1c70 msgr2=0x7f5ee00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:59:54.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:54.330+0000 7f5ee7145700 1 -- 172.21.15.86:0/3153059667 shutdown_connections 2024-02-19T21:59:54.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:54.330+0000 7f5ee7145700 1 -- 172.21.15.86:0/3153059667 wait complete. 2024-02-19T21:59:54.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:54.330+0000 7f5ee7145700 1 Processor -- start 2024-02-19T21:59:54.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:54.330+0000 7f5ee7145700 1 -- start start 2024-02-19T21:59:54.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:54.330+0000 7f5ee7145700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5ee00f6510 0x7f5ee01067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:59:54.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:54.330+0000 7f5ee7145700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ee0106cd0 con 0x7f5ee00f6510 2024-02-19T21:59:54.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:54.331+0000 7f5ee4ee1700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5ee00f6510 0x7f5ee01067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:59:54.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:54.331+0000 7f5ee4ee1700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5ee00f6510 0x7f5ee01067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:34482/0 (socket says 172.21.15.86:34482) 2024-02-19T21:59:54.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:54.331+0000 7f5ee4ee1700 1 -- 172.21.15.86:0/2779835813 learned_addr learned my addr 172.21.15.86:0/2779835813 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T21:59:54.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:54.331+0000 7f5ee4ee1700 1 -- 172.21.15.86:0/2779835813 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ed4005040 con 0x7f5ee00f6510 2024-02-19T21:59:54.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:54.331+0000 7f5ee4ee1700 1 --2- 172.21.15.86:0/2779835813 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5ee00f6510 0x7f5ee01067e0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f5ed4005010 tx=0x7f5ed400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:59:54.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:54.331+0000 7f5edd7fa700 1 -- 172.21.15.86:0/2779835813 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5ed4009320 con 0x7f5ee00f6510 2024-02-19T21:59:54.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:54.331+0000 7f5edd7fa700 1 -- 172.21.15.86:0/2779835813 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5ed4013380 con 0x7f5ee00f6510 2024-02-19T21:59:54.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:54.331+0000 7f5ee7145700 1 -- 172.21.15.86:0/2779835813 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ee0106ed0 con 0x7f5ee00f6510 2024-02-19T21:59:54.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:54.331+0000 7f5edd7fa700 1 -- 172.21.15.86:0/2779835813 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5ed4014e50 con 0x7f5ee00f6510 2024-02-19T21:59:54.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:54.331+0000 7f5ee7145700 1 -- 172.21.15.86:0/2779835813 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ee0107360 con 0x7f5ee00f6510 2024-02-19T21:59:54.333 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:54.332+0000 7f5edd7fa700 1 -- 172.21.15.86:0/2779835813 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5ed40095f0 con 0x7f5ee00f6510 2024-02-19T21:59:54.333 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:54.332+0000 7f5ee7145700 1 -- 172.21.15.86:0/2779835813 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ee0045ad0 con 0x7f5ee00f6510 2024-02-19T21:59:54.333 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:54.332+0000 7f5edd7fa700 1 --2- 172.21.15.86:0/2779835813 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f5ed002ff10 0x7f5ed00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:59:54.333 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:54.332+0000 7f5edd7fa700 1 -- 172.21.15.86:0/2779835813 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5ed4040fa0 con 0x7f5ee00f6510 2024-02-19T21:59:54.333 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:54.332+0000 7f5edffff700 1 --2- 172.21.15.86:0/2779835813 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f5ed002ff10 0x7f5ed00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:59:54.334 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:54.333+0000 7f5edffff700 1 --2- 172.21.15.86:0/2779835813 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f5ed002ff10 0x7f5ed00323d0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7f5ecc0096a0 tx=0x7f5ecc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:59:54.336 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:54.337+0000 7f5edd7fa700 1 -- 172.21.15.86:0/2779835813 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5ed4014420 con 0x7f5ee00f6510 2024-02-19T21:59:54.540 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:54.540+0000 7f5ee7145700 1 -- 172.21.15.86:0/2779835813 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5ee0108210 con 0x7f5ee00f6510 2024-02-19T21:59:54.541 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:54.541+0000 7f5edd7fa700 1 -- 172.21.15.86:0/2779835813 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5ed4010230 con 0x7f5ee00f6510 2024-02-19T21:59:54.541 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T21:59:54.541 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T21:59:54.543 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:54.542+0000 7f5ee7145700 1 -- 172.21.15.86:0/2779835813 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f5ed002ff10 msgr2=0x7f5ed00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:59:54.543 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:54.542+0000 7f5ee7145700 1 --2- 172.21.15.86:0/2779835813 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f5ed002ff10 0x7f5ed00323d0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7f5ecc0096a0 tx=0x7f5ecc006b40).stop 2024-02-19T21:59:54.543 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:54.542+0000 7f5ee7145700 1 -- 172.21.15.86:0/2779835813 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5ee00f6510 msgr2=0x7f5ee01067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:59:54.543 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:54.542+0000 7f5ee7145700 1 --2- 172.21.15.86:0/2779835813 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5ee00f6510 0x7f5ee01067e0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f5ed4005010 tx=0x7f5ed400b040).stop 2024-02-19T21:59:54.543 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:54.542+0000 7f5ee7145700 1 -- 172.21.15.86:0/2779835813 shutdown_connections 2024-02-19T21:59:54.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:54.542+0000 7f5ee7145700 1 --2- 172.21.15.86:0/2779835813 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f5ed002ff10 0x7f5ed00323d0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:59:54.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:54.542+0000 7f5ee7145700 1 --2- 172.21.15.86:0/2779835813 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5ee00f6510 0x7f5ee01067e0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:59:54.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:54.542+0000 7f5ee7145700 1 -- 172.21.15.86:0/2779835813 >> 172.21.15.86:0/2779835813 conn(0x7f5ee00f1c70 msgr2=0x7f5ee00f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:59:54.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:54.542+0000 7f5ee7145700 1 -- 172.21.15.86:0/2779835813 shutdown_connections 2024-02-19T21:59:54.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:54.542+0000 7f5ee7145700 1 -- 172.21.15.86:0/2779835813 wait complete. 2024-02-19T21:59:54.544 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T21:59:54.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:59:54 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T21:59:53.465676+0000 mgr.smithi053.fyhcdi (mgr.14162) 100 : cluster [DBG] pgmap v93: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:59:55.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:59:55 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:59:54.541992+0000 mon.smithi053 (mon.0) 166 : audit [DBG] from='client.? 172.21.15.86:0/2779835813' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T21:59:56.270 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T21:59:56.270 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T21:59:56.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:59:56 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T21:59:55.466073+0000 mgr.smithi053.fyhcdi (mgr.14162) 101 : cluster [DBG] pgmap v94: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:59:58.152 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:58.143+0000 7f7abe14d700 1 -- 172.21.15.86:0/3130570370 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7ab80f62a0 msgr2=0x7f7ab80f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:59:58.153 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:58.143+0000 7f7abe14d700 1 --2- 172.21.15.86:0/3130570370 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7ab80f62a0 0x7f7ab80f66c0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7f7aa8009fb0 tx=0x7f7aa8009b30).stop 2024-02-19T21:59:58.153 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:58.143+0000 7f7abe14d700 1 -- 172.21.15.86:0/3130570370 shutdown_connections 2024-02-19T21:59:58.153 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:58.143+0000 7f7abe14d700 1 --2- 172.21.15.86:0/3130570370 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7ab80f62a0 0x7f7ab80f66c0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:59:58.153 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:58.143+0000 7f7abe14d700 1 -- 172.21.15.86:0/3130570370 >> 172.21.15.86:0/3130570370 conn(0x7f7ab80f1ac0 msgr2=0x7f7ab80f3ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:59:58.153 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:58.143+0000 7f7abe14d700 1 -- 172.21.15.86:0/3130570370 shutdown_connections 2024-02-19T21:59:58.153 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:58.143+0000 7f7abe14d700 1 -- 172.21.15.86:0/3130570370 wait complete. 2024-02-19T21:59:58.153 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:58.144+0000 7f7abe14d700 1 Processor -- start 2024-02-19T21:59:58.153 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:58.144+0000 7f7abe14d700 1 -- start start 2024-02-19T21:59:58.153 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:58.144+0000 7f7abe14d700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7ab80f62a0 0x7f7ab8106570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:59:58.153 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:58.144+0000 7f7abe14d700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ab8106a60 con 0x7f7ab80f62a0 2024-02-19T21:59:58.153 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:58.144+0000 7f7abd14b700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7ab80f62a0 0x7f7ab8106570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:59:58.153 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:58.144+0000 7f7abd14b700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7ab80f62a0 0x7f7ab8106570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:34500/0 (socket says 172.21.15.86:34500) 2024-02-19T21:59:58.153 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:58.144+0000 7f7abd14b700 1 -- 172.21.15.86:0/71364762 learned_addr learned my addr 172.21.15.86:0/71364762 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T21:59:58.154 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:58.144+0000 7f7abd14b700 1 -- 172.21.15.86:0/71364762 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7aa8005040 con 0x7f7ab80f62a0 2024-02-19T21:59:58.154 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:58.144+0000 7f7abd14b700 1 --2- 172.21.15.86:0/71364762 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7ab80f62a0 0x7f7ab8106570 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f7aa8009830 tx=0x7f7aa800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:59:58.154 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:58.145+0000 7f7aadffb700 1 -- 172.21.15.86:0/71364762 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7aa8009320 con 0x7f7ab80f62a0 2024-02-19T21:59:58.154 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:58.145+0000 7f7abe14d700 1 -- 172.21.15.86:0/71364762 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ab8106c60 con 0x7f7ab80f62a0 2024-02-19T21:59:58.154 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:58.145+0000 7f7aadffb700 1 -- 172.21.15.86:0/71364762 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7aa8013380 con 0x7f7ab80f62a0 2024-02-19T21:59:58.154 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:58.145+0000 7f7aadffb700 1 -- 172.21.15.86:0/71364762 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7aa8014420 con 0x7f7ab80f62a0 2024-02-19T21:59:58.154 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:58.145+0000 7f7abe14d700 1 -- 172.21.15.86:0/71364762 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ab81070f0 con 0x7f7ab80f62a0 2024-02-19T21:59:58.154 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:58.146+0000 7f7abe14d700 1 -- 172.21.15.86:0/71364762 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ab8045ad0 con 0x7f7ab80f62a0 2024-02-19T21:59:58.154 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:58.146+0000 7f7aadffb700 1 -- 172.21.15.86:0/71364762 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7aa801e420 con 0x7f7ab80f62a0 2024-02-19T21:59:58.154 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:58.146+0000 7f7aadffb700 1 --2- 172.21.15.86:0/71364762 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f7aa402ff60 0x7f7aa4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T21:59:58.154 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:58.146+0000 7f7aadffb700 1 -- 172.21.15.86:0/71364762 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7aa8040e30 con 0x7f7ab80f62a0 2024-02-19T21:59:58.154 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:58.146+0000 7f7abc94a700 1 --2- 172.21.15.86:0/71364762 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f7aa402ff60 0x7f7aa4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T21:59:58.154 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:58.146+0000 7f7abc94a700 1 --2- 172.21.15.86:0/71364762 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f7aa402ff60 0x7f7aa4032420 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7f7ab40096a0 tx=0x7f7ab4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T21:59:58.154 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:58.151+0000 7f7aadffb700 1 -- 172.21.15.86:0/71364762 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7aa801cb10 con 0x7f7ab80f62a0 2024-02-19T21:59:58.355 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:58.356+0000 7f7abe14d700 1 -- 172.21.15.86:0/71364762 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7ab80fbe30 con 0x7f7ab80f62a0 2024-02-19T21:59:58.356 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:58.356+0000 7f7aadffb700 1 -- 172.21.15.86:0/71364762 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7aa8004490 con 0x7f7ab80f62a0 2024-02-19T21:59:58.356 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T21:59:58.356 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T21:59:58.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:58.358+0000 7f7abe14d700 1 -- 172.21.15.86:0/71364762 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f7aa402ff60 msgr2=0x7f7aa4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:59:58.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:58.358+0000 7f7abe14d700 1 --2- 172.21.15.86:0/71364762 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f7aa402ff60 0x7f7aa4032420 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7f7ab40096a0 tx=0x7f7ab4006b40).stop 2024-02-19T21:59:58.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:58.358+0000 7f7abe14d700 1 -- 172.21.15.86:0/71364762 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7ab80f62a0 msgr2=0x7f7ab8106570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:59:58.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:58.358+0000 7f7abe14d700 1 --2- 172.21.15.86:0/71364762 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7ab80f62a0 0x7f7ab8106570 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f7aa8009830 tx=0x7f7aa800b040).stop 2024-02-19T21:59:58.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:58.358+0000 7f7abe14d700 1 -- 172.21.15.86:0/71364762 shutdown_connections 2024-02-19T21:59:58.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:58.358+0000 7f7abe14d700 1 --2- 172.21.15.86:0/71364762 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f7aa402ff60 0x7f7aa4032420 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:59:58.359 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:58.358+0000 7f7abe14d700 1 --2- 172.21.15.86:0/71364762 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7ab80f62a0 0x7f7ab8106570 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T21:59:58.359 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:58.358+0000 7f7abe14d700 1 -- 172.21.15.86:0/71364762 >> 172.21.15.86:0/71364762 conn(0x7f7ab80f1ac0 msgr2=0x7f7ab80f24b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:59:58.359 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:58.358+0000 7f7abe14d700 1 -- 172.21.15.86:0/71364762 shutdown_connections 2024-02-19T21:59:58.359 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T21:59:58.358+0000 7f7abe14d700 1 -- 172.21.15.86:0/71364762 wait complete. 2024-02-19T21:59:58.359 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T21:59:58.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:59:58 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T21:59:57.466395+0000 mgr.smithi053.fyhcdi (mgr.14162) 102 : cluster [DBG] pgmap v95: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T21:59:58.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 21:59:58 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T21:59:58.357685+0000 mon.smithi053 (mon.0) 167 : audit [DBG] from='client.? 172.21.15.86:0/71364762' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:00:00.071 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:00:00.071 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:00:00.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:00:00 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T21:59:59.466773+0000 mgr.smithi053.fyhcdi (mgr.14162) 103 : cluster [DBG] pgmap v96: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:00:01.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:01.863+0000 7f512493e700 1 -- 172.21.15.86:0/464662004 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f51200f6510 msgr2=0x7f51200f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:00:01.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:01.863+0000 7f512493e700 1 --2- 172.21.15.86:0/464662004 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f51200f6510 0x7f51200f6930 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7f5114009fb0 tx=0x7f5114009b30).stop 2024-02-19T22:00:01.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:01.863+0000 7f512493e700 1 -- 172.21.15.86:0/464662004 shutdown_connections 2024-02-19T22:00:01.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:01.863+0000 7f512493e700 1 --2- 172.21.15.86:0/464662004 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f51200f6510 0x7f51200f6930 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:00:01.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:01.863+0000 7f512493e700 1 -- 172.21.15.86:0/464662004 >> 172.21.15.86:0/464662004 conn(0x7f51200f1c70 msgr2=0x7f51200f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:00:01.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:01.863+0000 7f512493e700 1 -- 172.21.15.86:0/464662004 shutdown_connections 2024-02-19T22:00:01.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:01.863+0000 7f512493e700 1 -- 172.21.15.86:0/464662004 wait complete. 2024-02-19T22:00:01.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:01.864+0000 7f512493e700 1 Processor -- start 2024-02-19T22:00:01.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:01.864+0000 7f512493e700 1 -- start start 2024-02-19T22:00:01.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:01.864+0000 7f512493e700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f51200f6510 0x7f512006b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:00:01.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:01.864+0000 7f512493e700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5120069e20 con 0x7f51200f6510 2024-02-19T22:00:01.865 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:01.864+0000 7f511e59c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f51200f6510 0x7f512006b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:00:01.865 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:01.864+0000 7f511e59c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f51200f6510 0x7f512006b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:43666/0 (socket says 172.21.15.86:43666) 2024-02-19T22:00:01.865 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:01.864+0000 7f511e59c700 1 -- 172.21.15.86:0/3196753819 learned_addr learned my addr 172.21.15.86:0/3196753819 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T22:00:01.865 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:01.864+0000 7f511e59c700 1 -- 172.21.15.86:0/3196753819 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5114005040 con 0x7f51200f6510 2024-02-19T22:00:01.865 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:01.865+0000 7f511e59c700 1 --2- 172.21.15.86:0/3196753819 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f51200f6510 0x7f512006b9a0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f5114005010 tx=0x7f511400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:00:01.865 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:01.865+0000 7f510effd700 1 -- 172.21.15.86:0/3196753819 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5114009320 con 0x7f51200f6510 2024-02-19T22:00:01.865 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:01.865+0000 7f512493e700 1 -- 172.21.15.86:0/3196753819 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5120069fc0 con 0x7f51200f6510 2024-02-19T22:00:01.865 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:01.865+0000 7f510effd700 1 -- 172.21.15.86:0/3196753819 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5114013380 con 0x7f51200f6510 2024-02-19T22:00:01.865 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:01.865+0000 7f510effd700 1 -- 172.21.15.86:0/3196753819 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5114014e50 con 0x7f51200f6510 2024-02-19T22:00:01.865 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:01.865+0000 7f512493e700 1 -- 172.21.15.86:0/3196753819 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f512006a450 con 0x7f51200f6510 2024-02-19T22:00:01.866 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:01.866+0000 7f510effd700 1 -- 172.21.15.86:0/3196753819 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f51140263a0 con 0x7f51200f6510 2024-02-19T22:00:01.866 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:01.866+0000 7f510effd700 1 --2- 172.21.15.86:0/3196753819 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f510802ff10 0x7f51080323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:00:01.866 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:01.866+0000 7f512493e700 1 -- 172.21.15.86:0/3196753819 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5120059100 con 0x7f51200f6510 2024-02-19T22:00:01.866 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:01.866+0000 7f510effd700 1 -- 172.21.15.86:0/3196753819 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5114040fa0 con 0x7f51200f6510 2024-02-19T22:00:01.866 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:01.866+0000 7f511dd9b700 1 --2- 172.21.15.86:0/3196753819 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f510802ff10 0x7f51080323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:00:01.866 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:01.866+0000 7f511dd9b700 1 --2- 172.21.15.86:0/3196753819 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f510802ff10 0x7f51080323d0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7f51100096a0 tx=0x7f5110006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:00:01.869 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:01.871+0000 7f510effd700 1 -- 172.21.15.86:0/3196753819 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5114012920 con 0x7f51200f6510 2024-02-19T22:00:02.075 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:02.076+0000 7f512493e700 1 -- 172.21.15.86:0/3196753819 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f51200fc0a0 con 0x7f51200f6510 2024-02-19T22:00:02.075 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:02.076+0000 7f510effd700 1 -- 172.21.15.86:0/3196753819 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f511400f340 con 0x7f51200f6510 2024-02-19T22:00:02.075 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T22:00:02.076 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:00:02.077 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:02.078+0000 7f512493e700 1 -- 172.21.15.86:0/3196753819 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f510802ff10 msgr2=0x7f51080323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:00:02.077 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:02.078+0000 7f512493e700 1 --2- 172.21.15.86:0/3196753819 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f510802ff10 0x7f51080323d0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7f51100096a0 tx=0x7f5110006b40).stop 2024-02-19T22:00:02.077 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:02.078+0000 7f512493e700 1 -- 172.21.15.86:0/3196753819 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f51200f6510 msgr2=0x7f512006b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:00:02.077 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:02.078+0000 7f512493e700 1 --2- 172.21.15.86:0/3196753819 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f51200f6510 0x7f512006b9a0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f5114005010 tx=0x7f511400b040).stop 2024-02-19T22:00:02.077 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:02.078+0000 7f512493e700 1 -- 172.21.15.86:0/3196753819 shutdown_connections 2024-02-19T22:00:02.077 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:02.078+0000 7f512493e700 1 --2- 172.21.15.86:0/3196753819 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f510802ff10 0x7f51080323d0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:00:02.077 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:02.078+0000 7f512493e700 1 --2- 172.21.15.86:0/3196753819 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f51200f6510 0x7f512006b9a0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:00:02.077 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:02.078+0000 7f512493e700 1 -- 172.21.15.86:0/3196753819 >> 172.21.15.86:0/3196753819 conn(0x7f51200f1c70 msgr2=0x7f51200f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:00:02.077 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:02.078+0000 7f512493e700 1 -- 172.21.15.86:0/3196753819 shutdown_connections 2024-02-19T22:00:02.078 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:02.078+0000 7f512493e700 1 -- 172.21.15.86:0/3196753819 wait complete. 2024-02-19T22:00:02.078 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T22:00:02.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:00:02 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:00:01.467108+0000 mgr.smithi053.fyhcdi (mgr.14162) 104 : cluster [DBG] pgmap v97: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:00:02.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:00:02 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:00:02.076825+0000 mon.smithi053 (mon.0) 168 : audit [DBG] from='client.? 172.21.15.86:0/3196753819' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:00:03.772 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:00:03.773 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:00:04.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:00:04 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:00:03.467508+0000 mgr.smithi053.fyhcdi (mgr.14162) 105 : cluster [DBG] pgmap v98: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:00:05.645 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:05.646+0000 7fce62f28700 1 -- 172.21.15.86:0/52192794 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fce5c0f6510 msgr2=0x7fce5c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:00:05.645 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:05.646+0000 7fce62f28700 1 --2- 172.21.15.86:0/52192794 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fce5c0f6510 0x7fce5c0f6930 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7fce50009fb0 tx=0x7fce50009b30).stop 2024-02-19T22:00:05.645 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:05.646+0000 7fce62f28700 1 -- 172.21.15.86:0/52192794 shutdown_connections 2024-02-19T22:00:05.646 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:05.646+0000 7fce62f28700 1 --2- 172.21.15.86:0/52192794 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fce5c0f6510 0x7fce5c0f6930 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:00:05.646 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:05.646+0000 7fce62f28700 1 -- 172.21.15.86:0/52192794 >> 172.21.15.86:0/52192794 conn(0x7fce5c0f1c70 msgr2=0x7fce5c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:00:05.646 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:05.646+0000 7fce62f28700 1 -- 172.21.15.86:0/52192794 shutdown_connections 2024-02-19T22:00:05.646 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:05.646+0000 7fce62f28700 1 -- 172.21.15.86:0/52192794 wait complete. 2024-02-19T22:00:05.646 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:05.646+0000 7fce62f28700 1 Processor -- start 2024-02-19T22:00:05.646 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:05.647+0000 7fce62f28700 1 -- start start 2024-02-19T22:00:05.647 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:05.647+0000 7fce62f28700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fce5c0f6510 0x7fce5c06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:00:05.648 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:05.647+0000 7fce62f28700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce5c069e20 con 0x7fce5c0f6510 2024-02-19T22:00:05.648 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:05.647+0000 7fce60cc4700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fce5c0f6510 0x7fce5c06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:00:05.648 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:05.647+0000 7fce60cc4700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fce5c0f6510 0x7fce5c06b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:43686/0 (socket says 172.21.15.86:43686) 2024-02-19T22:00:05.648 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:05.647+0000 7fce60cc4700 1 -- 172.21.15.86:0/2370306141 learned_addr learned my addr 172.21.15.86:0/2370306141 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T22:00:05.648 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:05.647+0000 7fce60cc4700 1 -- 172.21.15.86:0/2370306141 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce50005040 con 0x7fce5c0f6510 2024-02-19T22:00:05.648 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:05.648+0000 7fce60cc4700 1 --2- 172.21.15.86:0/2370306141 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fce5c0f6510 0x7fce5c06b9a0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7fce50005010 tx=0x7fce5000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:00:05.648 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:05.648+0000 7fce597fa700 1 -- 172.21.15.86:0/2370306141 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fce50009320 con 0x7fce5c0f6510 2024-02-19T22:00:05.648 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:05.648+0000 7fce597fa700 1 -- 172.21.15.86:0/2370306141 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fce50013380 con 0x7fce5c0f6510 2024-02-19T22:00:05.648 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:05.648+0000 7fce62f28700 1 -- 172.21.15.86:0/2370306141 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce5c069fc0 con 0x7fce5c0f6510 2024-02-19T22:00:05.648 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:05.648+0000 7fce597fa700 1 -- 172.21.15.86:0/2370306141 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fce50014e50 con 0x7fce5c0f6510 2024-02-19T22:00:05.648 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:05.648+0000 7fce62f28700 1 -- 172.21.15.86:0/2370306141 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce5c06a450 con 0x7fce5c0f6510 2024-02-19T22:00:05.648 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:05.649+0000 7fce597fa700 1 -- 172.21.15.86:0/2370306141 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fce500095f0 con 0x7fce5c0f6510 2024-02-19T22:00:05.648 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:05.649+0000 7fce62f28700 1 -- 172.21.15.86:0/2370306141 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce5c059100 con 0x7fce5c0f6510 2024-02-19T22:00:05.648 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:05.649+0000 7fce597fa700 1 --2- 172.21.15.86:0/2370306141 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fce4c02ff10 0x7fce4c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:00:05.649 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:05.649+0000 7fce597fa700 1 -- 172.21.15.86:0/2370306141 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fce50040e90 con 0x7fce5c0f6510 2024-02-19T22:00:05.649 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:05.649+0000 7fce5bfff700 1 --2- 172.21.15.86:0/2370306141 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fce4c02ff10 0x7fce4c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:00:05.649 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:05.649+0000 7fce5bfff700 1 --2- 172.21.15.86:0/2370306141 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fce4c02ff10 0x7fce4c0323d0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7fce480096a0 tx=0x7fce48006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:00:05.652 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:05.653+0000 7fce597fa700 1 -- 172.21.15.86:0/2370306141 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fce50014420 con 0x7fce5c0f6510 2024-02-19T22:00:05.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:05.860+0000 7fce62f28700 1 -- 172.21.15.86:0/2370306141 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fce5c0fc0a0 con 0x7fce5c0f6510 2024-02-19T22:00:05.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:05.861+0000 7fce597fa700 1 -- 172.21.15.86:0/2370306141 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fce50010230 con 0x7fce5c0f6510 2024-02-19T22:00:05.860 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T22:00:05.860 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:00:05.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:05.862+0000 7fce62f28700 1 -- 172.21.15.86:0/2370306141 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fce4c02ff10 msgr2=0x7fce4c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:00:05.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:05.862+0000 7fce62f28700 1 --2- 172.21.15.86:0/2370306141 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fce4c02ff10 0x7fce4c0323d0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7fce480096a0 tx=0x7fce48006b40).stop 2024-02-19T22:00:05.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:05.862+0000 7fce62f28700 1 -- 172.21.15.86:0/2370306141 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fce5c0f6510 msgr2=0x7fce5c06b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:00:05.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:05.862+0000 7fce62f28700 1 --2- 172.21.15.86:0/2370306141 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fce5c0f6510 0x7fce5c06b9a0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7fce50005010 tx=0x7fce5000b040).stop 2024-02-19T22:00:05.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:05.862+0000 7fce62f28700 1 -- 172.21.15.86:0/2370306141 shutdown_connections 2024-02-19T22:00:05.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:05.862+0000 7fce62f28700 1 --2- 172.21.15.86:0/2370306141 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fce4c02ff10 0x7fce4c0323d0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:00:05.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:05.862+0000 7fce62f28700 1 --2- 172.21.15.86:0/2370306141 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fce5c0f6510 0x7fce5c06b9a0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:00:05.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:05.862+0000 7fce62f28700 1 -- 172.21.15.86:0/2370306141 >> 172.21.15.86:0/2370306141 conn(0x7fce5c0f1c70 msgr2=0x7fce5c0f26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:00:05.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:05.863+0000 7fce62f28700 1 -- 172.21.15.86:0/2370306141 shutdown_connections 2024-02-19T22:00:05.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:05.863+0000 7fce62f28700 1 -- 172.21.15.86:0/2370306141 wait complete. 2024-02-19T22:00:05.862 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T22:00:06.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:00:06 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:00:05.467845+0000 mgr.smithi053.fyhcdi (mgr.14162) 106 : cluster [DBG] pgmap v99: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:00:06.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:00:06 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:00:05.861257+0000 mon.smithi053 (mon.0) 169 : audit [DBG] from='client.? 172.21.15.86:0/2370306141' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:00:07.491 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:00:07.492 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:00:08.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:00:08 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:00:07.468201+0000 mgr.smithi053.fyhcdi (mgr.14162) 107 : cluster [DBG] pgmap v100: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:00:09.309 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:09.309+0000 7f5618cbd700 1 -- 172.21.15.86:0/1416568468 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f56140f5410 msgr2=0x7f56140f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:00:09.309 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:09.309+0000 7f5618cbd700 1 --2- 172.21.15.86:0/1416568468 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f56140f5410 0x7f56140f5830 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7f5608009fb0 tx=0x7f5608009b30).stop 2024-02-19T22:00:09.310 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:09.309+0000 7f5618cbd700 1 -- 172.21.15.86:0/1416568468 shutdown_connections 2024-02-19T22:00:09.310 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:09.309+0000 7f5618cbd700 1 --2- 172.21.15.86:0/1416568468 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f56140f5410 0x7f56140f5830 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:00:09.310 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:09.309+0000 7f5618cbd700 1 -- 172.21.15.86:0/1416568468 >> 172.21.15.86:0/1416568468 conn(0x7f56140f0b70 msgr2=0x7f56140f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:00:09.310 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:09.309+0000 7f5618cbd700 1 -- 172.21.15.86:0/1416568468 shutdown_connections 2024-02-19T22:00:09.310 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:09.309+0000 7f5618cbd700 1 -- 172.21.15.86:0/1416568468 wait complete. 2024-02-19T22:00:09.312 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:09.310+0000 7f5618cbd700 1 Processor -- start 2024-02-19T22:00:09.312 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:09.310+0000 7f5618cbd700 1 -- start start 2024-02-19T22:00:09.312 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:09.310+0000 7f5618cbd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f56140f5410 0x7f56141056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:00:09.312 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:09.310+0000 7f5618cbd700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5614105bd0 con 0x7f56140f5410 2024-02-19T22:00:09.312 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:09.310+0000 7f561259c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f56140f5410 0x7f56141056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:00:09.312 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:09.310+0000 7f561259c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f56140f5410 0x7f56141056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:39514/0 (socket says 172.21.15.86:39514) 2024-02-19T22:00:09.312 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:09.310+0000 7f561259c700 1 -- 172.21.15.86:0/3976270909 learned_addr learned my addr 172.21.15.86:0/3976270909 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T22:00:09.312 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:09.311+0000 7f561259c700 1 -- 172.21.15.86:0/3976270909 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5608005040 con 0x7f56140f5410 2024-02-19T22:00:09.312 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:09.311+0000 7f561259c700 1 --2- 172.21.15.86:0/3976270909 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f56140f5410 0x7f56141056e0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f5608005010 tx=0x7f560800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:00:09.312 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:09.311+0000 7f5602ffd700 1 -- 172.21.15.86:0/3976270909 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5608009320 con 0x7f56140f5410 2024-02-19T22:00:09.313 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:09.311+0000 7f5618cbd700 1 -- 172.21.15.86:0/3976270909 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5614105dd0 con 0x7f56140f5410 2024-02-19T22:00:09.313 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:09.311+0000 7f5602ffd700 1 -- 172.21.15.86:0/3976270909 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5608013380 con 0x7f56140f5410 2024-02-19T22:00:09.313 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:09.311+0000 7f5602ffd700 1 -- 172.21.15.86:0/3976270909 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5608014e50 con 0x7f56140f5410 2024-02-19T22:00:09.313 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:09.311+0000 7f5618cbd700 1 -- 172.21.15.86:0/3976270909 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5614106260 con 0x7f56140f5410 2024-02-19T22:00:09.313 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:09.312+0000 7f5602ffd700 1 -- 172.21.15.86:0/3976270909 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f56080095f0 con 0x7f56140f5410 2024-02-19T22:00:09.313 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:09.312+0000 7f5618cbd700 1 -- 172.21.15.86:0/3976270909 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5614045ad0 con 0x7f56140f5410 2024-02-19T22:00:09.313 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:09.312+0000 7f5602ffd700 1 --2- 172.21.15.86:0/3976270909 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f55fc02ff10 0x7f55fc0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:00:09.313 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:09.312+0000 7f5602ffd700 1 -- 172.21.15.86:0/3976270909 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5608040fa0 con 0x7f56140f5410 2024-02-19T22:00:09.313 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:09.312+0000 7f5611d9b700 1 --2- 172.21.15.86:0/3976270909 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f55fc02ff10 0x7f55fc0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:00:09.314 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:09.313+0000 7f5611d9b700 1 --2- 172.21.15.86:0/3976270909 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f55fc02ff10 0x7f55fc0323d0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f56040096a0 tx=0x7f5604006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:00:09.316 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:09.317+0000 7f5602ffd700 1 -- 172.21.15.86:0/3976270909 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5608014420 con 0x7f56140f5410 2024-02-19T22:00:09.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:09.522+0000 7f5618cbd700 1 -- 172.21.15.86:0/3976270909 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5614107110 con 0x7f56140f5410 2024-02-19T22:00:09.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:09.523+0000 7f5602ffd700 1 -- 172.21.15.86:0/3976270909 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5608010230 con 0x7f56140f5410 2024-02-19T22:00:09.524 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T22:00:09.525 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:00:09.525 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:09.524+0000 7f5618cbd700 1 -- 172.21.15.86:0/3976270909 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f55fc02ff10 msgr2=0x7f55fc0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:00:09.526 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:09.524+0000 7f5618cbd700 1 --2- 172.21.15.86:0/3976270909 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f55fc02ff10 0x7f55fc0323d0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f56040096a0 tx=0x7f5604006b40).stop 2024-02-19T22:00:09.526 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:09.524+0000 7f5618cbd700 1 -- 172.21.15.86:0/3976270909 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f56140f5410 msgr2=0x7f56141056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:00:09.526 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:09.524+0000 7f5618cbd700 1 --2- 172.21.15.86:0/3976270909 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f56140f5410 0x7f56141056e0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f5608005010 tx=0x7f560800b040).stop 2024-02-19T22:00:09.526 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:09.524+0000 7f5618cbd700 1 -- 172.21.15.86:0/3976270909 shutdown_connections 2024-02-19T22:00:09.526 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:09.524+0000 7f5618cbd700 1 --2- 172.21.15.86:0/3976270909 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f55fc02ff10 0x7f55fc0323d0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:00:09.526 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:09.524+0000 7f5618cbd700 1 --2- 172.21.15.86:0/3976270909 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f56140f5410 0x7f56141056e0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:00:09.526 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:09.524+0000 7f5618cbd700 1 -- 172.21.15.86:0/3976270909 >> 172.21.15.86:0/3976270909 conn(0x7f56140f0b70 msgr2=0x7f56140f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:00:09.526 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:09.524+0000 7f5618cbd700 1 -- 172.21.15.86:0/3976270909 shutdown_connections 2024-02-19T22:00:09.526 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:09.524+0000 7f5618cbd700 1 -- 172.21.15.86:0/3976270909 wait complete. 2024-02-19T22:00:09.526 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T22:00:10.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:00:10 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:00:09.468552+0000 mgr.smithi053.fyhcdi (mgr.14162) 108 : cluster [DBG] pgmap v101: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:00:10.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:00:10 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:00:09.523910+0000 mon.smithi053 (mon.0) 170 : audit [DBG] from='client.? 172.21.15.86:0/3976270909' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:00:11.199 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:00:11.199 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:00:12.796 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:00:12 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:00:11.468859+0000 mgr.smithi053.fyhcdi (mgr.14162) 109 : cluster [DBG] pgmap v102: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:00:12.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:12.946+0000 7fc9a2fec700 1 -- 172.21.15.86:0/3615733454 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc99c0f5410 msgr2=0x7fc99c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:00:12.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:12.946+0000 7fc9a2fec700 1 --2- 172.21.15.86:0/3615733454 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc99c0f5410 0x7fc99c0f5830 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7fc990009fb0 tx=0x7fc990009b30).stop 2024-02-19T22:00:12.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:12.946+0000 7fc9a2fec700 1 -- 172.21.15.86:0/3615733454 shutdown_connections 2024-02-19T22:00:12.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:12.946+0000 7fc9a2fec700 1 --2- 172.21.15.86:0/3615733454 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc99c0f5410 0x7fc99c0f5830 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:00:12.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:12.946+0000 7fc9a2fec700 1 -- 172.21.15.86:0/3615733454 >> 172.21.15.86:0/3615733454 conn(0x7fc99c0f0b70 msgr2=0x7fc99c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:00:12.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:12.946+0000 7fc9a2fec700 1 -- 172.21.15.86:0/3615733454 shutdown_connections 2024-02-19T22:00:12.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:12.947+0000 7fc9a2fec700 1 -- 172.21.15.86:0/3615733454 wait complete. 2024-02-19T22:00:12.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:12.947+0000 7fc9a2fec700 1 Processor -- start 2024-02-19T22:00:12.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:12.947+0000 7fc9a2fec700 1 -- start start 2024-02-19T22:00:12.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:12.947+0000 7fc9a2fec700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc99c0f5410 0x7fc99c1056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:00:12.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:12.947+0000 7fc9a2fec700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc99c105bd0 con 0x7fc99c0f5410 2024-02-19T22:00:12.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:12.947+0000 7fc9a0d88700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc99c0f5410 0x7fc99c1056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:00:12.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:12.948+0000 7fc9a0d88700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc99c0f5410 0x7fc99c1056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:39536/0 (socket says 172.21.15.86:39536) 2024-02-19T22:00:12.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:12.948+0000 7fc9a0d88700 1 -- 172.21.15.86:0/3132339122 learned_addr learned my addr 172.21.15.86:0/3132339122 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T22:00:12.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:12.948+0000 7fc9a0d88700 1 -- 172.21.15.86:0/3132339122 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc990005040 con 0x7fc99c0f5410 2024-02-19T22:00:12.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:12.948+0000 7fc9a0d88700 1 --2- 172.21.15.86:0/3132339122 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc99c0f5410 0x7fc99c1056e0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7fc9900089f0 tx=0x7fc99000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:00:12.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:12.948+0000 7fc9997fa700 1 -- 172.21.15.86:0/3132339122 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc990009320 con 0x7fc99c0f5410 2024-02-19T22:00:12.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:12.948+0000 7fc9a2fec700 1 -- 172.21.15.86:0/3132339122 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc99c105dd0 con 0x7fc99c0f5410 2024-02-19T22:00:12.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:12.948+0000 7fc9997fa700 1 -- 172.21.15.86:0/3132339122 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc990013380 con 0x7fc99c0f5410 2024-02-19T22:00:12.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:12.948+0000 7fc9997fa700 1 -- 172.21.15.86:0/3132339122 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc99001ed80 con 0x7fc99c0f5410 2024-02-19T22:00:12.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:12.948+0000 7fc9a2fec700 1 -- 172.21.15.86:0/3132339122 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc99c106260 con 0x7fc99c0f5410 2024-02-19T22:00:12.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:12.949+0000 7fc9997fa700 1 -- 172.21.15.86:0/3132339122 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc99001e420 con 0x7fc99c0f5410 2024-02-19T22:00:12.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:12.949+0000 7fc9a2fec700 1 -- 172.21.15.86:0/3132339122 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc99c045ad0 con 0x7fc99c0f5410 2024-02-19T22:00:12.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:12.949+0000 7fc9997fa700 1 --2- 172.21.15.86:0/3132339122 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fc98c02ff60 0x7fc98c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:00:12.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:12.949+0000 7fc9997fa700 1 -- 172.21.15.86:0/3132339122 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc990040fe0 con 0x7fc99c0f5410 2024-02-19T22:00:12.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:12.949+0000 7fc99bfff700 1 --2- 172.21.15.86:0/3132339122 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fc98c02ff60 0x7fc98c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:00:12.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:12.950+0000 7fc99bfff700 1 --2- 172.21.15.86:0/3132339122 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fc98c02ff60 0x7fc98c032420 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7fc9880096a0 tx=0x7fc988006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:00:12.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:12.954+0000 7fc9997fa700 1 -- 172.21.15.86:0/3132339122 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc9900095f0 con 0x7fc99c0f5410 2024-02-19T22:00:13.159 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:13.159+0000 7fc9a2fec700 1 -- 172.21.15.86:0/3132339122 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc99c1070b0 con 0x7fc99c0f5410 2024-02-19T22:00:13.159 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:13.159+0000 7fc9997fa700 1 -- 172.21.15.86:0/3132339122 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc99001cb10 con 0x7fc99c0f5410 2024-02-19T22:00:13.159 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T22:00:13.159 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:00:13.161 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:13.161+0000 7fc9a2fec700 1 -- 172.21.15.86:0/3132339122 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fc98c02ff60 msgr2=0x7fc98c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:00:13.161 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:13.161+0000 7fc9a2fec700 1 --2- 172.21.15.86:0/3132339122 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fc98c02ff60 0x7fc98c032420 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7fc9880096a0 tx=0x7fc988006b40).stop 2024-02-19T22:00:13.161 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:13.161+0000 7fc9a2fec700 1 -- 172.21.15.86:0/3132339122 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc99c0f5410 msgr2=0x7fc99c1056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:00:13.161 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:13.161+0000 7fc9a2fec700 1 --2- 172.21.15.86:0/3132339122 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc99c0f5410 0x7fc99c1056e0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7fc9900089f0 tx=0x7fc99000b040).stop 2024-02-19T22:00:13.161 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:13.161+0000 7fc9a2fec700 1 -- 172.21.15.86:0/3132339122 shutdown_connections 2024-02-19T22:00:13.161 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:13.161+0000 7fc9a2fec700 1 --2- 172.21.15.86:0/3132339122 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fc98c02ff60 0x7fc98c032420 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:00:13.161 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:13.161+0000 7fc9a2fec700 1 --2- 172.21.15.86:0/3132339122 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc99c0f5410 0x7fc99c1056e0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:00:13.162 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:13.161+0000 7fc9a2fec700 1 -- 172.21.15.86:0/3132339122 >> 172.21.15.86:0/3132339122 conn(0x7fc99c0f0b70 msgr2=0x7fc99c0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:00:13.162 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:13.161+0000 7fc9a2fec700 1 -- 172.21.15.86:0/3132339122 shutdown_connections 2024-02-19T22:00:13.162 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:13.161+0000 7fc9a2fec700 1 -- 172.21.15.86:0/3132339122 wait complete. 2024-02-19T22:00:13.162 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T22:00:13.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:00:13 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:00:13.160585+0000 mon.smithi053 (mon.0) 171 : audit [DBG] from='client.? 172.21.15.86:0/3132339122' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:00:14.778 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:00:14.778 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:00:14.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:00:14 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:00:13.469138+0000 mgr.smithi053.fyhcdi (mgr.14162) 110 : cluster [DBG] pgmap v103: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:00:16.583 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:16.582+0000 7f24eb59e700 1 -- 172.21.15.86:0/3265427054 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f24ec0f62a0 msgr2=0x7f24ec0f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:00:16.583 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:16.582+0000 7f24eb59e700 1 --2- 172.21.15.86:0/3265427054 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f24ec0f62a0 0x7f24ec0f66c0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f24d4009fb0 tx=0x7f24d4009b30).stop 2024-02-19T22:00:16.583 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:16.582+0000 7f24eb59e700 1 -- 172.21.15.86:0/3265427054 shutdown_connections 2024-02-19T22:00:16.583 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:16.582+0000 7f24eb59e700 1 --2- 172.21.15.86:0/3265427054 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f24ec0f62a0 0x7f24ec0f66c0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:00:16.583 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:16.582+0000 7f24eb59e700 1 -- 172.21.15.86:0/3265427054 >> 172.21.15.86:0/3265427054 conn(0x7f24ec0f1aa0 msgr2=0x7f24ec0f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:00:16.583 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:16.582+0000 7f24eb59e700 1 -- 172.21.15.86:0/3265427054 shutdown_connections 2024-02-19T22:00:16.584 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:16.582+0000 7f24eb59e700 1 -- 172.21.15.86:0/3265427054 wait complete. 2024-02-19T22:00:16.584 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:16.582+0000 7f24eb59e700 1 Processor -- start 2024-02-19T22:00:16.584 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:16.582+0000 7f24eb59e700 1 -- start start 2024-02-19T22:00:16.584 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:16.582+0000 7f24eb59e700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f24ec0f62a0 0x7f24ec1065d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:00:16.584 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:16.582+0000 7f24eb59e700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24ec106ac0 con 0x7f24ec0f62a0 2024-02-19T22:00:16.584 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:16.583+0000 7f24ea59c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f24ec0f62a0 0x7f24ec1065d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:00:16.584 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:16.583+0000 7f24ea59c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f24ec0f62a0 0x7f24ec1065d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:39546/0 (socket says 172.21.15.86:39546) 2024-02-19T22:00:16.584 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:16.583+0000 7f24ea59c700 1 -- 172.21.15.86:0/1774166855 learned_addr learned my addr 172.21.15.86:0/1774166855 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T22:00:16.584 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:16.583+0000 7f24ea59c700 1 -- 172.21.15.86:0/1774166855 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24d4005040 con 0x7f24ec0f62a0 2024-02-19T22:00:16.584 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:16.583+0000 7f24ea59c700 1 --2- 172.21.15.86:0/1774166855 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f24ec0f62a0 0x7f24ec1065d0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f24d40089f0 tx=0x7f24d400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:00:16.584 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:16.583+0000 7f24e2ffd700 1 -- 172.21.15.86:0/1774166855 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f24d4009320 con 0x7f24ec0f62a0 2024-02-19T22:00:16.584 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:16.583+0000 7f24eb59e700 1 -- 172.21.15.86:0/1774166855 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24ec106cc0 con 0x7f24ec0f62a0 2024-02-19T22:00:16.584 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:16.583+0000 7f24e2ffd700 1 -- 172.21.15.86:0/1774166855 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f24d4013380 con 0x7f24ec0f62a0 2024-02-19T22:00:16.585 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:16.583+0000 7f24e2ffd700 1 -- 172.21.15.86:0/1774166855 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f24d4014e20 con 0x7f24ec0f62a0 2024-02-19T22:00:16.585 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:16.583+0000 7f24eb59e700 1 -- 172.21.15.86:0/1774166855 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24ec107150 con 0x7f24ec0f62a0 2024-02-19T22:00:16.585 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:16.584+0000 7f24e2ffd700 1 -- 172.21.15.86:0/1774166855 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f24d40095f0 con 0x7f24ec0f62a0 2024-02-19T22:00:16.586 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:16.584+0000 7f24eb59e700 1 -- 172.21.15.86:0/1774166855 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24ec045ad0 con 0x7f24ec0f62a0 2024-02-19T22:00:16.586 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:16.584+0000 7f24e2ffd700 1 --2- 172.21.15.86:0/1774166855 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f24d802ff10 0x7f24d80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:00:16.586 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:16.584+0000 7f24e2ffd700 1 -- 172.21.15.86:0/1774166855 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f24d4040fb0 con 0x7f24ec0f62a0 2024-02-19T22:00:16.586 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:16.584+0000 7f24e9d9b700 1 --2- 172.21.15.86:0/1774166855 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f24d802ff10 0x7f24d80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:00:16.586 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:16.585+0000 7f24e9d9b700 1 --2- 172.21.15.86:0/1774166855 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f24d802ff10 0x7f24d80323d0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f24dc0096a0 tx=0x7f24dc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:00:16.589 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:16.589+0000 7f24e2ffd700 1 -- 172.21.15.86:0/1774166855 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f24d406d2b0 con 0x7f24ec0f62a0 2024-02-19T22:00:16.791 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:16.791+0000 7f24eb59e700 1 -- 172.21.15.86:0/1774166855 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f24ec107fa0 con 0x7f24ec0f62a0 2024-02-19T22:00:16.792 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:16.792+0000 7f24e2ffd700 1 -- 172.21.15.86:0/1774166855 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f24d4014420 con 0x7f24ec0f62a0 2024-02-19T22:00:16.792 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T22:00:16.792 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:00:16.793 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:16.793+0000 7f24eb59e700 1 -- 172.21.15.86:0/1774166855 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f24d802ff10 msgr2=0x7f24d80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:00:16.793 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:16.793+0000 7f24eb59e700 1 --2- 172.21.15.86:0/1774166855 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f24d802ff10 0x7f24d80323d0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f24dc0096a0 tx=0x7f24dc006b40).stop 2024-02-19T22:00:16.794 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:16.793+0000 7f24eb59e700 1 -- 172.21.15.86:0/1774166855 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f24ec0f62a0 msgr2=0x7f24ec1065d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:00:16.794 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:16.793+0000 7f24eb59e700 1 --2- 172.21.15.86:0/1774166855 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f24ec0f62a0 0x7f24ec1065d0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f24d40089f0 tx=0x7f24d400b040).stop 2024-02-19T22:00:16.794 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:16.793+0000 7f24eb59e700 1 -- 172.21.15.86:0/1774166855 shutdown_connections 2024-02-19T22:00:16.794 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:16.793+0000 7f24eb59e700 1 --2- 172.21.15.86:0/1774166855 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f24d802ff10 0x7f24d80323d0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:00:16.794 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:16.793+0000 7f24eb59e700 1 --2- 172.21.15.86:0/1774166855 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f24ec0f62a0 0x7f24ec1065d0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:00:16.794 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:16.793+0000 7f24eb59e700 1 -- 172.21.15.86:0/1774166855 >> 172.21.15.86:0/1774166855 conn(0x7f24ec0f1aa0 msgr2=0x7f24ec0f2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:00:16.794 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:16.793+0000 7f24eb59e700 1 -- 172.21.15.86:0/1774166855 shutdown_connections 2024-02-19T22:00:16.794 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:16.794+0000 7f24eb59e700 1 -- 172.21.15.86:0/1774166855 wait complete. 2024-02-19T22:00:16.795 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:00:16 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:00:15.469460+0000 mgr.smithi053.fyhcdi (mgr.14162) 111 : cluster [DBG] pgmap v104: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:00:16.795 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T22:00:17.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:00:17 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:00:16.793270+0000 mon.smithi053 (mon.0) 172 : audit [DBG] from='client.? 172.21.15.86:0/1774166855' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:00:18.711 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:00:18.711 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:00:18.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:00:18 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:00:17.469754+0000 mgr.smithi053.fyhcdi (mgr.14162) 112 : cluster [DBG] pgmap v105: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:00:20.565 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:20.560+0000 7f30a9bfb700 1 -- 172.21.15.86:0/753723842 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f30a40f62c0 msgr2=0x7f30a40f66e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:00:20.566 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:20.560+0000 7f30a9bfb700 1 --2- 172.21.15.86:0/753723842 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f30a40f62c0 0x7f30a40f66e0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f3098009fb0 tx=0x7f3098009b30).stop 2024-02-19T22:00:20.566 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:20.560+0000 7f30a9bfb700 1 -- 172.21.15.86:0/753723842 shutdown_connections 2024-02-19T22:00:20.566 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:20.560+0000 7f30a9bfb700 1 --2- 172.21.15.86:0/753723842 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f30a40f62c0 0x7f30a40f66e0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:00:20.566 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:20.560+0000 7f30a9bfb700 1 -- 172.21.15.86:0/753723842 >> 172.21.15.86:0/753723842 conn(0x7f30a40f1ac0 msgr2=0x7f30a40f3ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:00:20.566 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:20.560+0000 7f30a9bfb700 1 -- 172.21.15.86:0/753723842 shutdown_connections 2024-02-19T22:00:20.566 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:20.560+0000 7f30a9bfb700 1 -- 172.21.15.86:0/753723842 wait complete. 2024-02-19T22:00:20.566 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:20.560+0000 7f30a9bfb700 1 Processor -- start 2024-02-19T22:00:20.567 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:20.560+0000 7f30a9bfb700 1 -- start start 2024-02-19T22:00:20.567 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:20.561+0000 7f30a9bfb700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f30a40f62c0 0x7f30a406b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:00:20.567 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:20.561+0000 7f30a9bfb700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30a4069e20 con 0x7f30a40f62c0 2024-02-19T22:00:20.567 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:20.561+0000 7f30a8bf9700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f30a40f62c0 0x7f30a406b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:00:20.567 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:20.561+0000 7f30a8bf9700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f30a40f62c0 0x7f30a406b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:35406/0 (socket says 172.21.15.86:35406) 2024-02-19T22:00:20.567 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:20.561+0000 7f30a8bf9700 1 -- 172.21.15.86:0/1602782913 learned_addr learned my addr 172.21.15.86:0/1602782913 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T22:00:20.567 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:20.561+0000 7f30a8bf9700 1 -- 172.21.15.86:0/1602782913 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3098005040 con 0x7f30a40f62c0 2024-02-19T22:00:20.567 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:20.561+0000 7f30a8bf9700 1 --2- 172.21.15.86:0/1602782913 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f30a40f62c0 0x7f30a406b9a0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f30980089f0 tx=0x7f309800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:00:20.567 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:20.562+0000 7f30a17fa700 1 -- 172.21.15.86:0/1602782913 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3098009320 con 0x7f30a40f62c0 2024-02-19T22:00:20.567 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:20.562+0000 7f30a9bfb700 1 -- 172.21.15.86:0/1602782913 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30a4069fc0 con 0x7f30a40f62c0 2024-02-19T22:00:20.567 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:20.562+0000 7f30a17fa700 1 -- 172.21.15.86:0/1602782913 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3098013380 con 0x7f30a40f62c0 2024-02-19T22:00:20.567 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:20.562+0000 7f30a17fa700 1 -- 172.21.15.86:0/1602782913 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3098014e20 con 0x7f30a40f62c0 2024-02-19T22:00:20.568 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:20.562+0000 7f30a9bfb700 1 -- 172.21.15.86:0/1602782913 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30a406a450 con 0x7f30a40f62c0 2024-02-19T22:00:20.568 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:20.562+0000 7f30a17fa700 1 -- 172.21.15.86:0/1602782913 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f30980263a0 con 0x7f30a40f62c0 2024-02-19T22:00:20.568 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:20.563+0000 7f30a17fa700 1 --2- 172.21.15.86:0/1602782913 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f309402ff60 0x7f3094032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:00:20.568 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:20.563+0000 7f30a17fa700 1 -- 172.21.15.86:0/1602782913 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3098040f00 con 0x7f30a40f62c0 2024-02-19T22:00:20.568 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:20.563+0000 7f30a9bfb700 1 -- 172.21.15.86:0/1602782913 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30a4059100 con 0x7f30a40f62c0 2024-02-19T22:00:20.568 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:20.563+0000 7f30a3fff700 1 --2- 172.21.15.86:0/1602782913 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f309402ff60 0x7f3094032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:00:20.568 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:20.563+0000 7f30a3fff700 1 --2- 172.21.15.86:0/1602782913 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f309402ff60 0x7f3094032420 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f30900096a0 tx=0x7f3090006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:00:20.568 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:20.567+0000 7f30a17fa700 1 -- 172.21.15.86:0/1602782913 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3098012920 con 0x7f30a40f62c0 2024-02-19T22:00:20.774 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:20.774+0000 7f30a9bfb700 1 -- 172.21.15.86:0/1602782913 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f30a40fbe50 con 0x7f30a40f62c0 2024-02-19T22:00:20.774 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:20.775+0000 7f30a17fa700 1 -- 172.21.15.86:0/1602782913 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f309802b020 con 0x7f30a40f62c0 2024-02-19T22:00:20.774 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T22:00:20.775 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:00:20.776 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:20.776+0000 7f30a9bfb700 1 -- 172.21.15.86:0/1602782913 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f309402ff60 msgr2=0x7f3094032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:00:20.776 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:20.776+0000 7f30a9bfb700 1 --2- 172.21.15.86:0/1602782913 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f309402ff60 0x7f3094032420 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f30900096a0 tx=0x7f3090006b40).stop 2024-02-19T22:00:20.776 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:20.776+0000 7f30a9bfb700 1 -- 172.21.15.86:0/1602782913 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f30a40f62c0 msgr2=0x7f30a406b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:00:20.776 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:20.776+0000 7f30a9bfb700 1 --2- 172.21.15.86:0/1602782913 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f30a40f62c0 0x7f30a406b9a0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f30980089f0 tx=0x7f309800b040).stop 2024-02-19T22:00:20.777 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:20.776+0000 7f30a9bfb700 1 -- 172.21.15.86:0/1602782913 shutdown_connections 2024-02-19T22:00:20.777 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:20.776+0000 7f30a9bfb700 1 --2- 172.21.15.86:0/1602782913 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f309402ff60 0x7f3094032420 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:00:20.777 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:20.776+0000 7f30a9bfb700 1 --2- 172.21.15.86:0/1602782913 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f30a40f62c0 0x7f30a406b9a0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:00:20.777 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:20.776+0000 7f30a9bfb700 1 -- 172.21.15.86:0/1602782913 >> 172.21.15.86:0/1602782913 conn(0x7f30a40f1ac0 msgr2=0x7f30a40f24c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:00:20.777 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:20.776+0000 7f30a9bfb700 1 -- 172.21.15.86:0/1602782913 shutdown_connections 2024-02-19T22:00:20.777 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:20.776+0000 7f30a9bfb700 1 -- 172.21.15.86:0/1602782913 wait complete. 2024-02-19T22:00:20.777 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T22:00:20.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:00:20 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:00:19.470117+0000 mgr.smithi053.fyhcdi (mgr.14162) 113 : cluster [DBG] pgmap v106: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:00:21.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:00:21 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:00:20.775926+0000 mon.smithi053 (mon.0) 173 : audit [DBG] from='client.? 172.21.15.86:0/1602782913' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:00:22.367 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:00:22.368 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:00:22.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:00:22 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:00:21.470378+0000 mgr.smithi053.fyhcdi (mgr.14162) 114 : cluster [DBG] pgmap v107: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:00:24.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:24.216+0000 7f1c85069700 1 -- 172.21.15.86:0/1990361214 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1c800f51a0 msgr2=0x7f1c800f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:00:24.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:24.216+0000 7f1c85069700 1 --2- 172.21.15.86:0/1990361214 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1c800f51a0 0x7f1c800f55c0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f1c74009fb0 tx=0x7f1c74009b30).stop 2024-02-19T22:00:24.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:24.216+0000 7f1c85069700 1 -- 172.21.15.86:0/1990361214 shutdown_connections 2024-02-19T22:00:24.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:24.216+0000 7f1c85069700 1 --2- 172.21.15.86:0/1990361214 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1c800f51a0 0x7f1c800f55c0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:00:24.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:24.216+0000 7f1c85069700 1 -- 172.21.15.86:0/1990361214 >> 172.21.15.86:0/1990361214 conn(0x7f1c800f09a0 msgr2=0x7f1c800f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:00:24.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:24.216+0000 7f1c85069700 1 -- 172.21.15.86:0/1990361214 shutdown_connections 2024-02-19T22:00:24.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:24.216+0000 7f1c85069700 1 -- 172.21.15.86:0/1990361214 wait complete. 2024-02-19T22:00:24.218 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:24.217+0000 7f1c85069700 1 Processor -- start 2024-02-19T22:00:24.218 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:24.217+0000 7f1c85069700 1 -- start start 2024-02-19T22:00:24.218 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:24.217+0000 7f1c85069700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1c800f51a0 0x7f1c801054a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:00:24.218 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:24.217+0000 7f1c85069700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c80105990 con 0x7f1c800f51a0 2024-02-19T22:00:24.218 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:24.217+0000 7f1c7ffff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1c800f51a0 0x7f1c801054a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:00:24.218 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:24.217+0000 7f1c7ffff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1c800f51a0 0x7f1c801054a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:35424/0 (socket says 172.21.15.86:35424) 2024-02-19T22:00:24.218 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:24.217+0000 7f1c7ffff700 1 -- 172.21.15.86:0/2361652673 learned_addr learned my addr 172.21.15.86:0/2361652673 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T22:00:24.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:24.218+0000 7f1c7ffff700 1 -- 172.21.15.86:0/2361652673 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c74005040 con 0x7f1c800f51a0 2024-02-19T22:00:24.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:24.218+0000 7f1c7ffff700 1 --2- 172.21.15.86:0/2361652673 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1c800f51a0 0x7f1c801054a0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f1c740089f0 tx=0x7f1c7400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:00:24.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:24.218+0000 7f1c7cff9700 1 -- 172.21.15.86:0/2361652673 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1c74009320 con 0x7f1c800f51a0 2024-02-19T22:00:24.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:24.218+0000 7f1c85069700 1 -- 172.21.15.86:0/2361652673 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c80105b90 con 0x7f1c800f51a0 2024-02-19T22:00:24.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:24.218+0000 7f1c7cff9700 1 -- 172.21.15.86:0/2361652673 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1c74013380 con 0x7f1c800f51a0 2024-02-19T22:00:24.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:24.218+0000 7f1c7cff9700 1 -- 172.21.15.86:0/2361652673 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1c74014e20 con 0x7f1c800f51a0 2024-02-19T22:00:24.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:24.218+0000 7f1c85069700 1 -- 172.21.15.86:0/2361652673 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c80106020 con 0x7f1c800f51a0 2024-02-19T22:00:24.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:24.219+0000 7f1c7cff9700 1 -- 172.21.15.86:0/2361652673 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1c74014420 con 0x7f1c800f51a0 2024-02-19T22:00:24.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:24.219+0000 7f1c7cff9700 1 --2- 172.21.15.86:0/2361652673 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f1c6802ff60 0x7f1c68032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:00:24.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:24.219+0000 7f1c85069700 1 -- 172.21.15.86:0/2361652673 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c80045ad0 con 0x7f1c800f51a0 2024-02-19T22:00:24.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:24.219+0000 7f1c7cff9700 1 -- 172.21.15.86:0/2361652673 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1c74040fc0 con 0x7f1c800f51a0 2024-02-19T22:00:24.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:24.219+0000 7f1c7f7fe700 1 --2- 172.21.15.86:0/2361652673 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f1c6802ff60 0x7f1c68032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:00:24.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:24.220+0000 7f1c7f7fe700 1 --2- 172.21.15.86:0/2361652673 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f1c6802ff60 0x7f1c68032420 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f1c700096a0 tx=0x7f1c70006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:00:24.224 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:24.224+0000 7f1c7cff9700 1 -- 172.21.15.86:0/2361652673 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1c740095f0 con 0x7f1c800f51a0 2024-02-19T22:00:24.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:24.431+0000 7f1c85069700 1 -- 172.21.15.86:0/2361652673 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1c80106ed0 con 0x7f1c800f51a0 2024-02-19T22:00:24.432 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:24.432+0000 7f1c7cff9700 1 -- 172.21.15.86:0/2361652673 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1c7401db70 con 0x7f1c800f51a0 2024-02-19T22:00:24.432 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T22:00:24.432 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:00:24.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:24.433+0000 7f1c85069700 1 -- 172.21.15.86:0/2361652673 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f1c6802ff60 msgr2=0x7f1c68032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:00:24.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:24.433+0000 7f1c85069700 1 --2- 172.21.15.86:0/2361652673 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f1c6802ff60 0x7f1c68032420 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f1c700096a0 tx=0x7f1c70006b40).stop 2024-02-19T22:00:24.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:24.433+0000 7f1c85069700 1 -- 172.21.15.86:0/2361652673 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1c800f51a0 msgr2=0x7f1c801054a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:00:24.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:24.433+0000 7f1c85069700 1 --2- 172.21.15.86:0/2361652673 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1c800f51a0 0x7f1c801054a0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f1c740089f0 tx=0x7f1c7400b040).stop 2024-02-19T22:00:24.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:24.433+0000 7f1c85069700 1 -- 172.21.15.86:0/2361652673 shutdown_connections 2024-02-19T22:00:24.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:24.433+0000 7f1c85069700 1 --2- 172.21.15.86:0/2361652673 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f1c6802ff60 0x7f1c68032420 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:00:24.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:24.433+0000 7f1c85069700 1 --2- 172.21.15.86:0/2361652673 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1c800f51a0 0x7f1c801054a0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:00:24.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:24.433+0000 7f1c85069700 1 -- 172.21.15.86:0/2361652673 >> 172.21.15.86:0/2361652673 conn(0x7f1c800f09a0 msgr2=0x7f1c800f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:00:24.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:24.433+0000 7f1c85069700 1 -- 172.21.15.86:0/2361652673 shutdown_connections 2024-02-19T22:00:24.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:24.433+0000 7f1c85069700 1 -- 172.21.15.86:0/2361652673 wait complete. 2024-02-19T22:00:24.434 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T22:00:24.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:00:24 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:00:23.470607+0000 mgr.smithi053.fyhcdi (mgr.14162) 115 : cluster [DBG] pgmap v108: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:00:24.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:00:24 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:00:24.433249+0000 mon.smithi053 (mon.0) 174 : audit [DBG] from='client.? 172.21.15.86:0/2361652673' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:00:26.114 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:00:26.115 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:00:26.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:00:26 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:00:25.470949+0000 mgr.smithi053.fyhcdi (mgr.14162) 116 : cluster [DBG] pgmap v109: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:00:27.894 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:27.893+0000 7f1690904700 1 -- 172.21.15.86:0/2505389088 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f168c0f51a0 msgr2=0x7f168c0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:00:27.895 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:27.893+0000 7f1690904700 1 --2- 172.21.15.86:0/2505389088 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f168c0f51a0 0x7f168c0f55c0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f167c009fb0 tx=0x7f167c009b30).stop 2024-02-19T22:00:27.895 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:27.893+0000 7f1690904700 1 -- 172.21.15.86:0/2505389088 shutdown_connections 2024-02-19T22:00:27.895 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:27.893+0000 7f1690904700 1 --2- 172.21.15.86:0/2505389088 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f168c0f51a0 0x7f168c0f55c0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:00:27.895 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:27.893+0000 7f1690904700 1 -- 172.21.15.86:0/2505389088 >> 172.21.15.86:0/2505389088 conn(0x7f168c0f09a0 msgr2=0x7f168c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:00:27.895 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:27.893+0000 7f1690904700 1 -- 172.21.15.86:0/2505389088 shutdown_connections 2024-02-19T22:00:27.895 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:27.893+0000 7f1690904700 1 -- 172.21.15.86:0/2505389088 wait complete. 2024-02-19T22:00:27.895 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:27.894+0000 7f1690904700 1 Processor -- start 2024-02-19T22:00:27.895 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:27.894+0000 7f1690904700 1 -- start start 2024-02-19T22:00:27.895 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:27.894+0000 7f1690904700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f168c0f51a0 0x7f168c0f2f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:00:27.895 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:27.894+0000 7f1690904700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f168c0f13e0 con 0x7f168c0f51a0 2024-02-19T22:00:27.895 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:27.895+0000 7f168b7fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f168c0f51a0 0x7f168c0f2f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:00:27.895 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:27.895+0000 7f168b7fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f168c0f51a0 0x7f168c0f2f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:35456/0 (socket says 172.21.15.86:35456) 2024-02-19T22:00:27.895 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:27.895+0000 7f168b7fe700 1 -- 172.21.15.86:0/1659839939 learned_addr learned my addr 172.21.15.86:0/1659839939 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T22:00:27.895 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:27.895+0000 7f168b7fe700 1 -- 172.21.15.86:0/1659839939 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f167c005040 con 0x7f168c0f51a0 2024-02-19T22:00:27.896 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:27.895+0000 7f168b7fe700 1 --2- 172.21.15.86:0/1659839939 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f168c0f51a0 0x7f168c0f2f60 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f167c009830 tx=0x7f167c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:00:27.897 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:27.895+0000 7f1673fff700 1 -- 172.21.15.86:0/1659839939 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f167c009320 con 0x7f168c0f51a0 2024-02-19T22:00:27.897 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:27.895+0000 7f1690904700 1 -- 172.21.15.86:0/1659839939 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f168c0f1580 con 0x7f168c0f51a0 2024-02-19T22:00:27.897 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:27.895+0000 7f1673fff700 1 -- 172.21.15.86:0/1659839939 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f167c013380 con 0x7f168c0f51a0 2024-02-19T22:00:27.897 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:27.895+0000 7f1673fff700 1 -- 172.21.15.86:0/1659839939 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f167c014420 con 0x7f168c0f51a0 2024-02-19T22:00:27.897 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:27.895+0000 7f1690904700 1 -- 172.21.15.86:0/1659839939 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f168c0f1a10 con 0x7f168c0f51a0 2024-02-19T22:00:27.897 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:27.896+0000 7f1673fff700 1 -- 172.21.15.86:0/1659839939 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f167c014580 con 0x7f168c0f51a0 2024-02-19T22:00:27.897 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:27.896+0000 7f1673fff700 1 --2- 172.21.15.86:0/1659839939 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f167402ff60 0x7f1674032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:00:27.897 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:27.896+0000 7f1673fff700 1 -- 172.21.15.86:0/1659839939 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f167c040e10 con 0x7f168c0f51a0 2024-02-19T22:00:27.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:27.896+0000 7f1690904700 1 -- 172.21.15.86:0/1659839939 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f168c059100 con 0x7f168c0f51a0 2024-02-19T22:00:27.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:27.896+0000 7f168affd700 1 --2- 172.21.15.86:0/1659839939 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f167402ff60 0x7f1674032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:00:27.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:27.897+0000 7f168affd700 1 --2- 172.21.15.86:0/1659839939 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f167402ff60 0x7f1674032420 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f16800096a0 tx=0x7f1680006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:00:27.901 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:27.901+0000 7f1673fff700 1 -- 172.21.15.86:0/1659839939 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f167c0672b0 con 0x7f168c0f51a0 2024-02-19T22:00:28.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:28.104+0000 7f1690904700 1 -- 172.21.15.86:0/1659839939 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f168c045ad0 con 0x7f168c0f51a0 2024-02-19T22:00:28.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:28.105+0000 7f1673fff700 1 -- 172.21.15.86:0/1659839939 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f167c0273f0 con 0x7f168c0f51a0 2024-02-19T22:00:28.105 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T22:00:28.105 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:00:28.106 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:28.106+0000 7f1690904700 1 -- 172.21.15.86:0/1659839939 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f167402ff60 msgr2=0x7f1674032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:00:28.106 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:28.106+0000 7f1690904700 1 --2- 172.21.15.86:0/1659839939 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f167402ff60 0x7f1674032420 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f16800096a0 tx=0x7f1680006b40).stop 2024-02-19T22:00:28.106 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:28.106+0000 7f1690904700 1 -- 172.21.15.86:0/1659839939 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f168c0f51a0 msgr2=0x7f168c0f2f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:00:28.106 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:28.106+0000 7f1690904700 1 --2- 172.21.15.86:0/1659839939 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f168c0f51a0 0x7f168c0f2f60 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f167c009830 tx=0x7f167c00b040).stop 2024-02-19T22:00:28.106 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:28.106+0000 7f1690904700 1 -- 172.21.15.86:0/1659839939 shutdown_connections 2024-02-19T22:00:28.106 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:28.106+0000 7f1690904700 1 --2- 172.21.15.86:0/1659839939 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f167402ff60 0x7f1674032420 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:00:28.106 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:28.106+0000 7f1690904700 1 --2- 172.21.15.86:0/1659839939 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f168c0f51a0 0x7f168c0f2f60 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:00:28.107 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:28.106+0000 7f1690904700 1 -- 172.21.15.86:0/1659839939 >> 172.21.15.86:0/1659839939 conn(0x7f168c0f09a0 msgr2=0x7f168c05fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:00:28.107 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:28.106+0000 7f1690904700 1 -- 172.21.15.86:0/1659839939 shutdown_connections 2024-02-19T22:00:28.107 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:28.106+0000 7f1690904700 1 -- 172.21.15.86:0/1659839939 wait complete. 2024-02-19T22:00:28.107 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T22:00:28.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:00:28 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:00:27.471281+0000 mgr.smithi053.fyhcdi (mgr.14162) 117 : cluster [DBG] pgmap v110: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:00:28.794 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:00:28 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:00:28.106008+0000 mon.smithi053 (mon.0) 175 : audit [DBG] from='client.? 172.21.15.86:0/1659839939' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:00:29.718 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:00:29.718 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:00:30.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:00:30 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:00:29.471512+0000 mgr.smithi053.fyhcdi (mgr.14162) 118 : cluster [DBG] pgmap v111: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:00:31.665 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:31.663+0000 7fdc025c2700 1 -- 172.21.15.86:0/1593257067 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdbfc060220 msgr2=0x7fdbfc0f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:00:31.665 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:31.663+0000 7fdc025c2700 1 --2- 172.21.15.86:0/1593257067 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdbfc060220 0x7fdbfc0f9150 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7fdbec009fb0 tx=0x7fdbec009b30).stop 2024-02-19T22:00:31.665 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:31.664+0000 7fdc025c2700 1 -- 172.21.15.86:0/1593257067 shutdown_connections 2024-02-19T22:00:31.665 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:31.664+0000 7fdc025c2700 1 --2- 172.21.15.86:0/1593257067 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdbfc060220 0x7fdbfc0f9150 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:00:31.665 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:31.664+0000 7fdc025c2700 1 -- 172.21.15.86:0/1593257067 >> 172.21.15.86:0/1593257067 conn(0x7fdbfc0f0b70 msgr2=0x7fdbfc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:00:31.666 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:31.664+0000 7fdc025c2700 1 -- 172.21.15.86:0/1593257067 shutdown_connections 2024-02-19T22:00:31.666 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:31.664+0000 7fdc025c2700 1 -- 172.21.15.86:0/1593257067 wait complete. 2024-02-19T22:00:31.666 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:31.664+0000 7fdc025c2700 1 Processor -- start 2024-02-19T22:00:31.666 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:31.664+0000 7fdc025c2700 1 -- start start 2024-02-19T22:00:31.666 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:31.664+0000 7fdc025c2700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdbfc060220 0x7fdbfc0f6bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:00:31.666 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:31.664+0000 7fdc025c2700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbfc0f5050 con 0x7fdbfc060220 2024-02-19T22:00:31.666 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:31.665+0000 7fdbfbfff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdbfc060220 0x7fdbfc0f6bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:00:31.666 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:31.665+0000 7fdbfbfff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdbfc060220 0x7fdbfc0f6bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:48746/0 (socket says 172.21.15.86:48746) 2024-02-19T22:00:31.666 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:31.665+0000 7fdbfbfff700 1 -- 172.21.15.86:0/2016139249 learned_addr learned my addr 172.21.15.86:0/2016139249 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T22:00:31.666 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:31.665+0000 7fdbfbfff700 1 -- 172.21.15.86:0/2016139249 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbec005040 con 0x7fdbfc060220 2024-02-19T22:00:31.666 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:31.665+0000 7fdbfbfff700 1 --2- 172.21.15.86:0/2016139249 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdbfc060220 0x7fdbfc0f6bd0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7fdbec0089f0 tx=0x7fdbec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:00:31.666 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:31.665+0000 7fdbf8ff9700 1 -- 172.21.15.86:0/2016139249 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdbec009320 con 0x7fdbfc060220 2024-02-19T22:00:31.666 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:31.665+0000 7fdc025c2700 1 -- 172.21.15.86:0/2016139249 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbfc0f51f0 con 0x7fdbfc060220 2024-02-19T22:00:31.667 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:31.665+0000 7fdbf8ff9700 1 -- 172.21.15.86:0/2016139249 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdbec013380 con 0x7fdbfc060220 2024-02-19T22:00:31.667 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:31.665+0000 7fdbf8ff9700 1 -- 172.21.15.86:0/2016139249 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdbec014e20 con 0x7fdbfc060220 2024-02-19T22:00:31.667 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:31.665+0000 7fdc025c2700 1 -- 172.21.15.86:0/2016139249 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbfc0f5680 con 0x7fdbfc060220 2024-02-19T22:00:31.668 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:31.666+0000 7fdbf8ff9700 1 -- 172.21.15.86:0/2016139249 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdbec0095f0 con 0x7fdbfc060220 2024-02-19T22:00:31.668 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:31.666+0000 7fdc025c2700 1 -- 172.21.15.86:0/2016139249 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbfc059100 con 0x7fdbfc060220 2024-02-19T22:00:31.668 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:31.666+0000 7fdbf8ff9700 1 --2- 172.21.15.86:0/2016139249 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fdbe402ff10 0x7fdbe40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:00:31.668 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:31.666+0000 7fdbf8ff9700 1 -- 172.21.15.86:0/2016139249 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdbec040f10 con 0x7fdbfc060220 2024-02-19T22:00:31.668 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:31.666+0000 7fdbfb7fe700 1 --2- 172.21.15.86:0/2016139249 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fdbe402ff10 0x7fdbe40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:00:31.668 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:31.667+0000 7fdbfb7fe700 1 --2- 172.21.15.86:0/2016139249 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fdbe402ff10 0x7fdbe40323d0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7fdbf00096a0 tx=0x7fdbf0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:00:31.670 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:31.671+0000 7fdbf8ff9700 1 -- 172.21.15.86:0/2016139249 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdbec014420 con 0x7fdbfc060220 2024-02-19T22:00:31.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:31.874+0000 7fdc025c2700 1 -- 172.21.15.86:0/2016139249 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdbfc0fc090 con 0x7fdbfc060220 2024-02-19T22:00:31.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:31.875+0000 7fdbf8ff9700 1 -- 172.21.15.86:0/2016139249 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fdbec01d980 con 0x7fdbfc060220 2024-02-19T22:00:31.875 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T22:00:31.875 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:00:31.876 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:31.876+0000 7fdc025c2700 1 -- 172.21.15.86:0/2016139249 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fdbe402ff10 msgr2=0x7fdbe40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:00:31.876 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:31.876+0000 7fdc025c2700 1 --2- 172.21.15.86:0/2016139249 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fdbe402ff10 0x7fdbe40323d0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7fdbf00096a0 tx=0x7fdbf0006b40).stop 2024-02-19T22:00:31.876 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:31.876+0000 7fdc025c2700 1 -- 172.21.15.86:0/2016139249 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdbfc060220 msgr2=0x7fdbfc0f6bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:00:31.876 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:31.876+0000 7fdc025c2700 1 --2- 172.21.15.86:0/2016139249 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdbfc060220 0x7fdbfc0f6bd0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7fdbec0089f0 tx=0x7fdbec00b040).stop 2024-02-19T22:00:31.876 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:31.876+0000 7fdc025c2700 1 -- 172.21.15.86:0/2016139249 shutdown_connections 2024-02-19T22:00:31.876 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:31.876+0000 7fdc025c2700 1 --2- 172.21.15.86:0/2016139249 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fdbe402ff10 0x7fdbe40323d0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:00:31.877 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:31.876+0000 7fdc025c2700 1 --2- 172.21.15.86:0/2016139249 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdbfc060220 0x7fdbfc0f6bd0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:00:31.877 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:31.876+0000 7fdc025c2700 1 -- 172.21.15.86:0/2016139249 >> 172.21.15.86:0/2016139249 conn(0x7fdbfc0f0b70 msgr2=0x7fdbfc0f15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:00:31.877 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:31.876+0000 7fdc025c2700 1 -- 172.21.15.86:0/2016139249 shutdown_connections 2024-02-19T22:00:31.877 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:31.876+0000 7fdc025c2700 1 -- 172.21.15.86:0/2016139249 wait complete. 2024-02-19T22:00:31.877 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T22:00:32.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:00:32 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:00:31.471907+0000 mgr.smithi053.fyhcdi (mgr.14162) 119 : cluster [DBG] pgmap v112: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:00:32.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:00:32 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:00:31.876190+0000 mon.smithi053 (mon.0) 176 : audit [DBG] from='client.? 172.21.15.86:0/2016139249' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:00:33.582 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:00:33.582 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:00:34.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:00:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:00:33.472327+0000 mgr.smithi053.fyhcdi (mgr.14162) 120 : cluster [DBG] pgmap v113: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:00:35.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:35.433+0000 7fdc3384f700 1 -- 172.21.15.86:0/4042325412 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdc2c0f5410 msgr2=0x7fdc2c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:00:35.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:35.433+0000 7fdc3384f700 1 --2- 172.21.15.86:0/4042325412 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdc2c0f5410 0x7fdc2c0f5830 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7fdc28009fb0 tx=0x7fdc28009b30).stop 2024-02-19T22:00:35.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:35.433+0000 7fdc3384f700 1 -- 172.21.15.86:0/4042325412 shutdown_connections 2024-02-19T22:00:35.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:35.433+0000 7fdc3384f700 1 --2- 172.21.15.86:0/4042325412 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdc2c0f5410 0x7fdc2c0f5830 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:00:35.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:35.433+0000 7fdc3384f700 1 -- 172.21.15.86:0/4042325412 >> 172.21.15.86:0/4042325412 conn(0x7fdc2c0f0b70 msgr2=0x7fdc2c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:00:35.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:35.433+0000 7fdc3384f700 1 -- 172.21.15.86:0/4042325412 shutdown_connections 2024-02-19T22:00:35.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:35.433+0000 7fdc3384f700 1 -- 172.21.15.86:0/4042325412 wait complete. 2024-02-19T22:00:35.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:35.434+0000 7fdc3384f700 1 Processor -- start 2024-02-19T22:00:35.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:35.434+0000 7fdc3384f700 1 -- start start 2024-02-19T22:00:35.436 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:35.434+0000 7fdc3384f700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdc2c0f5410 0x7fdc2c105710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:00:35.436 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:35.434+0000 7fdc3384f700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc2c105c00 con 0x7fdc2c0f5410 2024-02-19T22:00:35.436 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:35.434+0000 7fdc315eb700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdc2c0f5410 0x7fdc2c105710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:00:35.436 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:35.434+0000 7fdc315eb700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdc2c0f5410 0x7fdc2c105710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:48774/0 (socket says 172.21.15.86:48774) 2024-02-19T22:00:35.436 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:35.434+0000 7fdc315eb700 1 -- 172.21.15.86:0/37260138 learned_addr learned my addr 172.21.15.86:0/37260138 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T22:00:35.436 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:35.434+0000 7fdc315eb700 1 -- 172.21.15.86:0/37260138 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc28005040 con 0x7fdc2c0f5410 2024-02-19T22:00:35.436 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:35.435+0000 7fdc315eb700 1 --2- 172.21.15.86:0/37260138 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdc2c0f5410 0x7fdc2c105710 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7fdc28005010 tx=0x7fdc2800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:00:35.436 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:35.435+0000 7fdc21ffb700 1 -- 172.21.15.86:0/37260138 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdc28009320 con 0x7fdc2c0f5410 2024-02-19T22:00:35.436 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:35.435+0000 7fdc3384f700 1 -- 172.21.15.86:0/37260138 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc2c105e00 con 0x7fdc2c0f5410 2024-02-19T22:00:35.436 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:35.435+0000 7fdc21ffb700 1 -- 172.21.15.86:0/37260138 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdc28013380 con 0x7fdc2c0f5410 2024-02-19T22:00:35.436 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:35.435+0000 7fdc21ffb700 1 -- 172.21.15.86:0/37260138 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdc28014e50 con 0x7fdc2c0f5410 2024-02-19T22:00:35.436 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:35.435+0000 7fdc3384f700 1 -- 172.21.15.86:0/37260138 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc2c106290 con 0x7fdc2c0f5410 2024-02-19T22:00:35.437 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:35.436+0000 7fdc21ffb700 1 -- 172.21.15.86:0/37260138 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdc280263a0 con 0x7fdc2c0f5410 2024-02-19T22:00:35.437 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:35.436+0000 7fdc3384f700 1 -- 172.21.15.86:0/37260138 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc2c045ad0 con 0x7fdc2c0f5410 2024-02-19T22:00:35.437 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:35.436+0000 7fdc21ffb700 1 --2- 172.21.15.86:0/37260138 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fdc1802ff10 0x7fdc180323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:00:35.437 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:35.436+0000 7fdc21ffb700 1 -- 172.21.15.86:0/37260138 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdc28041010 con 0x7fdc2c0f5410 2024-02-19T22:00:35.438 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:35.436+0000 7fdc30dea700 1 --2- 172.21.15.86:0/37260138 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fdc1802ff10 0x7fdc180323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:00:35.438 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:35.437+0000 7fdc30dea700 1 --2- 172.21.15.86:0/37260138 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fdc1802ff10 0x7fdc180323d0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7fdc1c0096a0 tx=0x7fdc1c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:00:35.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:35.441+0000 7fdc21ffb700 1 -- 172.21.15.86:0/37260138 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdc2801dde0 con 0x7fdc2c0f5410 2024-02-19T22:00:35.644 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:35.644+0000 7fdc3384f700 1 -- 172.21.15.86:0/37260138 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdc2c1070e0 con 0x7fdc2c0f5410 2024-02-19T22:00:35.645 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:35.645+0000 7fdc21ffb700 1 -- 172.21.15.86:0/37260138 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fdc2801d9d0 con 0x7fdc2c0f5410 2024-02-19T22:00:35.645 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T22:00:35.645 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:00:35.647 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:35.647+0000 7fdc3384f700 1 -- 172.21.15.86:0/37260138 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fdc1802ff10 msgr2=0x7fdc180323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:00:35.647 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:35.647+0000 7fdc3384f700 1 --2- 172.21.15.86:0/37260138 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fdc1802ff10 0x7fdc180323d0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7fdc1c0096a0 tx=0x7fdc1c006b40).stop 2024-02-19T22:00:35.647 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:35.647+0000 7fdc3384f700 1 -- 172.21.15.86:0/37260138 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdc2c0f5410 msgr2=0x7fdc2c105710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:00:35.647 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:35.647+0000 7fdc3384f700 1 --2- 172.21.15.86:0/37260138 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdc2c0f5410 0x7fdc2c105710 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7fdc28005010 tx=0x7fdc2800b040).stop 2024-02-19T22:00:35.647 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:35.647+0000 7fdc3384f700 1 -- 172.21.15.86:0/37260138 shutdown_connections 2024-02-19T22:00:35.647 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:35.647+0000 7fdc3384f700 1 --2- 172.21.15.86:0/37260138 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fdc1802ff10 0x7fdc180323d0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:00:35.647 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:35.647+0000 7fdc3384f700 1 --2- 172.21.15.86:0/37260138 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdc2c0f5410 0x7fdc2c105710 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:00:35.647 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:35.647+0000 7fdc3384f700 1 -- 172.21.15.86:0/37260138 >> 172.21.15.86:0/37260138 conn(0x7fdc2c0f0b70 msgr2=0x7fdc2c0f15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:00:35.647 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:35.647+0000 7fdc3384f700 1 -- 172.21.15.86:0/37260138 shutdown_connections 2024-02-19T22:00:35.647 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:35.647+0000 7fdc3384f700 1 -- 172.21.15.86:0/37260138 wait complete. 2024-02-19T22:00:35.648 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T22:00:36.794 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:00:36 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:00:35.472630+0000 mgr.smithi053.fyhcdi (mgr.14162) 121 : cluster [DBG] pgmap v114: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:00:36.794 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:00:36 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:00:35.600070+0000 mon.smithi053 (mon.0) 177 : audit [INF] from='mgr.14162 172.21.15.53:0/2860358250' entity='mgr.smithi053.fyhcdi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi053.fyhcdi/mirror_snapshot_schedule"}]: dispatch 2024-02-19T22:00:36.794 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:00:36 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:00:35.603565+0000 mon.smithi053 (mon.0) 178 : audit [INF] from='mgr.14162 172.21.15.53:0/2860358250' entity='mgr.smithi053.fyhcdi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi053.fyhcdi/trash_purge_schedule"}]: dispatch 2024-02-19T22:00:36.794 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:00:36 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:00:35.646399+0000 mon.smithi053 (mon.0) 179 : audit [DBG] from='client.? 172.21.15.86:0/37260138' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:00:37.291 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:00:37.291 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:00:38.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:00:38 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:00:37.472971+0000 mgr.smithi053.fyhcdi (mgr.14162) 122 : cluster [DBG] pgmap v115: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:00:39.192 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:39.189+0000 7f5706192700 1 -- 172.21.15.86:0/3984594577 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5700060220 msgr2=0x7f57000f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:00:39.192 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:39.189+0000 7f5706192700 1 --2- 172.21.15.86:0/3984594577 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5700060220 0x7f57000f9150 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 rx=0x7f56f0009fb0 tx=0x7f56f0009b30).stop 2024-02-19T22:00:39.192 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:39.189+0000 7f5706192700 1 -- 172.21.15.86:0/3984594577 shutdown_connections 2024-02-19T22:00:39.192 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:39.189+0000 7f5706192700 1 --2- 172.21.15.86:0/3984594577 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5700060220 0x7f57000f9150 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:00:39.192 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:39.189+0000 7f5706192700 1 -- 172.21.15.86:0/3984594577 >> 172.21.15.86:0/3984594577 conn(0x7f57000f0b70 msgr2=0x7f57000f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:00:39.192 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:39.189+0000 7f5706192700 1 -- 172.21.15.86:0/3984594577 shutdown_connections 2024-02-19T22:00:39.192 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:39.189+0000 7f5706192700 1 -- 172.21.15.86:0/3984594577 wait complete. 2024-02-19T22:00:39.193 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:39.189+0000 7f5706192700 1 Processor -- start 2024-02-19T22:00:39.193 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:39.189+0000 7f5706192700 1 -- start start 2024-02-19T22:00:39.193 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:39.190+0000 7f5706192700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5700060220 0x7f57001067a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:00:39.193 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:39.190+0000 7f5706192700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5700106c90 con 0x7f5700060220 2024-02-19T22:00:39.193 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:39.190+0000 7f56ff7fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5700060220 0x7f57001067a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:00:39.193 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:39.190+0000 7f56ff7fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5700060220 0x7f57001067a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:48786/0 (socket says 172.21.15.86:48786) 2024-02-19T22:00:39.193 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:39.190+0000 7f56ff7fe700 1 -- 172.21.15.86:0/1294061065 learned_addr learned my addr 172.21.15.86:0/1294061065 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T22:00:39.193 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:39.190+0000 7f56ff7fe700 1 -- 172.21.15.86:0/1294061065 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56f0005040 con 0x7f5700060220 2024-02-19T22:00:39.193 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:39.190+0000 7f56ff7fe700 1 --2- 172.21.15.86:0/1294061065 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5700060220 0x7f57001067a0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7f56f00089f0 tx=0x7f56f000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:00:39.193 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:39.190+0000 7f56e7fff700 1 -- 172.21.15.86:0/1294061065 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f56f0009320 con 0x7f5700060220 2024-02-19T22:00:39.193 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:39.191+0000 7f5706192700 1 -- 172.21.15.86:0/1294061065 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5700106e90 con 0x7f5700060220 2024-02-19T22:00:39.193 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:39.191+0000 7f56e7fff700 1 -- 172.21.15.86:0/1294061065 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f56f0013380 con 0x7f5700060220 2024-02-19T22:00:39.194 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:39.191+0000 7f56e7fff700 1 -- 172.21.15.86:0/1294061065 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f56f0014e20 con 0x7f5700060220 2024-02-19T22:00:39.194 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:39.191+0000 7f5706192700 1 -- 172.21.15.86:0/1294061065 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5700107320 con 0x7f5700060220 2024-02-19T22:00:39.194 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:39.191+0000 7f56e7fff700 1 -- 172.21.15.86:0/1294061065 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f56f00095f0 con 0x7f5700060220 2024-02-19T22:00:39.194 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:39.191+0000 7f5706192700 1 -- 172.21.15.86:0/1294061065 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5700045ad0 con 0x7f5700060220 2024-02-19T22:00:39.194 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:39.192+0000 7f56e7fff700 1 --2- 172.21.15.86:0/1294061065 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f56e802ff10 0x7f56e80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:00:39.194 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:39.192+0000 7f56e7fff700 1 -- 172.21.15.86:0/1294061065 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f56f0040f80 con 0x7f5700060220 2024-02-19T22:00:39.194 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:39.192+0000 7f56feffd700 1 --2- 172.21.15.86:0/1294061065 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f56e802ff10 0x7f56e80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:00:39.194 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:39.192+0000 7f56feffd700 1 --2- 172.21.15.86:0/1294061065 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f56e802ff10 0x7f56e80323d0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f56f40096a0 tx=0x7f56f4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:00:39.196 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:39.196+0000 7f56e7fff700 1 -- 172.21.15.86:0/1294061065 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f56f0014420 con 0x7f5700060220 2024-02-19T22:00:39.401 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:39.401+0000 7f5706192700 1 -- 172.21.15.86:0/1294061065 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5700108170 con 0x7f5700060220 2024-02-19T22:00:39.401 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:39.402+0000 7f56e7fff700 1 -- 172.21.15.86:0/1294061065 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f56f0010230 con 0x7f5700060220 2024-02-19T22:00:39.402 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T22:00:39.402 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:00:39.403 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:39.403+0000 7f5706192700 1 -- 172.21.15.86:0/1294061065 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f56e802ff10 msgr2=0x7f56e80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:00:39.403 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:39.403+0000 7f5706192700 1 --2- 172.21.15.86:0/1294061065 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f56e802ff10 0x7f56e80323d0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f56f40096a0 tx=0x7f56f4006b40).stop 2024-02-19T22:00:39.403 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:39.403+0000 7f5706192700 1 -- 172.21.15.86:0/1294061065 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5700060220 msgr2=0x7f57001067a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:00:39.403 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:39.403+0000 7f5706192700 1 --2- 172.21.15.86:0/1294061065 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5700060220 0x7f57001067a0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7f56f00089f0 tx=0x7f56f000b040).stop 2024-02-19T22:00:39.404 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:39.403+0000 7f5706192700 1 -- 172.21.15.86:0/1294061065 shutdown_connections 2024-02-19T22:00:39.404 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:39.403+0000 7f5706192700 1 --2- 172.21.15.86:0/1294061065 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f56e802ff10 0x7f56e80323d0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:00:39.404 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:39.403+0000 7f5706192700 1 --2- 172.21.15.86:0/1294061065 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5700060220 0x7f57001067a0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:00:39.404 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:39.403+0000 7f5706192700 1 -- 172.21.15.86:0/1294061065 >> 172.21.15.86:0/1294061065 conn(0x7f57000f0b70 msgr2=0x7f57000f15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:00:39.404 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:39.403+0000 7f5706192700 1 -- 172.21.15.86:0/1294061065 shutdown_connections 2024-02-19T22:00:39.404 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:39.403+0000 7f5706192700 1 -- 172.21.15.86:0/1294061065 wait complete. 2024-02-19T22:00:39.405 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T22:00:39.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:00:39 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:00:39.403034+0000 mon.smithi053 (mon.0) 180 : audit [DBG] from='client.? 172.21.15.86:0/1294061065' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:00:40.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:00:40 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:00:39.473331+0000 mgr.smithi053.fyhcdi (mgr.14162) 123 : cluster [DBG] pgmap v116: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:00:41.070 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:00:41.070 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:00:42.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:00:42 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:00:41.473638+0000 mgr.smithi053.fyhcdi (mgr.14162) 124 : cluster [DBG] pgmap v117: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:00:42.934 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:42.932+0000 7f12a2498700 1 -- 172.21.15.86:0/719819990 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f129c0f4b80 msgr2=0x7f129c0f4fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:00:42.934 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:42.932+0000 7f12a2498700 1 --2- 172.21.15.86:0/719819990 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f129c0f4b80 0x7f129c0f4fa0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 rx=0x7f1290004770 tx=0x7f129000ac40).stop 2024-02-19T22:00:42.934 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:42.932+0000 7f12a2498700 1 -- 172.21.15.86:0/719819990 shutdown_connections 2024-02-19T22:00:42.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:42.932+0000 7f12a2498700 1 --2- 172.21.15.86:0/719819990 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f129c0f4b80 0x7f129c0f4fa0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:00:42.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:42.932+0000 7f12a2498700 1 -- 172.21.15.86:0/719819990 >> 172.21.15.86:0/719819990 conn(0x7f129c0f02e0 msgr2=0x7f129c0f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:00:42.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:42.932+0000 7f12a2498700 1 -- 172.21.15.86:0/719819990 shutdown_connections 2024-02-19T22:00:42.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:42.932+0000 7f12a2498700 1 -- 172.21.15.86:0/719819990 wait complete. 2024-02-19T22:00:42.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:42.933+0000 7f12a2498700 1 Processor -- start 2024-02-19T22:00:42.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:42.933+0000 7f12a2498700 1 -- start start 2024-02-19T22:00:42.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:42.933+0000 7f12a2498700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f129c0f4b80 0x7f129c106790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:00:42.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:42.933+0000 7f12a2498700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f129c106c80 con 0x7f129c0f4b80 2024-02-19T22:00:42.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:42.933+0000 7f129bfff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f129c0f4b80 0x7f129c106790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:00:42.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:42.933+0000 7f129bfff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f129c0f4b80 0x7f129c106790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:38472/0 (socket says 172.21.15.86:38472) 2024-02-19T22:00:42.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:42.933+0000 7f129bfff700 1 -- 172.21.15.86:0/2531202264 learned_addr learned my addr 172.21.15.86:0/2531202264 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T22:00:42.936 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:42.934+0000 7f129bfff700 1 -- 172.21.15.86:0/2531202264 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1290005040 con 0x7f129c0f4b80 2024-02-19T22:00:42.936 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:42.934+0000 7f129bfff700 1 --2- 172.21.15.86:0/2531202264 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f129c0f4b80 0x7f129c106790 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7f129000aaa0 tx=0x7f129000ac40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:00:42.936 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:42.934+0000 7f1298ff9700 1 -- 172.21.15.86:0/2531202264 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1290019070 con 0x7f129c0f4b80 2024-02-19T22:00:42.936 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:42.934+0000 7f12a2498700 1 -- 172.21.15.86:0/2531202264 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f129c106e80 con 0x7f129c0f4b80 2024-02-19T22:00:42.936 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:42.934+0000 7f1298ff9700 1 -- 172.21.15.86:0/2531202264 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f12900148d0 con 0x7f129c0f4b80 2024-02-19T22:00:42.936 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:42.934+0000 7f1298ff9700 1 -- 172.21.15.86:0/2531202264 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1290008480 con 0x7f129c0f4b80 2024-02-19T22:00:42.936 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:42.934+0000 7f12a2498700 1 -- 172.21.15.86:0/2531202264 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f129c107310 con 0x7f129c0f4b80 2024-02-19T22:00:42.937 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:42.935+0000 7f1298ff9700 1 -- 172.21.15.86:0/2531202264 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1290025070 con 0x7f129c0f4b80 2024-02-19T22:00:42.937 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:42.935+0000 7f12a2498700 1 -- 172.21.15.86:0/2531202264 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f129c045ad0 con 0x7f129c0f4b80 2024-02-19T22:00:42.937 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:42.935+0000 7f1298ff9700 1 --2- 172.21.15.86:0/2531202264 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f128402ff60 0x7f1284032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:00:42.937 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:42.935+0000 7f1298ff9700 1 -- 172.21.15.86:0/2531202264 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f12900406a0 con 0x7f129c0f4b80 2024-02-19T22:00:42.937 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:42.935+0000 7f129b7fe700 1 --2- 172.21.15.86:0/2531202264 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f128402ff60 0x7f1284032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:00:42.937 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:42.936+0000 7f129b7fe700 1 --2- 172.21.15.86:0/2531202264 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f128402ff60 0x7f1284032420 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f128c0096a0 tx=0x7f128c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:00:42.939 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:42.940+0000 7f1298ff9700 1 -- 172.21.15.86:0/2531202264 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f129001b2c0 con 0x7f129c0f4b80 2024-02-19T22:00:43.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:43.146+0000 7f12a2498700 1 -- 172.21.15.86:0/2531202264 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f129c108160 con 0x7f129c0f4b80 2024-02-19T22:00:43.147 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:43.147+0000 7f1298ff9700 1 -- 172.21.15.86:0/2531202264 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f129001b2c0 con 0x7f129c0f4b80 2024-02-19T22:00:43.147 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T22:00:43.147 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:00:43.148 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:43.148+0000 7f12a2498700 1 -- 172.21.15.86:0/2531202264 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f128402ff60 msgr2=0x7f1284032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:00:43.148 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:43.148+0000 7f12a2498700 1 --2- 172.21.15.86:0/2531202264 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f128402ff60 0x7f1284032420 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f128c0096a0 tx=0x7f128c006b40).stop 2024-02-19T22:00:43.148 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:43.148+0000 7f12a2498700 1 -- 172.21.15.86:0/2531202264 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f129c0f4b80 msgr2=0x7f129c106790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:00:43.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:43.148+0000 7f12a2498700 1 --2- 172.21.15.86:0/2531202264 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f129c0f4b80 0x7f129c106790 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7f129000aaa0 tx=0x7f129000ac40).stop 2024-02-19T22:00:43.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:43.148+0000 7f12a2498700 1 -- 172.21.15.86:0/2531202264 shutdown_connections 2024-02-19T22:00:43.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:43.148+0000 7f12a2498700 1 --2- 172.21.15.86:0/2531202264 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f128402ff60 0x7f1284032420 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:00:43.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:43.148+0000 7f12a2498700 1 --2- 172.21.15.86:0/2531202264 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f129c0f4b80 0x7f129c106790 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:00:43.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:43.148+0000 7f12a2498700 1 -- 172.21.15.86:0/2531202264 >> 172.21.15.86:0/2531202264 conn(0x7f129c0f02e0 msgr2=0x7f129c0f0d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:00:43.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:43.149+0000 7f12a2498700 1 -- 172.21.15.86:0/2531202264 shutdown_connections 2024-02-19T22:00:43.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:43.149+0000 7f12a2498700 1 -- 172.21.15.86:0/2531202264 wait complete. 2024-02-19T22:00:43.149 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T22:00:43.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:00:43 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:00:43.148329+0000 mon.smithi053 (mon.0) 181 : audit [DBG] from='client.? 172.21.15.86:0/2531202264' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:00:44.744 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:00:44.744 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:00:44.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:00:44 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:00:43.473962+0000 mgr.smithi053.fyhcdi (mgr.14162) 125 : cluster [DBG] pgmap v118: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:00:46.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:46.543+0000 7fea3f59e700 1 -- 172.21.15.86:0/2156578762 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fea400f62a0 msgr2=0x7fea400f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:00:46.547 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:46.543+0000 7fea3f59e700 1 --2- 172.21.15.86:0/2156578762 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fea400f62a0 0x7fea400f66c0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 rx=0x7fea30009fb0 tx=0x7fea30009b30).stop 2024-02-19T22:00:46.547 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:46.544+0000 7fea3f59e700 1 -- 172.21.15.86:0/2156578762 shutdown_connections 2024-02-19T22:00:46.547 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:46.544+0000 7fea3f59e700 1 --2- 172.21.15.86:0/2156578762 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fea400f62a0 0x7fea400f66c0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:00:46.547 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:46.544+0000 7fea3f59e700 1 -- 172.21.15.86:0/2156578762 >> 172.21.15.86:0/2156578762 conn(0x7fea400f1aa0 msgr2=0x7fea400f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:00:46.547 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:46.544+0000 7fea3f59e700 1 -- 172.21.15.86:0/2156578762 shutdown_connections 2024-02-19T22:00:46.547 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:46.544+0000 7fea3f59e700 1 -- 172.21.15.86:0/2156578762 wait complete. 2024-02-19T22:00:46.547 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:46.544+0000 7fea3f59e700 1 Processor -- start 2024-02-19T22:00:46.547 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:46.544+0000 7fea3f59e700 1 -- start start 2024-02-19T22:00:46.547 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:46.544+0000 7fea3f59e700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fea400f62a0 0x7fea4006b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:00:46.547 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:46.544+0000 7fea3f59e700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea40069e20 con 0x7fea400f62a0 2024-02-19T22:00:46.547 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:46.544+0000 7fea3e59c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fea400f62a0 0x7fea4006b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:00:46.548 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:46.544+0000 7fea3e59c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fea400f62a0 0x7fea4006b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:38500/0 (socket says 172.21.15.86:38500) 2024-02-19T22:00:46.548 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:46.544+0000 7fea3e59c700 1 -- 172.21.15.86:0/3696896042 learned_addr learned my addr 172.21.15.86:0/3696896042 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T22:00:46.548 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:46.545+0000 7fea3e59c700 1 -- 172.21.15.86:0/3696896042 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea30005040 con 0x7fea400f62a0 2024-02-19T22:00:46.548 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:46.545+0000 7fea3e59c700 1 --2- 172.21.15.86:0/3696896042 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fea400f62a0 0x7fea4006b9a0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7fea30009830 tx=0x7fea3000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:00:46.548 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:46.545+0000 7fea36ffd700 1 -- 172.21.15.86:0/3696896042 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fea30009320 con 0x7fea400f62a0 2024-02-19T22:00:46.548 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:46.545+0000 7fea3f59e700 1 -- 172.21.15.86:0/3696896042 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea40069fc0 con 0x7fea400f62a0 2024-02-19T22:00:46.548 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:46.545+0000 7fea36ffd700 1 -- 172.21.15.86:0/3696896042 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fea30013380 con 0x7fea400f62a0 2024-02-19T22:00:46.548 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:46.545+0000 7fea36ffd700 1 -- 172.21.15.86:0/3696896042 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fea30014420 con 0x7fea400f62a0 2024-02-19T22:00:46.548 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:46.545+0000 7fea3f59e700 1 -- 172.21.15.86:0/3696896042 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea4006a450 con 0x7fea400f62a0 2024-02-19T22:00:46.548 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:46.546+0000 7fea36ffd700 1 -- 172.21.15.86:0/3696896042 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fea3001e420 con 0x7fea400f62a0 2024-02-19T22:00:46.548 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:46.546+0000 7fea3f59e700 1 -- 172.21.15.86:0/3696896042 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea40059100 con 0x7fea400f62a0 2024-02-19T22:00:46.548 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:46.546+0000 7fea36ffd700 1 --2- 172.21.15.86:0/3696896042 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fea2c02ff60 0x7fea2c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:00:46.548 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:46.546+0000 7fea36ffd700 1 -- 172.21.15.86:0/3696896042 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fea30040e30 con 0x7fea400f62a0 2024-02-19T22:00:46.548 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:46.546+0000 7fea3dd9b700 1 --2- 172.21.15.86:0/3696896042 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fea2c02ff60 0x7fea2c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:00:46.549 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:46.547+0000 7fea3dd9b700 1 --2- 172.21.15.86:0/3696896042 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fea2c02ff60 0x7fea2c032420 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7fea280096a0 tx=0x7fea28006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:00:46.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:46.552+0000 7fea36ffd700 1 -- 172.21.15.86:0/3696896042 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fea3001cb10 con 0x7fea400f62a0 2024-02-19T22:00:46.759 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:46.758+0000 7fea3f59e700 1 -- 172.21.15.86:0/3696896042 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fea40045ad0 con 0x7fea400f62a0 2024-02-19T22:00:46.760 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:46.758+0000 7fea36ffd700 1 -- 172.21.15.86:0/3696896042 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fea30004490 con 0x7fea400f62a0 2024-02-19T22:00:46.760 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T22:00:46.760 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:00:46.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:46.760+0000 7fea3f59e700 1 -- 172.21.15.86:0/3696896042 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fea2c02ff60 msgr2=0x7fea2c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:00:46.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:46.760+0000 7fea3f59e700 1 --2- 172.21.15.86:0/3696896042 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fea2c02ff60 0x7fea2c032420 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7fea280096a0 tx=0x7fea28006b40).stop 2024-02-19T22:00:46.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:46.760+0000 7fea3f59e700 1 -- 172.21.15.86:0/3696896042 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fea400f62a0 msgr2=0x7fea4006b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:00:46.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:46.760+0000 7fea3f59e700 1 --2- 172.21.15.86:0/3696896042 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fea400f62a0 0x7fea4006b9a0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7fea30009830 tx=0x7fea3000b040).stop 2024-02-19T22:00:46.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:46.760+0000 7fea3f59e700 1 -- 172.21.15.86:0/3696896042 shutdown_connections 2024-02-19T22:00:46.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:46.760+0000 7fea3f59e700 1 --2- 172.21.15.86:0/3696896042 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fea2c02ff60 0x7fea2c032420 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:00:46.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:46.760+0000 7fea3f59e700 1 --2- 172.21.15.86:0/3696896042 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fea400f62a0 0x7fea4006b9a0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:00:46.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:46.760+0000 7fea3f59e700 1 -- 172.21.15.86:0/3696896042 >> 172.21.15.86:0/3696896042 conn(0x7fea400f1aa0 msgr2=0x7fea400f2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:00:46.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:46.760+0000 7fea3f59e700 1 -- 172.21.15.86:0/3696896042 shutdown_connections 2024-02-19T22:00:46.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:46.760+0000 7fea3f59e700 1 -- 172.21.15.86:0/3696896042 wait complete. 2024-02-19T22:00:46.761 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T22:00:46.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:00:46 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:00:45.474287+0000 mgr.smithi053.fyhcdi (mgr.14162) 126 : cluster [DBG] pgmap v119: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:00:47.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:00:47 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:00:46.759749+0000 mon.smithi053 (mon.0) 182 : audit [DBG] from='client.? 172.21.15.86:0/3696896042' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:00:48.400 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:00:48.400 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:00:48.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:00:48 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:00:47.474641+0000 mgr.smithi053.fyhcdi (mgr.14162) 127 : cluster [DBG] pgmap v120: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:00:50.323 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:50.278+0000 7f74e1ce2700 1 -- 172.21.15.86:0/2365315721 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f74dc0f5410 msgr2=0x7f74dc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:00:50.323 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:50.278+0000 7f74e1ce2700 1 --2- 172.21.15.86:0/2365315721 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f74dc0f5410 0x7f74dc0f5830 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0x7f74cc009fb0 tx=0x7f74cc009b30).stop 2024-02-19T22:00:50.324 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:50.278+0000 7f74e1ce2700 1 -- 172.21.15.86:0/2365315721 shutdown_connections 2024-02-19T22:00:50.324 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:50.278+0000 7f74e1ce2700 1 --2- 172.21.15.86:0/2365315721 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f74dc0f5410 0x7f74dc0f5830 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:00:50.324 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:50.278+0000 7f74e1ce2700 1 -- 172.21.15.86:0/2365315721 >> 172.21.15.86:0/2365315721 conn(0x7f74dc0f0b70 msgr2=0x7f74dc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:00:50.324 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:50.278+0000 7f74e1ce2700 1 -- 172.21.15.86:0/2365315721 shutdown_connections 2024-02-19T22:00:50.324 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:50.278+0000 7f74e1ce2700 1 -- 172.21.15.86:0/2365315721 wait complete. 2024-02-19T22:00:50.324 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:50.278+0000 7f74e1ce2700 1 Processor -- start 2024-02-19T22:00:50.324 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:50.278+0000 7f74e1ce2700 1 -- start start 2024-02-19T22:00:50.324 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:50.278+0000 7f74e1ce2700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f74dc0f5410 0x7f74dc1094a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:00:50.324 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:50.279+0000 7f74e1ce2700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74dc060280 con 0x7f74dc0f5410 2024-02-19T22:00:50.324 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:50.279+0000 7f74db7fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f74dc0f5410 0x7f74dc1094a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:00:50.324 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:50.279+0000 7f74db7fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f74dc0f5410 0x7f74dc1094a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:43188/0 (socket says 172.21.15.86:43188) 2024-02-19T22:00:50.324 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:50.279+0000 7f74db7fe700 1 -- 172.21.15.86:0/413103609 learned_addr learned my addr 172.21.15.86:0/413103609 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T22:00:50.325 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:50.279+0000 7f74db7fe700 1 -- 172.21.15.86:0/413103609 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74cc005040 con 0x7f74dc0f5410 2024-02-19T22:00:50.325 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:50.279+0000 7f74db7fe700 1 --2- 172.21.15.86:0/413103609 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f74dc0f5410 0x7f74dc1094a0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7f74cc005010 tx=0x7f74cc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:00:50.325 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:50.279+0000 7f74c3fff700 1 -- 172.21.15.86:0/413103609 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f74cc009320 con 0x7f74dc0f5410 2024-02-19T22:00:50.325 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:50.279+0000 7f74c3fff700 1 -- 172.21.15.86:0/413103609 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f74cc013380 con 0x7f74dc0f5410 2024-02-19T22:00:50.325 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:50.279+0000 7f74c3fff700 1 -- 172.21.15.86:0/413103609 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f74cc014e50 con 0x7f74dc0f5410 2024-02-19T22:00:50.325 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:50.279+0000 7f74e1ce2700 1 -- 172.21.15.86:0/413103609 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74dc060480 con 0x7f74dc0f5410 2024-02-19T22:00:50.325 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:50.280+0000 7f74e1ce2700 1 -- 172.21.15.86:0/413103609 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74dc109bd0 con 0x7f74dc0f5410 2024-02-19T22:00:50.325 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:50.280+0000 7f74c3fff700 1 -- 172.21.15.86:0/413103609 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f74cc0095f0 con 0x7f74dc0f5410 2024-02-19T22:00:50.325 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:50.280+0000 7f74e1ce2700 1 -- 172.21.15.86:0/413103609 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74dc045ad0 con 0x7f74dc0f5410 2024-02-19T22:00:50.325 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:50.280+0000 7f74c3fff700 1 --2- 172.21.15.86:0/413103609 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f74c402ff10 0x7f74c40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:00:50.325 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:50.280+0000 7f74c3fff700 1 -- 172.21.15.86:0/413103609 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f74cc040f70 con 0x7f74dc0f5410 2024-02-19T22:00:50.325 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:50.281+0000 7f74daffd700 1 --2- 172.21.15.86:0/413103609 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f74c402ff10 0x7f74c40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:00:50.325 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:50.281+0000 7f74daffd700 1 --2- 172.21.15.86:0/413103609 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f74c402ff10 0x7f74c40323d0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7f74d00096a0 tx=0x7f74d0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:00:50.325 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:50.285+0000 7f74c3fff700 1 -- 172.21.15.86:0/413103609 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f74cc014420 con 0x7f74dc0f5410 2024-02-19T22:00:50.488 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:50.489+0000 7f74e1ce2700 1 -- 172.21.15.86:0/413103609 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f74dc0fafa0 con 0x7f74dc0f5410 2024-02-19T22:00:50.489 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:50.490+0000 7f74c3fff700 1 -- 172.21.15.86:0/413103609 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f74cc01d980 con 0x7f74dc0f5410 2024-02-19T22:00:50.490 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T22:00:50.490 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:00:50.491 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:50.491+0000 7f74e1ce2700 1 -- 172.21.15.86:0/413103609 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f74c402ff10 msgr2=0x7f74c40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:00:50.491 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:50.491+0000 7f74e1ce2700 1 --2- 172.21.15.86:0/413103609 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f74c402ff10 0x7f74c40323d0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7f74d00096a0 tx=0x7f74d0006b40).stop 2024-02-19T22:00:50.491 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:50.491+0000 7f74e1ce2700 1 -- 172.21.15.86:0/413103609 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f74dc0f5410 msgr2=0x7f74dc1094a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:00:50.492 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:50.491+0000 7f74e1ce2700 1 --2- 172.21.15.86:0/413103609 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f74dc0f5410 0x7f74dc1094a0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7f74cc005010 tx=0x7f74cc00b040).stop 2024-02-19T22:00:50.492 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:50.491+0000 7f74e1ce2700 1 -- 172.21.15.86:0/413103609 shutdown_connections 2024-02-19T22:00:50.492 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:50.491+0000 7f74e1ce2700 1 --2- 172.21.15.86:0/413103609 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f74c402ff10 0x7f74c40323d0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:00:50.492 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:50.491+0000 7f74e1ce2700 1 --2- 172.21.15.86:0/413103609 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f74dc0f5410 0x7f74dc1094a0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:00:50.492 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:50.491+0000 7f74e1ce2700 1 -- 172.21.15.86:0/413103609 >> 172.21.15.86:0/413103609 conn(0x7f74dc0f0b70 msgr2=0x7f74dc0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:00:50.493 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:50.491+0000 7f74e1ce2700 1 -- 172.21.15.86:0/413103609 shutdown_connections 2024-02-19T22:00:50.493 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:50.491+0000 7f74e1ce2700 1 -- 172.21.15.86:0/413103609 wait complete. 2024-02-19T22:00:50.493 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T22:00:50.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:00:50 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:00:49.475006+0000 mgr.smithi053.fyhcdi (mgr.14162) 128 : cluster [DBG] pgmap v121: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:00:51.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:00:51 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:00:50.490047+0000 mon.smithi053 (mon.0) 183 : audit [DBG] from='client.? 172.21.15.86:0/413103609' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:00:52.191 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:00:52.191 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:00:52.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:00:52 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:00:51.475223+0000 mgr.smithi053.fyhcdi (mgr.14162) 129 : cluster [DBG] pgmap v122: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:00:54.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:54.102+0000 7f846e3f4700 1 -- 172.21.15.86:0/3166572370 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f84680f51a0 msgr2=0x7f84680f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:00:54.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:54.102+0000 7f846e3f4700 1 --2- 172.21.15.86:0/3166572370 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f84680f51a0 0x7f84680f55c0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0x7f8464009fb0 tx=0x7f8464009b30).stop 2024-02-19T22:00:54.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:54.102+0000 7f846e3f4700 1 -- 172.21.15.86:0/3166572370 shutdown_connections 2024-02-19T22:00:54.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:54.102+0000 7f846e3f4700 1 --2- 172.21.15.86:0/3166572370 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f84680f51a0 0x7f84680f55c0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:00:54.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:54.102+0000 7f846e3f4700 1 -- 172.21.15.86:0/3166572370 >> 172.21.15.86:0/3166572370 conn(0x7f84680f09a0 msgr2=0x7f84680f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:00:54.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:54.102+0000 7f846e3f4700 1 -- 172.21.15.86:0/3166572370 shutdown_connections 2024-02-19T22:00:54.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:54.102+0000 7f846e3f4700 1 -- 172.21.15.86:0/3166572370 wait complete. 2024-02-19T22:00:54.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:54.102+0000 7f846e3f4700 1 Processor -- start 2024-02-19T22:00:54.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:54.102+0000 7f846e3f4700 1 -- start start 2024-02-19T22:00:54.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:54.102+0000 7f846e3f4700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f84680f51a0 0x7f84681092c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:00:54.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:54.102+0000 7f846e3f4700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8468060220 con 0x7f84680f51a0 2024-02-19T22:00:54.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:54.103+0000 7f846d3f2700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f84680f51a0 0x7f84681092c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:00:54.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:54.103+0000 7f846d3f2700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f84680f51a0 0x7f84681092c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:43212/0 (socket says 172.21.15.86:43212) 2024-02-19T22:00:54.107 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:54.103+0000 7f846d3f2700 1 -- 172.21.15.86:0/4005741371 learned_addr learned my addr 172.21.15.86:0/4005741371 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T22:00:54.107 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:54.103+0000 7f846d3f2700 1 -- 172.21.15.86:0/4005741371 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8464005040 con 0x7f84680f51a0 2024-02-19T22:00:54.107 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:54.103+0000 7f846d3f2700 1 --2- 172.21.15.86:0/4005741371 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f84680f51a0 0x7f84681092c0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7f84640089f0 tx=0x7f846400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:00:54.107 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:54.103+0000 7f845dffb700 1 -- 172.21.15.86:0/4005741371 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8464009320 con 0x7f84680f51a0 2024-02-19T22:00:54.107 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:54.103+0000 7f846e3f4700 1 -- 172.21.15.86:0/4005741371 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8468060420 con 0x7f84680f51a0 2024-02-19T22:00:54.107 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:54.103+0000 7f845dffb700 1 -- 172.21.15.86:0/4005741371 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8464013380 con 0x7f84680f51a0 2024-02-19T22:00:54.107 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:54.103+0000 7f845dffb700 1 -- 172.21.15.86:0/4005741371 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8464014e20 con 0x7f84680f51a0 2024-02-19T22:00:54.107 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:54.104+0000 7f846e3f4700 1 -- 172.21.15.86:0/4005741371 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8468109970 con 0x7f84680f51a0 2024-02-19T22:00:54.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:54.104+0000 7f845dffb700 1 -- 172.21.15.86:0/4005741371 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f84640095f0 con 0x7f84680f51a0 2024-02-19T22:00:54.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:54.104+0000 7f846e3f4700 1 -- 172.21.15.86:0/4005741371 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8468059100 con 0x7f84680f51a0 2024-02-19T22:00:54.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:54.104+0000 7f845dffb700 1 --2- 172.21.15.86:0/4005741371 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f845402ff10 0x7f84540323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:00:54.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:54.104+0000 7f845dffb700 1 -- 172.21.15.86:0/4005741371 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8464040ed0 con 0x7f84680f51a0 2024-02-19T22:00:54.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:54.105+0000 7f846cbf1700 1 --2- 172.21.15.86:0/4005741371 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f845402ff10 0x7f84540323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:00:54.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:54.105+0000 7f846cbf1700 1 --2- 172.21.15.86:0/4005741371 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f845402ff10 0x7f84540323d0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7f84580096a0 tx=0x7f8458006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:00:54.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:54.109+0000 7f845dffb700 1 -- 172.21.15.86:0/4005741371 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8464014420 con 0x7f84680f51a0 2024-02-19T22:00:54.314 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:54.314+0000 7f846e3f4700 1 -- 172.21.15.86:0/4005741371 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8468045ad0 con 0x7f84680f51a0 2024-02-19T22:00:54.315 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:54.315+0000 7f845dffb700 1 -- 172.21.15.86:0/4005741371 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f846401d980 con 0x7f84680f51a0 2024-02-19T22:00:54.315 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T22:00:54.315 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:00:54.316 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:54.316+0000 7f846e3f4700 1 -- 172.21.15.86:0/4005741371 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f845402ff10 msgr2=0x7f84540323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:00:54.316 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:54.316+0000 7f846e3f4700 1 --2- 172.21.15.86:0/4005741371 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f845402ff10 0x7f84540323d0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7f84580096a0 tx=0x7f8458006b40).stop 2024-02-19T22:00:54.316 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:54.316+0000 7f846e3f4700 1 -- 172.21.15.86:0/4005741371 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f84680f51a0 msgr2=0x7f84681092c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:00:54.316 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:54.316+0000 7f846e3f4700 1 --2- 172.21.15.86:0/4005741371 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f84680f51a0 0x7f84681092c0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7f84640089f0 tx=0x7f846400b040).stop 2024-02-19T22:00:54.316 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:54.316+0000 7f846e3f4700 1 -- 172.21.15.86:0/4005741371 shutdown_connections 2024-02-19T22:00:54.317 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:54.316+0000 7f846e3f4700 1 --2- 172.21.15.86:0/4005741371 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f845402ff10 0x7f84540323d0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:00:54.317 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:54.316+0000 7f846e3f4700 1 --2- 172.21.15.86:0/4005741371 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f84680f51a0 0x7f84681092c0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:00:54.317 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:54.316+0000 7f846e3f4700 1 -- 172.21.15.86:0/4005741371 >> 172.21.15.86:0/4005741371 conn(0x7f84680f09a0 msgr2=0x7f84680f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:00:54.317 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:54.316+0000 7f846e3f4700 1 -- 172.21.15.86:0/4005741371 shutdown_connections 2024-02-19T22:00:54.317 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:54.316+0000 7f846e3f4700 1 -- 172.21.15.86:0/4005741371 wait complete. 2024-02-19T22:00:54.317 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T22:00:54.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:00:54 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:00:53.475551+0000 mgr.smithi053.fyhcdi (mgr.14162) 130 : cluster [DBG] pgmap v123: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:00:54.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:00:54 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:00:54.316190+0000 mon.smithi053 (mon.0) 184 : audit [DBG] from='client.? 172.21.15.86:0/4005741371' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:00:55.967 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:00:55.967 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:00:56.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:00:56 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:00:55.475823+0000 mgr.smithi053.fyhcdi (mgr.14162) 131 : cluster [DBG] pgmap v124: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:00:57.844 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:57.842+0000 7fd3a7fb4700 1 -- 172.21.15.86:0/1091204810 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd3a00f5410 msgr2=0x7fd3a00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:00:57.844 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:57.842+0000 7fd3a7fb4700 1 --2- 172.21.15.86:0/1091204810 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd3a00f5410 0x7fd3a00f5830 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 rx=0x7fd390009fb0 tx=0x7fd390009b30).stop 2024-02-19T22:00:57.844 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:57.842+0000 7fd3a7fb4700 1 -- 172.21.15.86:0/1091204810 shutdown_connections 2024-02-19T22:00:57.844 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:57.842+0000 7fd3a7fb4700 1 --2- 172.21.15.86:0/1091204810 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd3a00f5410 0x7fd3a00f5830 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:00:57.844 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:57.842+0000 7fd3a7fb4700 1 -- 172.21.15.86:0/1091204810 >> 172.21.15.86:0/1091204810 conn(0x7fd3a00f0b70 msgr2=0x7fd3a00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:00:57.844 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:57.842+0000 7fd3a7fb4700 1 -- 172.21.15.86:0/1091204810 shutdown_connections 2024-02-19T22:00:57.844 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:57.842+0000 7fd3a7fb4700 1 -- 172.21.15.86:0/1091204810 wait complete. 2024-02-19T22:00:57.844 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:57.843+0000 7fd3a7fb4700 1 Processor -- start 2024-02-19T22:00:57.844 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:57.843+0000 7fd3a7fb4700 1 -- start start 2024-02-19T22:00:57.845 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:57.843+0000 7fd3a7fb4700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd3a00f5410 0x7fd3a01056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:00:57.845 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:57.843+0000 7fd3a7fb4700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3a0105bd0 con 0x7fd3a00f5410 2024-02-19T22:00:57.845 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:57.843+0000 7fd3a5d50700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd3a00f5410 0x7fd3a01056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:00:57.845 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:57.843+0000 7fd3a5d50700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd3a00f5410 0x7fd3a01056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:43220/0 (socket says 172.21.15.86:43220) 2024-02-19T22:00:57.845 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:57.843+0000 7fd3a5d50700 1 -- 172.21.15.86:0/3284811125 learned_addr learned my addr 172.21.15.86:0/3284811125 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T22:00:57.845 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:57.843+0000 7fd3a5d50700 1 -- 172.21.15.86:0/3284811125 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd390005040 con 0x7fd3a00f5410 2024-02-19T22:00:57.845 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:57.843+0000 7fd3a5d50700 1 --2- 172.21.15.86:0/3284811125 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd3a00f5410 0x7fd3a01056e0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7fd3900089f0 tx=0x7fd39000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:00:57.845 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:57.844+0000 7fd3967fc700 1 -- 172.21.15.86:0/3284811125 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd390009320 con 0x7fd3a00f5410 2024-02-19T22:00:57.845 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:57.844+0000 7fd3967fc700 1 -- 172.21.15.86:0/3284811125 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd390013380 con 0x7fd3a00f5410 2024-02-19T22:00:57.845 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:57.844+0000 7fd3a7fb4700 1 -- 172.21.15.86:0/3284811125 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3a0105dd0 con 0x7fd3a00f5410 2024-02-19T22:00:57.845 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:57.844+0000 7fd3967fc700 1 -- 172.21.15.86:0/3284811125 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd390020d80 con 0x7fd3a00f5410 2024-02-19T22:00:57.845 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:57.844+0000 7fd3a7fb4700 1 -- 172.21.15.86:0/3284811125 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3a0106260 con 0x7fd3a00f5410 2024-02-19T22:00:57.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:57.844+0000 7fd3967fc700 1 -- 172.21.15.86:0/3284811125 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd390020420 con 0x7fd3a00f5410 2024-02-19T22:00:57.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:57.844+0000 7fd3a7fb4700 1 -- 172.21.15.86:0/3284811125 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3a0045ad0 con 0x7fd3a00f5410 2024-02-19T22:00:57.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:57.845+0000 7fd3967fc700 1 --2- 172.21.15.86:0/3284811125 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fd38c02ff60 0x7fd38c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:00:57.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:57.845+0000 7fd3967fc700 1 -- 172.21.15.86:0/3284811125 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd390042c20 con 0x7fd3a00f5410 2024-02-19T22:00:57.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:57.845+0000 7fd3a554f700 1 --2- 172.21.15.86:0/3284811125 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fd38c02ff60 0x7fd38c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:00:57.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:57.845+0000 7fd3a554f700 1 --2- 172.21.15.86:0/3284811125 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fd38c02ff60 0x7fd38c032420 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7fd39c0096a0 tx=0x7fd39c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:00:57.849 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:57.849+0000 7fd3967fc700 1 -- 172.21.15.86:0/3284811125 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd3900095f0 con 0x7fd3a00f5410 2024-02-19T22:00:58.055 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:58.055+0000 7fd3a7fb4700 1 -- 172.21.15.86:0/3284811125 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd3a0107110 con 0x7fd3a00f5410 2024-02-19T22:00:58.055 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:58.055+0000 7fd3967fc700 1 -- 172.21.15.86:0/3284811125 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd39002e2d0 con 0x7fd3a00f5410 2024-02-19T22:00:58.056 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T22:00:58.056 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:00:58.057 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:58.057+0000 7fd3a7fb4700 1 -- 172.21.15.86:0/3284811125 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fd38c02ff60 msgr2=0x7fd38c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:00:58.057 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:58.057+0000 7fd3a7fb4700 1 --2- 172.21.15.86:0/3284811125 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fd38c02ff60 0x7fd38c032420 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7fd39c0096a0 tx=0x7fd39c006b40).stop 2024-02-19T22:00:58.057 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:58.057+0000 7fd3a7fb4700 1 -- 172.21.15.86:0/3284811125 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd3a00f5410 msgr2=0x7fd3a01056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:00:58.057 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:58.057+0000 7fd3a7fb4700 1 --2- 172.21.15.86:0/3284811125 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd3a00f5410 0x7fd3a01056e0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7fd3900089f0 tx=0x7fd39000b040).stop 2024-02-19T22:00:58.057 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:58.057+0000 7fd3a7fb4700 1 -- 172.21.15.86:0/3284811125 shutdown_connections 2024-02-19T22:00:58.058 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:58.057+0000 7fd3a7fb4700 1 --2- 172.21.15.86:0/3284811125 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fd38c02ff60 0x7fd38c032420 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:00:58.058 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:58.057+0000 7fd3a7fb4700 1 --2- 172.21.15.86:0/3284811125 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd3a00f5410 0x7fd3a01056e0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:00:58.058 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:58.057+0000 7fd3a7fb4700 1 -- 172.21.15.86:0/3284811125 >> 172.21.15.86:0/3284811125 conn(0x7fd3a00f0b70 msgr2=0x7fd3a00f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:00:58.058 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:58.057+0000 7fd3a7fb4700 1 -- 172.21.15.86:0/3284811125 shutdown_connections 2024-02-19T22:00:58.058 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:00:58.057+0000 7fd3a7fb4700 1 -- 172.21.15.86:0/3284811125 wait complete. 2024-02-19T22:00:58.058 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T22:00:58.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:00:58 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:00:57.476105+0000 mgr.smithi053.fyhcdi (mgr.14162) 132 : cluster [DBG] pgmap v125: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:00:58.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:00:58 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:00:58.056779+0000 mon.smithi053 (mon.0) 185 : audit [DBG] from='client.? 172.21.15.86:0/3284811125' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:00:59.627 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:00:59.627 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:01:00.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:01:00 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:00:59.476444+0000 mgr.smithi053.fyhcdi (mgr.14162) 133 : cluster [DBG] pgmap v126: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:01:01.462 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:01.459+0000 7f4d2f3f9700 1 -- 172.21.15.86:0/2786941947 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4d280f62c0 msgr2=0x7f4d280f66e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:01:01.462 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:01.459+0000 7f4d2f3f9700 1 --2- 172.21.15.86:0/2786941947 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4d280f62c0 0x7f4d280f66e0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 rx=0x7f4d24009fb0 tx=0x7f4d24009b30).stop 2024-02-19T22:01:01.462 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:01.459+0000 7f4d2f3f9700 1 -- 172.21.15.86:0/2786941947 shutdown_connections 2024-02-19T22:01:01.462 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:01.459+0000 7f4d2f3f9700 1 --2- 172.21.15.86:0/2786941947 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4d280f62c0 0x7f4d280f66e0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:01:01.462 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:01.459+0000 7f4d2f3f9700 1 -- 172.21.15.86:0/2786941947 >> 172.21.15.86:0/2786941947 conn(0x7f4d280f1ac0 msgr2=0x7f4d280f3ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:01:01.462 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:01.459+0000 7f4d2f3f9700 1 -- 172.21.15.86:0/2786941947 shutdown_connections 2024-02-19T22:01:01.462 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:01.459+0000 7f4d2f3f9700 1 -- 172.21.15.86:0/2786941947 wait complete. 2024-02-19T22:01:01.462 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:01.459+0000 7f4d2f3f9700 1 Processor -- start 2024-02-19T22:01:01.462 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:01.460+0000 7f4d2f3f9700 1 -- start start 2024-02-19T22:01:01.462 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:01.460+0000 7f4d2f3f9700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4d280f62c0 0x7f4d2806b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:01:01.462 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:01.460+0000 7f4d2f3f9700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d28069e20 con 0x7f4d280f62c0 2024-02-19T22:01:01.462 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:01.460+0000 7f4d2e3f7700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4d280f62c0 0x7f4d2806b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:01:01.463 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:01.460+0000 7f4d2e3f7700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4d280f62c0 0x7f4d2806b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:38810/0 (socket says 172.21.15.86:38810) 2024-02-19T22:01:01.463 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:01.460+0000 7f4d2e3f7700 1 -- 172.21.15.86:0/1148844171 learned_addr learned my addr 172.21.15.86:0/1148844171 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T22:01:01.463 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:01.460+0000 7f4d2e3f7700 1 -- 172.21.15.86:0/1148844171 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d24005040 con 0x7f4d280f62c0 2024-02-19T22:01:01.463 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:01.460+0000 7f4d2e3f7700 1 --2- 172.21.15.86:0/1148844171 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4d280f62c0 0x7f4d2806b9a0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7f4d240089f0 tx=0x7f4d2400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:01:01.463 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:01.460+0000 7f4d1effd700 1 -- 172.21.15.86:0/1148844171 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4d24009320 con 0x7f4d280f62c0 2024-02-19T22:01:01.463 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:01.460+0000 7f4d2f3f9700 1 -- 172.21.15.86:0/1148844171 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d28069fc0 con 0x7f4d280f62c0 2024-02-19T22:01:01.463 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:01.461+0000 7f4d1effd700 1 -- 172.21.15.86:0/1148844171 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4d24013380 con 0x7f4d280f62c0 2024-02-19T22:01:01.463 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:01.461+0000 7f4d1effd700 1 -- 172.21.15.86:0/1148844171 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4d24014e20 con 0x7f4d280f62c0 2024-02-19T22:01:01.463 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:01.461+0000 7f4d2f3f9700 1 -- 172.21.15.86:0/1148844171 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d2806a450 con 0x7f4d280f62c0 2024-02-19T22:01:01.463 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:01.461+0000 7f4d1effd700 1 -- 172.21.15.86:0/1148844171 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4d240263a0 con 0x7f4d280f62c0 2024-02-19T22:01:01.463 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:01.461+0000 7f4d1effd700 1 --2- 172.21.15.86:0/1148844171 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f4d1402ff10 0x7f4d140323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:01:01.464 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:01.462+0000 7f4d1effd700 1 -- 172.21.15.86:0/1148844171 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4d24040ef0 con 0x7f4d280f62c0 2024-02-19T22:01:01.464 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:01.461+0000 7f4d2f3f9700 1 -- 172.21.15.86:0/1148844171 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d28059100 con 0x7f4d280f62c0 2024-02-19T22:01:01.464 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:01.462+0000 7f4d2dbf6700 1 --2- 172.21.15.86:0/1148844171 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f4d1402ff10 0x7f4d140323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:01:01.464 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:01.462+0000 7f4d2dbf6700 1 --2- 172.21.15.86:0/1148844171 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f4d1402ff10 0x7f4d140323d0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f4d180096a0 tx=0x7f4d18006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:01:01.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:01.466+0000 7f4d1effd700 1 -- 172.21.15.86:0/1148844171 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4d24012920 con 0x7f4d280f62c0 2024-02-19T22:01:01.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:01.677+0000 7f4d2f3f9700 1 -- 172.21.15.86:0/1148844171 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4d280fbe50 con 0x7f4d280f62c0 2024-02-19T22:01:01.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:01.677+0000 7f4d1effd700 1 -- 172.21.15.86:0/1148844171 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4d240126f0 con 0x7f4d280f62c0 2024-02-19T22:01:01.678 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T22:01:01.678 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:01:01.679 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:01.679+0000 7f4d2f3f9700 1 -- 172.21.15.86:0/1148844171 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f4d1402ff10 msgr2=0x7f4d140323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:01:01.679 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:01.679+0000 7f4d2f3f9700 1 --2- 172.21.15.86:0/1148844171 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f4d1402ff10 0x7f4d140323d0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f4d180096a0 tx=0x7f4d18006b40).stop 2024-02-19T22:01:01.679 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:01.679+0000 7f4d2f3f9700 1 -- 172.21.15.86:0/1148844171 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4d280f62c0 msgr2=0x7f4d2806b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:01:01.679 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:01.679+0000 7f4d2f3f9700 1 --2- 172.21.15.86:0/1148844171 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4d280f62c0 0x7f4d2806b9a0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7f4d240089f0 tx=0x7f4d2400b040).stop 2024-02-19T22:01:01.679 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:01.679+0000 7f4d2f3f9700 1 -- 172.21.15.86:0/1148844171 shutdown_connections 2024-02-19T22:01:01.679 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:01.679+0000 7f4d2f3f9700 1 --2- 172.21.15.86:0/1148844171 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f4d1402ff10 0x7f4d140323d0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:01:01.679 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:01.679+0000 7f4d2f3f9700 1 --2- 172.21.15.86:0/1148844171 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4d280f62c0 0x7f4d2806b9a0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:01:01.679 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:01.679+0000 7f4d2f3f9700 1 -- 172.21.15.86:0/1148844171 >> 172.21.15.86:0/1148844171 conn(0x7f4d280f1ac0 msgr2=0x7f4d280f24c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:01:01.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:01.679+0000 7f4d2f3f9700 1 -- 172.21.15.86:0/1148844171 shutdown_connections 2024-02-19T22:01:01.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:01.679+0000 7f4d2f3f9700 1 -- 172.21.15.86:0/1148844171 wait complete. 2024-02-19T22:01:01.680 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T22:01:02.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:01:02 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:01:01.476734+0000 mgr.smithi053.fyhcdi (mgr.14162) 134 : cluster [DBG] pgmap v127: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:01:02.794 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:01:02 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:01:01.678717+0000 mon.smithi053 (mon.0) 186 : audit [DBG] from='client.? 172.21.15.86:0/1148844171' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:01:03.357 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:01:03.357 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:01:04.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:01:04 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:01:03.477053+0000 mgr.smithi053.fyhcdi (mgr.14162) 135 : cluster [DBG] pgmap v128: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:01:05.230 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:05.228+0000 7fbd7b223700 1 -- 172.21.15.86:0/636641762 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbd740f5410 msgr2=0x7fbd740f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:01:05.230 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:05.228+0000 7fbd7b223700 1 --2- 172.21.15.86:0/636641762 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbd740f5410 0x7fbd740f5830 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 rx=0x7fbd68009fb0 tx=0x7fbd68009b30).stop 2024-02-19T22:01:05.230 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:05.229+0000 7fbd7b223700 1 -- 172.21.15.86:0/636641762 shutdown_connections 2024-02-19T22:01:05.230 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:05.229+0000 7fbd7b223700 1 --2- 172.21.15.86:0/636641762 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbd740f5410 0x7fbd740f5830 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:01:05.230 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:05.229+0000 7fbd7b223700 1 -- 172.21.15.86:0/636641762 >> 172.21.15.86:0/636641762 conn(0x7fbd740f0b70 msgr2=0x7fbd740f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:01:05.231 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:05.229+0000 7fbd7b223700 1 -- 172.21.15.86:0/636641762 shutdown_connections 2024-02-19T22:01:05.231 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:05.229+0000 7fbd7b223700 1 -- 172.21.15.86:0/636641762 wait complete. 2024-02-19T22:01:05.231 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:05.229+0000 7fbd7b223700 1 Processor -- start 2024-02-19T22:01:05.231 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:05.229+0000 7fbd7b223700 1 -- start start 2024-02-19T22:01:05.231 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:05.229+0000 7fbd7b223700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbd740f5410 0x7fbd740fd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:01:05.231 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:05.229+0000 7fbd7b223700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd740fb960 con 0x7fbd740f5410 2024-02-19T22:01:05.231 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:05.230+0000 7fbd78fbf700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbd740f5410 0x7fbd740fd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:01:05.231 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:05.230+0000 7fbd78fbf700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbd740f5410 0x7fbd740fd4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:38816/0 (socket says 172.21.15.86:38816) 2024-02-19T22:01:05.231 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:05.230+0000 7fbd78fbf700 1 -- 172.21.15.86:0/4028581564 learned_addr learned my addr 172.21.15.86:0/4028581564 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T22:01:05.231 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:05.230+0000 7fbd78fbf700 1 -- 172.21.15.86:0/4028581564 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd68005040 con 0x7fbd740f5410 2024-02-19T22:01:05.231 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:05.230+0000 7fbd78fbf700 1 --2- 172.21.15.86:0/4028581564 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbd740f5410 0x7fbd740fd4e0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7fbd68005010 tx=0x7fbd6800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:01:05.231 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:05.230+0000 7fbd717fa700 1 -- 172.21.15.86:0/4028581564 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbd68009320 con 0x7fbd740f5410 2024-02-19T22:01:05.231 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:05.230+0000 7fbd7b223700 1 -- 172.21.15.86:0/4028581564 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd740fbb00 con 0x7fbd740f5410 2024-02-19T22:01:05.231 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:05.230+0000 7fbd717fa700 1 -- 172.21.15.86:0/4028581564 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbd68013380 con 0x7fbd740f5410 2024-02-19T22:01:05.232 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:05.230+0000 7fbd717fa700 1 -- 172.21.15.86:0/4028581564 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbd68014e50 con 0x7fbd740f5410 2024-02-19T22:01:05.232 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:05.230+0000 7fbd7b223700 1 -- 172.21.15.86:0/4028581564 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd740fbf90 con 0x7fbd740f5410 2024-02-19T22:01:05.232 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:05.231+0000 7fbd717fa700 1 -- 172.21.15.86:0/4028581564 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbd680095f0 con 0x7fbd740f5410 2024-02-19T22:01:05.233 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:05.231+0000 7fbd7b223700 1 -- 172.21.15.86:0/4028581564 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd74059100 con 0x7fbd740f5410 2024-02-19T22:01:05.233 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:05.231+0000 7fbd717fa700 1 --2- 172.21.15.86:0/4028581564 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fbd6402ff10 0x7fbd640323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:01:05.233 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:05.231+0000 7fbd717fa700 1 -- 172.21.15.86:0/4028581564 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbd68040fa0 con 0x7fbd740f5410 2024-02-19T22:01:05.233 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:05.232+0000 7fbd73fff700 1 --2- 172.21.15.86:0/4028581564 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fbd6402ff10 0x7fbd640323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:01:05.233 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:05.232+0000 7fbd73fff700 1 --2- 172.21.15.86:0/4028581564 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fbd6402ff10 0x7fbd640323d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7fbd600096a0 tx=0x7fbd60006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:01:05.236 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:05.236+0000 7fbd717fa700 1 -- 172.21.15.86:0/4028581564 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbd68014420 con 0x7fbd740f5410 2024-02-19T22:01:05.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:05.441+0000 7fbd7b223700 1 -- 172.21.15.86:0/4028581564 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbd740fafa0 con 0x7fbd740f5410 2024-02-19T22:01:05.442 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:05.442+0000 7fbd717fa700 1 -- 172.21.15.86:0/4028581564 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fbd68010230 con 0x7fbd740f5410 2024-02-19T22:01:05.442 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T22:01:05.442 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:01:05.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:05.443+0000 7fbd7b223700 1 -- 172.21.15.86:0/4028581564 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fbd6402ff10 msgr2=0x7fbd640323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:01:05.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:05.443+0000 7fbd7b223700 1 --2- 172.21.15.86:0/4028581564 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fbd6402ff10 0x7fbd640323d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7fbd600096a0 tx=0x7fbd60006b40).stop 2024-02-19T22:01:05.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:05.443+0000 7fbd7b223700 1 -- 172.21.15.86:0/4028581564 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbd740f5410 msgr2=0x7fbd740fd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:01:05.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:05.443+0000 7fbd7b223700 1 --2- 172.21.15.86:0/4028581564 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbd740f5410 0x7fbd740fd4e0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7fbd68005010 tx=0x7fbd6800b040).stop 2024-02-19T22:01:05.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:05.444+0000 7fbd7b223700 1 -- 172.21.15.86:0/4028581564 shutdown_connections 2024-02-19T22:01:05.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:05.444+0000 7fbd7b223700 1 --2- 172.21.15.86:0/4028581564 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fbd6402ff10 0x7fbd640323d0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:01:05.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:05.444+0000 7fbd7b223700 1 --2- 172.21.15.86:0/4028581564 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbd740f5410 0x7fbd740fd4e0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:01:05.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:05.444+0000 7fbd7b223700 1 -- 172.21.15.86:0/4028581564 >> 172.21.15.86:0/4028581564 conn(0x7fbd740f0b70 msgr2=0x7fbd7405fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:01:05.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:05.444+0000 7fbd7b223700 1 -- 172.21.15.86:0/4028581564 shutdown_connections 2024-02-19T22:01:05.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:05.444+0000 7fbd7b223700 1 -- 172.21.15.86:0/4028581564 wait complete. 2024-02-19T22:01:05.444 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T22:01:05.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:01:05 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:01:05.443318+0000 mon.smithi053 (mon.0) 187 : audit [DBG] from='client.? 172.21.15.86:0/4028581564' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:01:06.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:01:06 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:01:05.477404+0000 mgr.smithi053.fyhcdi (mgr.14162) 136 : cluster [DBG] pgmap v129: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:01:07.089 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:01:07.090 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:01:08.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:01:08 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:01:07.477800+0000 mgr.smithi053.fyhcdi (mgr.14162) 137 : cluster [DBG] pgmap v130: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:01:08.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:08.964+0000 7f58d1e84700 1 -- 172.21.15.86:0/1123616759 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f58cc0f6510 msgr2=0x7f58cc0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:01:08.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:08.964+0000 7f58d1e84700 1 --2- 172.21.15.86:0/1123616759 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f58cc0f6510 0x7f58cc0f6930 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 rx=0x7f58c0009fb0 tx=0x7f58c0009b30).stop 2024-02-19T22:01:08.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:08.964+0000 7f58d1e84700 1 -- 172.21.15.86:0/1123616759 shutdown_connections 2024-02-19T22:01:08.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:08.964+0000 7f58d1e84700 1 --2- 172.21.15.86:0/1123616759 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f58cc0f6510 0x7f58cc0f6930 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:01:08.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:08.964+0000 7f58d1e84700 1 -- 172.21.15.86:0/1123616759 >> 172.21.15.86:0/1123616759 conn(0x7f58cc0f1c70 msgr2=0x7f58cc0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:01:08.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:08.964+0000 7f58d1e84700 1 -- 172.21.15.86:0/1123616759 shutdown_connections 2024-02-19T22:01:08.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:08.964+0000 7f58d1e84700 1 -- 172.21.15.86:0/1123616759 wait complete. 2024-02-19T22:01:08.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:08.965+0000 7f58d1e84700 1 Processor -- start 2024-02-19T22:01:08.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:08.965+0000 7f58d1e84700 1 -- start start 2024-02-19T22:01:08.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:08.965+0000 7f58d1e84700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f58cc0f6510 0x7f58cc1067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:01:08.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:08.965+0000 7f58d1e84700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58cc106cd0 con 0x7f58cc0f6510 2024-02-19T22:01:08.967 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:08.965+0000 7f58cb7fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f58cc0f6510 0x7f58cc1067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:01:08.967 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:08.965+0000 7f58cb7fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f58cc0f6510 0x7f58cc1067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:38844/0 (socket says 172.21.15.86:38844) 2024-02-19T22:01:08.967 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:08.965+0000 7f58cb7fe700 1 -- 172.21.15.86:0/3499237619 learned_addr learned my addr 172.21.15.86:0/3499237619 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T22:01:08.967 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:08.966+0000 7f58cb7fe700 1 -- 172.21.15.86:0/3499237619 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58c0005040 con 0x7f58cc0f6510 2024-02-19T22:01:08.968 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:08.966+0000 7f58cb7fe700 1 --2- 172.21.15.86:0/3499237619 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f58cc0f6510 0x7f58cc1067e0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7f58c0005010 tx=0x7f58c000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:01:08.968 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:08.966+0000 7f58b3fff700 1 -- 172.21.15.86:0/3499237619 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f58c0009320 con 0x7f58cc0f6510 2024-02-19T22:01:08.968 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:08.966+0000 7f58d1e84700 1 -- 172.21.15.86:0/3499237619 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58cc106ed0 con 0x7f58cc0f6510 2024-02-19T22:01:08.968 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:08.966+0000 7f58b3fff700 1 -- 172.21.15.86:0/3499237619 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f58c0013380 con 0x7f58cc0f6510 2024-02-19T22:01:08.968 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:08.966+0000 7f58b3fff700 1 -- 172.21.15.86:0/3499237619 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f58c0014e50 con 0x7f58cc0f6510 2024-02-19T22:01:08.968 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:08.966+0000 7f58d1e84700 1 -- 172.21.15.86:0/3499237619 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58cc107360 con 0x7f58cc0f6510 2024-02-19T22:01:08.968 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:08.967+0000 7f58b3fff700 1 -- 172.21.15.86:0/3499237619 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f58c0014420 con 0x7f58cc0f6510 2024-02-19T22:01:08.968 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:08.967+0000 7f58d1e84700 1 -- 172.21.15.86:0/3499237619 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58cc045ad0 con 0x7f58cc0f6510 2024-02-19T22:01:08.968 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:08.967+0000 7f58b3fff700 1 --2- 172.21.15.86:0/3499237619 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f58b402ff10 0x7f58b40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:01:08.968 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:08.967+0000 7f58b3fff700 1 -- 172.21.15.86:0/3499237619 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f58c0040f50 con 0x7f58cc0f6510 2024-02-19T22:01:08.968 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:08.967+0000 7f58caffd700 1 --2- 172.21.15.86:0/3499237619 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f58b402ff10 0x7f58b40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:01:08.968 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:08.967+0000 7f58caffd700 1 --2- 172.21.15.86:0/3499237619 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f58b402ff10 0x7f58b40323d0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f58bc0096a0 tx=0x7f58bc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:01:08.971 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:08.971+0000 7f58b3fff700 1 -- 172.21.15.86:0/3499237619 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f58c0014790 con 0x7f58cc0f6510 2024-02-19T22:01:09.177 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:09.178+0000 7f58d1e84700 1 -- 172.21.15.86:0/3499237619 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f58cc108210 con 0x7f58cc0f6510 2024-02-19T22:01:09.178 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:09.178+0000 7f58b3fff700 1 -- 172.21.15.86:0/3499237619 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f58c001d380 con 0x7f58cc0f6510 2024-02-19T22:01:09.178 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T22:01:09.178 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:01:09.180 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:09.180+0000 7f58d1e84700 1 -- 172.21.15.86:0/3499237619 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f58b402ff10 msgr2=0x7f58b40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:01:09.180 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:09.180+0000 7f58d1e84700 1 --2- 172.21.15.86:0/3499237619 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f58b402ff10 0x7f58b40323d0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f58bc0096a0 tx=0x7f58bc006b40).stop 2024-02-19T22:01:09.180 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:09.180+0000 7f58d1e84700 1 -- 172.21.15.86:0/3499237619 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f58cc0f6510 msgr2=0x7f58cc1067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:01:09.180 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:09.180+0000 7f58d1e84700 1 --2- 172.21.15.86:0/3499237619 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f58cc0f6510 0x7f58cc1067e0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7f58c0005010 tx=0x7f58c000b040).stop 2024-02-19T22:01:09.180 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:09.180+0000 7f58d1e84700 1 -- 172.21.15.86:0/3499237619 shutdown_connections 2024-02-19T22:01:09.180 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:09.180+0000 7f58d1e84700 1 --2- 172.21.15.86:0/3499237619 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f58b402ff10 0x7f58b40323d0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:01:09.180 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:09.180+0000 7f58d1e84700 1 --2- 172.21.15.86:0/3499237619 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f58cc0f6510 0x7f58cc1067e0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:01:09.180 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:09.180+0000 7f58d1e84700 1 -- 172.21.15.86:0/3499237619 >> 172.21.15.86:0/3499237619 conn(0x7f58cc0f1c70 msgr2=0x7f58cc0f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:01:09.180 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:09.180+0000 7f58d1e84700 1 -- 172.21.15.86:0/3499237619 shutdown_connections 2024-02-19T22:01:09.180 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:09.180+0000 7f58d1e84700 1 -- 172.21.15.86:0/3499237619 wait complete. 2024-02-19T22:01:09.181 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T22:01:09.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:01:09 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:01:09.179679+0000 mon.smithi053 (mon.0) 188 : audit [DBG] from='client.? 172.21.15.86:0/3499237619' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:01:10.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:01:10 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:01:09.478223+0000 mgr.smithi053.fyhcdi (mgr.14162) 138 : cluster [DBG] pgmap v131: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:01:10.844 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:01:10.845 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:01:12.724 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:12.724+0000 7eff8e789700 1 -- 172.21.15.86:0/3421336538 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7eff880f62a0 msgr2=0x7eff880f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:01:12.724 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:12.724+0000 7eff8e789700 1 --2- 172.21.15.86:0/3421336538 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7eff880f62a0 0x7eff880f66c0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 rx=0x7eff84009fb0 tx=0x7eff84009b30).stop 2024-02-19T22:01:12.724 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:12.724+0000 7eff8e789700 1 -- 172.21.15.86:0/3421336538 shutdown_connections 2024-02-19T22:01:12.724 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:12.724+0000 7eff8e789700 1 --2- 172.21.15.86:0/3421336538 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7eff880f62a0 0x7eff880f66c0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:01:12.724 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:12.724+0000 7eff8e789700 1 -- 172.21.15.86:0/3421336538 >> 172.21.15.86:0/3421336538 conn(0x7eff880f1aa0 msgr2=0x7eff880f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:01:12.724 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:12.724+0000 7eff8e789700 1 -- 172.21.15.86:0/3421336538 shutdown_connections 2024-02-19T22:01:12.724 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:12.724+0000 7eff8e789700 1 -- 172.21.15.86:0/3421336538 wait complete. 2024-02-19T22:01:12.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:12.724+0000 7eff8e789700 1 Processor -- start 2024-02-19T22:01:12.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:12.724+0000 7eff8e789700 1 -- start start 2024-02-19T22:01:12.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:12.725+0000 7eff8e789700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7eff880f62a0 0x7eff8806b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:01:12.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:12.725+0000 7eff8e789700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff88069e20 con 0x7eff880f62a0 2024-02-19T22:01:12.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:12.725+0000 7eff8d787700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7eff880f62a0 0x7eff8806b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:01:12.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:12.725+0000 7eff8d787700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7eff880f62a0 0x7eff8806b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:59718/0 (socket says 172.21.15.86:59718) 2024-02-19T22:01:12.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:12.725+0000 7eff8d787700 1 -- 172.21.15.86:0/601086573 learned_addr learned my addr 172.21.15.86:0/601086573 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T22:01:12.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:12.725+0000 7eff8d787700 1 -- 172.21.15.86:0/601086573 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff84005040 con 0x7eff880f62a0 2024-02-19T22:01:12.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:12.725+0000 7eff8d787700 1 --2- 172.21.15.86:0/601086573 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7eff880f62a0 0x7eff8806b9a0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7eff840089f0 tx=0x7eff8400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:01:12.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:12.726+0000 7eff7dffb700 1 -- 172.21.15.86:0/601086573 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7eff84009320 con 0x7eff880f62a0 2024-02-19T22:01:12.727 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:12.726+0000 7eff8e789700 1 -- 172.21.15.86:0/601086573 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff88069fc0 con 0x7eff880f62a0 2024-02-19T22:01:12.727 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:12.726+0000 7eff7dffb700 1 -- 172.21.15.86:0/601086573 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7eff84013380 con 0x7eff880f62a0 2024-02-19T22:01:12.727 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:12.726+0000 7eff7dffb700 1 -- 172.21.15.86:0/601086573 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7eff84014e20 con 0x7eff880f62a0 2024-02-19T22:01:12.727 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:12.726+0000 7eff8e789700 1 -- 172.21.15.86:0/601086573 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff8806a450 con 0x7eff880f62a0 2024-02-19T22:01:12.728 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:12.727+0000 7eff7dffb700 1 -- 172.21.15.86:0/601086573 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7eff840095f0 con 0x7eff880f62a0 2024-02-19T22:01:12.728 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:12.727+0000 7eff8e789700 1 -- 172.21.15.86:0/601086573 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff88059100 con 0x7eff880f62a0 2024-02-19T22:01:12.728 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:12.727+0000 7eff7dffb700 1 --2- 172.21.15.86:0/601086573 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7eff7402ff10 0x7eff740323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:01:12.728 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:12.727+0000 7eff7dffb700 1 -- 172.21.15.86:0/601086573 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7eff84040ef0 con 0x7eff880f62a0 2024-02-19T22:01:12.728 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:12.727+0000 7eff8cf86700 1 --2- 172.21.15.86:0/601086573 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7eff7402ff10 0x7eff740323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:01:12.728 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:12.727+0000 7eff8cf86700 1 --2- 172.21.15.86:0/601086573 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7eff7402ff10 0x7eff740323d0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7eff780096a0 tx=0x7eff78006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:01:12.731 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:12.732+0000 7eff7dffb700 1 -- 172.21.15.86:0/601086573 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7eff84014420 con 0x7eff880f62a0 2024-02-19T22:01:12.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:01:12 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:01:11.478515+0000 mgr.smithi053.fyhcdi (mgr.14162) 139 : cluster [DBG] pgmap v132: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:01:12.936 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:12.936+0000 7eff8e789700 1 -- 172.21.15.86:0/601086573 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7eff880fbe30 con 0x7eff880f62a0 2024-02-19T22:01:12.936 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:12.937+0000 7eff7dffb700 1 -- 172.21.15.86:0/601086573 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7eff84010230 con 0x7eff880f62a0 2024-02-19T22:01:12.937 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T22:01:12.937 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:01:12.939 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:12.939+0000 7eff8e789700 1 -- 172.21.15.86:0/601086573 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7eff7402ff10 msgr2=0x7eff740323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:01:12.939 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:12.939+0000 7eff8e789700 1 --2- 172.21.15.86:0/601086573 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7eff7402ff10 0x7eff740323d0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7eff780096a0 tx=0x7eff78006b40).stop 2024-02-19T22:01:12.939 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:12.939+0000 7eff8e789700 1 -- 172.21.15.86:0/601086573 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7eff880f62a0 msgr2=0x7eff8806b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:01:12.939 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:12.939+0000 7eff8e789700 1 --2- 172.21.15.86:0/601086573 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7eff880f62a0 0x7eff8806b9a0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7eff840089f0 tx=0x7eff8400b040).stop 2024-02-19T22:01:12.939 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:12.939+0000 7eff8e789700 1 -- 172.21.15.86:0/601086573 shutdown_connections 2024-02-19T22:01:12.939 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:12.939+0000 7eff8e789700 1 --2- 172.21.15.86:0/601086573 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7eff7402ff10 0x7eff740323d0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:01:12.939 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:12.939+0000 7eff8e789700 1 --2- 172.21.15.86:0/601086573 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7eff880f62a0 0x7eff8806b9a0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:01:12.939 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:12.939+0000 7eff8e789700 1 -- 172.21.15.86:0/601086573 >> 172.21.15.86:0/601086573 conn(0x7eff880f1aa0 msgr2=0x7eff880f2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:01:12.939 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:12.939+0000 7eff8e789700 1 -- 172.21.15.86:0/601086573 shutdown_connections 2024-02-19T22:01:12.939 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:12.939+0000 7eff8e789700 1 -- 172.21.15.86:0/601086573 wait complete. 2024-02-19T22:01:12.939 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T22:01:13.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:01:13 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:01:12.938146+0000 mon.smithi053 (mon.0) 189 : audit [DBG] from='client.? 172.21.15.86:0/601086573' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:01:14.630 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:01:14.631 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:01:14.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:01:14 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:01:13.478878+0000 mgr.smithi053.fyhcdi (mgr.14162) 140 : cluster [DBG] pgmap v133: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:01:16.477 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:16.476+0000 7fc6e8090700 1 -- 172.21.15.86:0/2087340243 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc6e00f6510 msgr2=0x7fc6e00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:01:16.477 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:16.476+0000 7fc6e8090700 1 --2- 172.21.15.86:0/2087340243 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc6e00f6510 0x7fc6e00f6930 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 rx=0x7fc6d0009fb0 tx=0x7fc6d0009b30).stop 2024-02-19T22:01:16.477 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:16.476+0000 7fc6e8090700 1 -- 172.21.15.86:0/2087340243 shutdown_connections 2024-02-19T22:01:16.477 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:16.476+0000 7fc6e8090700 1 --2- 172.21.15.86:0/2087340243 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc6e00f6510 0x7fc6e00f6930 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:01:16.478 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:16.476+0000 7fc6e8090700 1 -- 172.21.15.86:0/2087340243 >> 172.21.15.86:0/2087340243 conn(0x7fc6e00f1c70 msgr2=0x7fc6e00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:01:16.478 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:16.476+0000 7fc6e8090700 1 -- 172.21.15.86:0/2087340243 shutdown_connections 2024-02-19T22:01:16.478 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:16.476+0000 7fc6e8090700 1 -- 172.21.15.86:0/2087340243 wait complete. 2024-02-19T22:01:16.478 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:16.476+0000 7fc6e8090700 1 Processor -- start 2024-02-19T22:01:16.478 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:16.476+0000 7fc6e8090700 1 -- start start 2024-02-19T22:01:16.478 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:16.477+0000 7fc6e8090700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc6e00f6510 0x7fc6e006b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:01:16.478 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:16.477+0000 7fc6e8090700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6e0069e20 con 0x7fc6e00f6510 2024-02-19T22:01:16.478 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:16.477+0000 7fc6e5e2c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc6e00f6510 0x7fc6e006b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:01:16.478 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:16.477+0000 7fc6e5e2c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc6e00f6510 0x7fc6e006b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:59746/0 (socket says 172.21.15.86:59746) 2024-02-19T22:01:16.478 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:16.477+0000 7fc6e5e2c700 1 -- 172.21.15.86:0/663662678 learned_addr learned my addr 172.21.15.86:0/663662678 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T22:01:16.478 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:16.477+0000 7fc6e5e2c700 1 -- 172.21.15.86:0/663662678 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6d0005040 con 0x7fc6e00f6510 2024-02-19T22:01:16.478 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:16.477+0000 7fc6e5e2c700 1 --2- 172.21.15.86:0/663662678 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc6e00f6510 0x7fc6e006b9a0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7fc6d0005010 tx=0x7fc6d000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:01:16.478 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:16.477+0000 7fc6d67fc700 1 -- 172.21.15.86:0/663662678 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc6d0009320 con 0x7fc6e00f6510 2024-02-19T22:01:16.478 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:16.478+0000 7fc6e8090700 1 -- 172.21.15.86:0/663662678 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6e0069fc0 con 0x7fc6e00f6510 2024-02-19T22:01:16.478 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:16.478+0000 7fc6e8090700 1 -- 172.21.15.86:0/663662678 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6e006a450 con 0x7fc6e00f6510 2024-02-19T22:01:16.479 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:16.478+0000 7fc6d67fc700 1 -- 172.21.15.86:0/663662678 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc6d0013380 con 0x7fc6e00f6510 2024-02-19T22:01:16.479 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:16.478+0000 7fc6d67fc700 1 -- 172.21.15.86:0/663662678 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc6d0014df0 con 0x7fc6e00f6510 2024-02-19T22:01:16.479 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:16.478+0000 7fc6d67fc700 1 -- 172.21.15.86:0/663662678 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc6d00095f0 con 0x7fc6e00f6510 2024-02-19T22:01:16.479 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:16.478+0000 7fc6e8090700 1 -- 172.21.15.86:0/663662678 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6c8004fa0 con 0x7fc6e00f6510 2024-02-19T22:01:16.479 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:16.479+0000 7fc6d67fc700 1 --2- 172.21.15.86:0/663662678 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fc6cc02ff60 0x7fc6cc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:01:16.480 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:16.479+0000 7fc6d67fc700 1 -- 172.21.15.86:0/663662678 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc6d0040eb0 con 0x7fc6e00f6510 2024-02-19T22:01:16.480 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:16.479+0000 7fc6e562b700 1 --2- 172.21.15.86:0/663662678 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fc6cc02ff60 0x7fc6cc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:01:16.480 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:16.479+0000 7fc6e562b700 1 --2- 172.21.15.86:0/663662678 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fc6cc02ff60 0x7fc6cc032420 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7fc6dc0096a0 tx=0x7fc6dc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:01:16.483 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:16.483+0000 7fc6d67fc700 1 -- 172.21.15.86:0/663662678 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc6d0014420 con 0x7fc6e00f6510 2024-02-19T22:01:16.686 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:16.687+0000 7fc6e8090700 1 -- 172.21.15.86:0/663662678 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc6c8005670 con 0x7fc6e00f6510 2024-02-19T22:01:16.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:16.687+0000 7fc6d67fc700 1 -- 172.21.15.86:0/663662678 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc6d001d9d0 con 0x7fc6e00f6510 2024-02-19T22:01:16.687 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T22:01:16.687 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:01:16.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:16.689+0000 7fc6e8090700 1 -- 172.21.15.86:0/663662678 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fc6cc02ff60 msgr2=0x7fc6cc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:01:16.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:16.689+0000 7fc6e8090700 1 --2- 172.21.15.86:0/663662678 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fc6cc02ff60 0x7fc6cc032420 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7fc6dc0096a0 tx=0x7fc6dc006b40).stop 2024-02-19T22:01:16.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:16.689+0000 7fc6e8090700 1 -- 172.21.15.86:0/663662678 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc6e00f6510 msgr2=0x7fc6e006b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:01:16.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:16.689+0000 7fc6e8090700 1 --2- 172.21.15.86:0/663662678 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc6e00f6510 0x7fc6e006b9a0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7fc6d0005010 tx=0x7fc6d000b040).stop 2024-02-19T22:01:16.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:16.689+0000 7fc6e8090700 1 -- 172.21.15.86:0/663662678 shutdown_connections 2024-02-19T22:01:16.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:16.689+0000 7fc6e8090700 1 --2- 172.21.15.86:0/663662678 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fc6cc02ff60 0x7fc6cc032420 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:01:16.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:16.689+0000 7fc6e8090700 1 --2- 172.21.15.86:0/663662678 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc6e00f6510 0x7fc6e006b9a0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:01:16.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:16.689+0000 7fc6e8090700 1 -- 172.21.15.86:0/663662678 >> 172.21.15.86:0/663662678 conn(0x7fc6e00f1c70 msgr2=0x7fc6e00f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:01:16.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:16.689+0000 7fc6e8090700 1 -- 172.21.15.86:0/663662678 shutdown_connections 2024-02-19T22:01:16.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:16.689+0000 7fc6e8090700 1 -- 172.21.15.86:0/663662678 wait complete. 2024-02-19T22:01:16.690 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T22:01:16.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:01:16 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:01:15.479269+0000 mgr.smithi053.fyhcdi (mgr.14162) 141 : cluster [DBG] pgmap v134: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:01:17.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:01:17 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:01:16.688444+0000 mon.smithi053 (mon.0) 190 : audit [DBG] from='client.? 172.21.15.86:0/663662678' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:01:18.390 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:01:18.391 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:01:18.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:01:18 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:01:17.479608+0000 mgr.smithi053.fyhcdi (mgr.14162) 142 : cluster [DBG] pgmap v135: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:01:20.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:20.168+0000 7f7a6678a700 1 -- 172.21.15.86:0/3002761405 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7a600f62a0 msgr2=0x7f7a600f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:01:20.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:20.168+0000 7f7a6678a700 1 --2- 172.21.15.86:0/3002761405 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7a600f62a0 0x7f7a600f66c0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 rx=0x7f7a5c009fb0 tx=0x7f7a5c009b30).stop 2024-02-19T22:01:20.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:20.168+0000 7f7a6678a700 1 -- 172.21.15.86:0/3002761405 shutdown_connections 2024-02-19T22:01:20.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:20.168+0000 7f7a6678a700 1 --2- 172.21.15.86:0/3002761405 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7a600f62a0 0x7f7a600f66c0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:01:20.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:20.168+0000 7f7a6678a700 1 -- 172.21.15.86:0/3002761405 >> 172.21.15.86:0/3002761405 conn(0x7f7a600f1aa0 msgr2=0x7f7a600f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:01:20.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:20.168+0000 7f7a6678a700 1 -- 172.21.15.86:0/3002761405 shutdown_connections 2024-02-19T22:01:20.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:20.168+0000 7f7a6678a700 1 -- 172.21.15.86:0/3002761405 wait complete. 2024-02-19T22:01:20.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:20.169+0000 7f7a6678a700 1 Processor -- start 2024-02-19T22:01:20.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:20.169+0000 7f7a6678a700 1 -- start start 2024-02-19T22:01:20.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:20.169+0000 7f7a6678a700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7a600f62a0 0x7f7a601065d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:01:20.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:20.169+0000 7f7a6678a700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a60106ac0 con 0x7f7a600f62a0 2024-02-19T22:01:20.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:20.169+0000 7f7a65788700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7a600f62a0 0x7f7a601065d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:01:20.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:20.169+0000 7f7a65788700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7a600f62a0 0x7f7a601065d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:41154/0 (socket says 172.21.15.86:41154) 2024-02-19T22:01:20.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:20.169+0000 7f7a65788700 1 -- 172.21.15.86:0/4031716667 learned_addr learned my addr 172.21.15.86:0/4031716667 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T22:01:20.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:20.169+0000 7f7a65788700 1 -- 172.21.15.86:0/4031716667 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a5c005040 con 0x7f7a600f62a0 2024-02-19T22:01:20.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:20.170+0000 7f7a65788700 1 --2- 172.21.15.86:0/4031716667 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7a600f62a0 0x7f7a601065d0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7f7a5c009830 tx=0x7f7a5c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:01:20.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:20.170+0000 7f7a55ffb700 1 -- 172.21.15.86:0/4031716667 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7a5c009320 con 0x7f7a600f62a0 2024-02-19T22:01:20.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:20.170+0000 7f7a6678a700 1 -- 172.21.15.86:0/4031716667 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a60106cc0 con 0x7f7a600f62a0 2024-02-19T22:01:20.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:20.170+0000 7f7a55ffb700 1 -- 172.21.15.86:0/4031716667 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7a5c013380 con 0x7f7a600f62a0 2024-02-19T22:01:20.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:20.170+0000 7f7a55ffb700 1 -- 172.21.15.86:0/4031716667 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7a5c014420 con 0x7f7a600f62a0 2024-02-19T22:01:20.172 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:20.170+0000 7f7a6678a700 1 -- 172.21.15.86:0/4031716667 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a60107150 con 0x7f7a600f62a0 2024-02-19T22:01:20.172 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:20.171+0000 7f7a55ffb700 1 -- 172.21.15.86:0/4031716667 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7a5c01e420 con 0x7f7a600f62a0 2024-02-19T22:01:20.173 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:20.171+0000 7f7a6678a700 1 -- 172.21.15.86:0/4031716667 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a60045ad0 con 0x7f7a600f62a0 2024-02-19T22:01:20.173 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:20.171+0000 7f7a55ffb700 1 --2- 172.21.15.86:0/4031716667 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f7a4c02ff60 0x7f7a4c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:01:20.173 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:20.171+0000 7f7a55ffb700 1 -- 172.21.15.86:0/4031716667 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7a5c040e20 con 0x7f7a600f62a0 2024-02-19T22:01:20.173 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:20.171+0000 7f7a64f87700 1 --2- 172.21.15.86:0/4031716667 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f7a4c02ff60 0x7f7a4c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:01:20.173 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:20.172+0000 7f7a64f87700 1 --2- 172.21.15.86:0/4031716667 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f7a4c02ff60 0x7f7a4c032420 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7f7a500096a0 tx=0x7f7a50006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:01:20.175 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:20.176+0000 7f7a55ffb700 1 -- 172.21.15.86:0/4031716667 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7a5c01c930 con 0x7f7a600f62a0 2024-02-19T22:01:20.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:20.383+0000 7f7a6678a700 1 -- 172.21.15.86:0/4031716667 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7a600fbe30 con 0x7f7a600f62a0 2024-02-19T22:01:20.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:20.383+0000 7f7a55ffb700 1 -- 172.21.15.86:0/4031716667 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7a5c02c2d0 con 0x7f7a600f62a0 2024-02-19T22:01:20.383 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T22:01:20.383 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:01:20.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:20.385+0000 7f7a6678a700 1 -- 172.21.15.86:0/4031716667 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f7a4c02ff60 msgr2=0x7f7a4c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:01:20.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:20.385+0000 7f7a6678a700 1 --2- 172.21.15.86:0/4031716667 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f7a4c02ff60 0x7f7a4c032420 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7f7a500096a0 tx=0x7f7a50006b40).stop 2024-02-19T22:01:20.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:20.385+0000 7f7a6678a700 1 -- 172.21.15.86:0/4031716667 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7a600f62a0 msgr2=0x7f7a601065d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:01:20.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:20.385+0000 7f7a6678a700 1 --2- 172.21.15.86:0/4031716667 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7a600f62a0 0x7f7a601065d0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7f7a5c009830 tx=0x7f7a5c00b040).stop 2024-02-19T22:01:20.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:20.385+0000 7f7a6678a700 1 -- 172.21.15.86:0/4031716667 shutdown_connections 2024-02-19T22:01:20.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:20.385+0000 7f7a6678a700 1 --2- 172.21.15.86:0/4031716667 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f7a4c02ff60 0x7f7a4c032420 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:01:20.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:20.385+0000 7f7a6678a700 1 --2- 172.21.15.86:0/4031716667 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7a600f62a0 0x7f7a601065d0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:01:20.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:20.385+0000 7f7a6678a700 1 -- 172.21.15.86:0/4031716667 >> 172.21.15.86:0/4031716667 conn(0x7f7a600f1aa0 msgr2=0x7f7a600f2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:01:20.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:20.385+0000 7f7a6678a700 1 -- 172.21.15.86:0/4031716667 shutdown_connections 2024-02-19T22:01:20.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:20.385+0000 7f7a6678a700 1 -- 172.21.15.86:0/4031716667 wait complete. 2024-02-19T22:01:20.386 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T22:01:20.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:01:20 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:01:19.479975+0000 mgr.smithi053.fyhcdi (mgr.14162) 143 : cluster [DBG] pgmap v136: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:01:20.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:01:20 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:01:20.384531+0000 mon.smithi053 (mon.0) 191 : audit [DBG] from='client.? 172.21.15.86:0/4031716667' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:01:22.105 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:01:22.105 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:01:22.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:01:22 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:01:21.480300+0000 mgr.smithi053.fyhcdi (mgr.14162) 144 : cluster [DBG] pgmap v137: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:01:24.218 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:24.216+0000 7fae8b26d700 1 -- 172.21.15.86:0/925514128 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fae840f5410 msgr2=0x7fae840f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:01:24.218 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:24.216+0000 7fae8b26d700 1 --2- 172.21.15.86:0/925514128 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fae840f5410 0x7fae840f5830 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 rx=0x7fae78009fb0 tx=0x7fae78009b30).stop 2024-02-19T22:01:24.218 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:24.217+0000 7fae8b26d700 1 -- 172.21.15.86:0/925514128 shutdown_connections 2024-02-19T22:01:24.218 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:24.217+0000 7fae8b26d700 1 --2- 172.21.15.86:0/925514128 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fae840f5410 0x7fae840f5830 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:01:24.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:24.217+0000 7fae8b26d700 1 -- 172.21.15.86:0/925514128 >> 172.21.15.86:0/925514128 conn(0x7fae840f0b70 msgr2=0x7fae840f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:01:24.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:24.217+0000 7fae8b26d700 1 -- 172.21.15.86:0/925514128 shutdown_connections 2024-02-19T22:01:24.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:24.217+0000 7fae8b26d700 1 -- 172.21.15.86:0/925514128 wait complete. 2024-02-19T22:01:24.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:24.217+0000 7fae8b26d700 1 Processor -- start 2024-02-19T22:01:24.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:24.217+0000 7fae8b26d700 1 -- start start 2024-02-19T22:01:24.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:24.218+0000 7fae8b26d700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fae840f5410 0x7fae841056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:01:24.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:24.218+0000 7fae8b26d700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae84105bd0 con 0x7fae840f5410 2024-02-19T22:01:24.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:24.218+0000 7fae89009700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fae840f5410 0x7fae841056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:01:24.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:24.218+0000 7fae89009700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fae840f5410 0x7fae841056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:41158/0 (socket says 172.21.15.86:41158) 2024-02-19T22:01:24.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:24.218+0000 7fae89009700 1 -- 172.21.15.86:0/703176356 learned_addr learned my addr 172.21.15.86:0/703176356 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T22:01:24.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:24.218+0000 7fae89009700 1 -- 172.21.15.86:0/703176356 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae78005040 con 0x7fae840f5410 2024-02-19T22:01:24.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:24.218+0000 7fae89009700 1 --2- 172.21.15.86:0/703176356 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fae840f5410 0x7fae841056e0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7fae78005010 tx=0x7fae7800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:01:24.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:24.218+0000 7fae75ffb700 1 -- 172.21.15.86:0/703176356 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fae78009320 con 0x7fae840f5410 2024-02-19T22:01:24.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:24.218+0000 7fae8b26d700 1 -- 172.21.15.86:0/703176356 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae84105dd0 con 0x7fae840f5410 2024-02-19T22:01:24.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:24.218+0000 7fae75ffb700 1 -- 172.21.15.86:0/703176356 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fae78013380 con 0x7fae840f5410 2024-02-19T22:01:24.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:24.218+0000 7fae75ffb700 1 -- 172.21.15.86:0/703176356 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fae78014e50 con 0x7fae840f5410 2024-02-19T22:01:24.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:24.219+0000 7fae8b26d700 1 -- 172.21.15.86:0/703176356 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae84106260 con 0x7fae840f5410 2024-02-19T22:01:24.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:24.219+0000 7fae75ffb700 1 -- 172.21.15.86:0/703176356 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fae780095f0 con 0x7fae840f5410 2024-02-19T22:01:24.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:24.219+0000 7fae8b26d700 1 -- 172.21.15.86:0/703176356 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae84045ad0 con 0x7fae840f5410 2024-02-19T22:01:24.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:24.220+0000 7fae75ffb700 1 --2- 172.21.15.86:0/703176356 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fae7002ff10 0x7fae700323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:01:24.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:24.220+0000 7fae75ffb700 1 -- 172.21.15.86:0/703176356 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fae78040ef0 con 0x7fae840f5410 2024-02-19T22:01:24.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:24.220+0000 7fae88808700 1 --2- 172.21.15.86:0/703176356 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fae7002ff10 0x7fae700323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:01:24.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:24.220+0000 7fae88808700 1 --2- 172.21.15.86:0/703176356 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fae7002ff10 0x7fae700323d0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7fae800096a0 tx=0x7fae80006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:01:24.223 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:24.224+0000 7fae75ffb700 1 -- 172.21.15.86:0/703176356 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fae78014420 con 0x7fae840f5410 2024-02-19T22:01:24.429 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:24.429+0000 7fae8b26d700 1 -- 172.21.15.86:0/703176356 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fae84107110 con 0x7fae840f5410 2024-02-19T22:01:24.429 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:24.430+0000 7fae75ffb700 1 -- 172.21.15.86:0/703176356 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fae78010230 con 0x7fae840f5410 2024-02-19T22:01:24.430 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T22:01:24.430 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:01:24.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:24.431+0000 7fae8b26d700 1 -- 172.21.15.86:0/703176356 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fae7002ff10 msgr2=0x7fae700323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:01:24.432 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:24.431+0000 7fae8b26d700 1 --2- 172.21.15.86:0/703176356 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fae7002ff10 0x7fae700323d0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7fae800096a0 tx=0x7fae80006b40).stop 2024-02-19T22:01:24.432 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:24.431+0000 7fae8b26d700 1 -- 172.21.15.86:0/703176356 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fae840f5410 msgr2=0x7fae841056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:01:24.432 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:24.431+0000 7fae8b26d700 1 --2- 172.21.15.86:0/703176356 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fae840f5410 0x7fae841056e0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7fae78005010 tx=0x7fae7800b040).stop 2024-02-19T22:01:24.432 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:24.432+0000 7fae8b26d700 1 -- 172.21.15.86:0/703176356 shutdown_connections 2024-02-19T22:01:24.432 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:24.432+0000 7fae8b26d700 1 --2- 172.21.15.86:0/703176356 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fae7002ff10 0x7fae700323d0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:01:24.432 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:24.432+0000 7fae8b26d700 1 --2- 172.21.15.86:0/703176356 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fae840f5410 0x7fae841056e0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:01:24.432 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:24.432+0000 7fae8b26d700 1 -- 172.21.15.86:0/703176356 >> 172.21.15.86:0/703176356 conn(0x7fae840f0b70 msgr2=0x7fae840f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:01:24.432 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:24.432+0000 7fae8b26d700 1 -- 172.21.15.86:0/703176356 shutdown_connections 2024-02-19T22:01:24.432 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:24.432+0000 7fae8b26d700 1 -- 172.21.15.86:0/703176356 wait complete. 2024-02-19T22:01:24.433 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T22:01:24.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:01:24 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:01:23.480600+0000 mgr.smithi053.fyhcdi (mgr.14162) 145 : cluster [DBG] pgmap v138: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:01:24.794 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:01:24 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:01:24.431186+0000 mon.smithi053 (mon.0) 192 : audit [DBG] from='client.? 172.21.15.86:0/703176356' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:01:26.142 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:01:26.143 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:01:26.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:01:26 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:01:25.480995+0000 mgr.smithi053.fyhcdi (mgr.14162) 146 : cluster [DBG] pgmap v139: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:01:28.017 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:28.016+0000 7fc935dbb700 1 -- 172.21.15.86:0/3757382170 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc9300f5410 msgr2=0x7fc9300f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:01:28.017 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:28.016+0000 7fc935dbb700 1 --2- 172.21.15.86:0/3757382170 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc9300f5410 0x7fc9300f5830 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 rx=0x7fc924009fb0 tx=0x7fc924009b30).stop 2024-02-19T22:01:28.017 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:28.016+0000 7fc935dbb700 1 -- 172.21.15.86:0/3757382170 shutdown_connections 2024-02-19T22:01:28.017 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:28.016+0000 7fc935dbb700 1 --2- 172.21.15.86:0/3757382170 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc9300f5410 0x7fc9300f5830 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:01:28.017 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:28.016+0000 7fc935dbb700 1 -- 172.21.15.86:0/3757382170 >> 172.21.15.86:0/3757382170 conn(0x7fc9300f0b70 msgr2=0x7fc9300f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:01:28.017 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:28.016+0000 7fc935dbb700 1 -- 172.21.15.86:0/3757382170 shutdown_connections 2024-02-19T22:01:28.017 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:28.016+0000 7fc935dbb700 1 -- 172.21.15.86:0/3757382170 wait complete. 2024-02-19T22:01:28.017 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:28.016+0000 7fc935dbb700 1 Processor -- start 2024-02-19T22:01:28.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:28.017+0000 7fc935dbb700 1 -- start start 2024-02-19T22:01:28.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:28.017+0000 7fc935dbb700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc9300f5410 0x7fc930109510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:01:28.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:28.017+0000 7fc935dbb700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc930060240 con 0x7fc9300f5410 2024-02-19T22:01:28.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:28.017+0000 7fc92f7fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc9300f5410 0x7fc930109510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:01:28.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:28.017+0000 7fc92f7fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc9300f5410 0x7fc930109510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:41188/0 (socket says 172.21.15.86:41188) 2024-02-19T22:01:28.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:28.017+0000 7fc92f7fe700 1 -- 172.21.15.86:0/2422105014 learned_addr learned my addr 172.21.15.86:0/2422105014 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T22:01:28.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:28.017+0000 7fc92f7fe700 1 -- 172.21.15.86:0/2422105014 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc924005040 con 0x7fc9300f5410 2024-02-19T22:01:28.020 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:28.018+0000 7fc92f7fe700 1 --2- 172.21.15.86:0/2422105014 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc9300f5410 0x7fc930109510 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7fc924005010 tx=0x7fc92400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:01:28.020 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:28.018+0000 7fc917fff700 1 -- 172.21.15.86:0/2422105014 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc924009320 con 0x7fc9300f5410 2024-02-19T22:01:28.020 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:28.018+0000 7fc935dbb700 1 -- 172.21.15.86:0/2422105014 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc930060440 con 0x7fc9300f5410 2024-02-19T22:01:28.020 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:28.018+0000 7fc917fff700 1 -- 172.21.15.86:0/2422105014 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc924013380 con 0x7fc9300f5410 2024-02-19T22:01:28.020 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:28.018+0000 7fc917fff700 1 -- 172.21.15.86:0/2422105014 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc924014e50 con 0x7fc9300f5410 2024-02-19T22:01:28.020 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:28.018+0000 7fc935dbb700 1 -- 172.21.15.86:0/2422105014 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc930109bc0 con 0x7fc9300f5410 2024-02-19T22:01:28.020 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:28.019+0000 7fc917fff700 1 -- 172.21.15.86:0/2422105014 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc9240263a0 con 0x7fc9300f5410 2024-02-19T22:01:28.020 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:28.019+0000 7fc917fff700 1 --2- 172.21.15.86:0/2422105014 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fc91802ff10 0x7fc9180323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:01:28.020 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:28.019+0000 7fc917fff700 1 -- 172.21.15.86:0/2422105014 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc924040e80 con 0x7fc9300f5410 2024-02-19T22:01:28.020 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:28.019+0000 7fc935dbb700 1 -- 172.21.15.86:0/2422105014 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc930045ad0 con 0x7fc9300f5410 2024-02-19T22:01:28.020 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:28.019+0000 7fc92effd700 1 --2- 172.21.15.86:0/2422105014 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fc91802ff10 0x7fc9180323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:01:28.021 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:28.020+0000 7fc92effd700 1 --2- 172.21.15.86:0/2422105014 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fc91802ff10 0x7fc9180323d0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7fc9200096a0 tx=0x7fc920006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:01:28.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:28.024+0000 7fc917fff700 1 -- 172.21.15.86:0/2422105014 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc924012920 con 0x7fc9300f5410 2024-02-19T22:01:28.226 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:28.227+0000 7fc935dbb700 1 -- 172.21.15.86:0/2422105014 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc93010a330 con 0x7fc9300f5410 2024-02-19T22:01:28.227 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:28.227+0000 7fc917fff700 1 -- 172.21.15.86:0/2422105014 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc92402b020 con 0x7fc9300f5410 2024-02-19T22:01:28.227 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T22:01:28.227 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:01:28.229 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:28.229+0000 7fc935dbb700 1 -- 172.21.15.86:0/2422105014 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fc91802ff10 msgr2=0x7fc9180323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:01:28.229 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:28.229+0000 7fc935dbb700 1 --2- 172.21.15.86:0/2422105014 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fc91802ff10 0x7fc9180323d0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7fc9200096a0 tx=0x7fc920006b40).stop 2024-02-19T22:01:28.229 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:28.229+0000 7fc935dbb700 1 -- 172.21.15.86:0/2422105014 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc9300f5410 msgr2=0x7fc930109510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:01:28.229 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:28.229+0000 7fc935dbb700 1 --2- 172.21.15.86:0/2422105014 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc9300f5410 0x7fc930109510 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7fc924005010 tx=0x7fc92400b040).stop 2024-02-19T22:01:28.229 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:28.229+0000 7fc935dbb700 1 -- 172.21.15.86:0/2422105014 shutdown_connections 2024-02-19T22:01:28.229 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:28.229+0000 7fc935dbb700 1 --2- 172.21.15.86:0/2422105014 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fc91802ff10 0x7fc9180323d0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:01:28.229 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:28.229+0000 7fc935dbb700 1 --2- 172.21.15.86:0/2422105014 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc9300f5410 0x7fc930109510 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:01:28.230 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:28.229+0000 7fc935dbb700 1 -- 172.21.15.86:0/2422105014 >> 172.21.15.86:0/2422105014 conn(0x7fc9300f0b70 msgr2=0x7fc9300f15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:01:28.230 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:28.229+0000 7fc935dbb700 1 -- 172.21.15.86:0/2422105014 shutdown_connections 2024-02-19T22:01:28.230 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:28.229+0000 7fc935dbb700 1 -- 172.21.15.86:0/2422105014 wait complete. 2024-02-19T22:01:28.230 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T22:01:28.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:01:28 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:01:27.481327+0000 mgr.smithi053.fyhcdi (mgr.14162) 147 : cluster [DBG] pgmap v140: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:01:28.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:01:28 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:01:28.228746+0000 mon.smithi053 (mon.0) 193 : audit [DBG] from='client.? 172.21.15.86:0/2422105014' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:01:29.855 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:01:29.856 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:01:30.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:01:30 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:01:29.481697+0000 mgr.smithi053.fyhcdi (mgr.14162) 148 : cluster [DBG] pgmap v141: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:01:31.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:31.677+0000 7fc8b359e700 1 -- 172.21.15.86:0/4003053950 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc8b40f51a0 msgr2=0x7fc8b40f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:01:31.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:31.678+0000 7fc8b359e700 1 --2- 172.21.15.86:0/4003053950 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc8b40f51a0 0x7fc8b40f55c0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 rx=0x7fc8a8009fb0 tx=0x7fc8a8009b30).stop 2024-02-19T22:01:31.679 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:31.678+0000 7fc8b359e700 1 -- 172.21.15.86:0/4003053950 shutdown_connections 2024-02-19T22:01:31.679 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:31.678+0000 7fc8b359e700 1 --2- 172.21.15.86:0/4003053950 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc8b40f51a0 0x7fc8b40f55c0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:01:31.679 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:31.678+0000 7fc8b359e700 1 -- 172.21.15.86:0/4003053950 >> 172.21.15.86:0/4003053950 conn(0x7fc8b40f09a0 msgr2=0x7fc8b40f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:01:31.679 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:31.678+0000 7fc8b359e700 1 -- 172.21.15.86:0/4003053950 shutdown_connections 2024-02-19T22:01:31.679 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:31.678+0000 7fc8b359e700 1 -- 172.21.15.86:0/4003053950 wait complete. 2024-02-19T22:01:31.679 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:31.678+0000 7fc8b359e700 1 Processor -- start 2024-02-19T22:01:31.679 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:31.678+0000 7fc8b359e700 1 -- start start 2024-02-19T22:01:31.679 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:31.678+0000 7fc8b359e700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc8b40f51a0 0x7fc8b41065e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:01:31.679 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:31.678+0000 7fc8b359e700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8b4106ad0 con 0x7fc8b40f51a0 2024-02-19T22:01:31.679 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:31.679+0000 7fc8b259c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc8b40f51a0 0x7fc8b41065e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:01:31.679 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:31.679+0000 7fc8b259c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc8b40f51a0 0x7fc8b41065e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:38686/0 (socket says 172.21.15.86:38686) 2024-02-19T22:01:31.679 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:31.679+0000 7fc8b259c700 1 -- 172.21.15.86:0/2978633836 learned_addr learned my addr 172.21.15.86:0/2978633836 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T22:01:31.679 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:31.679+0000 7fc8b259c700 1 -- 172.21.15.86:0/2978633836 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8a8005040 con 0x7fc8b40f51a0 2024-02-19T22:01:31.679 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:31.679+0000 7fc8b259c700 1 --2- 172.21.15.86:0/2978633836 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc8b40f51a0 0x7fc8b41065e0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7fc8a80089f0 tx=0x7fc8a800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:01:31.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:31.679+0000 7fc8a2ffd700 1 -- 172.21.15.86:0/2978633836 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc8a8009320 con 0x7fc8b40f51a0 2024-02-19T22:01:31.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:31.679+0000 7fc8b359e700 1 -- 172.21.15.86:0/2978633836 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8b4106cd0 con 0x7fc8b40f51a0 2024-02-19T22:01:31.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:31.679+0000 7fc8a2ffd700 1 -- 172.21.15.86:0/2978633836 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc8a8013380 con 0x7fc8b40f51a0 2024-02-19T22:01:31.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:31.679+0000 7fc8a2ffd700 1 -- 172.21.15.86:0/2978633836 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc8a8014e20 con 0x7fc8b40f51a0 2024-02-19T22:01:31.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:31.679+0000 7fc8b359e700 1 -- 172.21.15.86:0/2978633836 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8b4107160 con 0x7fc8b40f51a0 2024-02-19T22:01:31.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:31.680+0000 7fc8b359e700 1 -- 172.21.15.86:0/2978633836 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8b4045ad0 con 0x7fc8b40f51a0 2024-02-19T22:01:31.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:31.680+0000 7fc8a2ffd700 1 -- 172.21.15.86:0/2978633836 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc8a80095f0 con 0x7fc8b40f51a0 2024-02-19T22:01:31.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:31.680+0000 7fc8a2ffd700 1 --2- 172.21.15.86:0/2978633836 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fc89c02ff60 0x7fc89c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:01:31.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:31.680+0000 7fc8a2ffd700 1 -- 172.21.15.86:0/2978633836 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc8a8040e60 con 0x7fc8b40f51a0 2024-02-19T22:01:31.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:31.681+0000 7fc8b1d9b700 1 --2- 172.21.15.86:0/2978633836 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fc89c02ff60 0x7fc89c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:01:31.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:31.681+0000 7fc8b1d9b700 1 --2- 172.21.15.86:0/2978633836 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fc89c02ff60 0x7fc89c032420 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7fc8a40096a0 tx=0x7fc8a4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:01:31.684 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:31.686+0000 7fc8a2ffd700 1 -- 172.21.15.86:0/2978633836 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc8a8014420 con 0x7fc8b40f51a0 2024-02-19T22:01:31.889 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:31.889+0000 7fc8b359e700 1 -- 172.21.15.86:0/2978633836 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc8b4108010 con 0x7fc8b40f51a0 2024-02-19T22:01:31.890 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:31.890+0000 7fc8a2ffd700 1 -- 172.21.15.86:0/2978633836 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc8a8010230 con 0x7fc8b40f51a0 2024-02-19T22:01:31.890 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T22:01:31.890 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:01:31.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:31.892+0000 7fc8b359e700 1 -- 172.21.15.86:0/2978633836 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fc89c02ff60 msgr2=0x7fc89c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:01:31.892 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:31.892+0000 7fc8b359e700 1 --2- 172.21.15.86:0/2978633836 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fc89c02ff60 0x7fc89c032420 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7fc8a40096a0 tx=0x7fc8a4006b40).stop 2024-02-19T22:01:31.892 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:31.892+0000 7fc8b359e700 1 -- 172.21.15.86:0/2978633836 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc8b40f51a0 msgr2=0x7fc8b41065e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:01:31.892 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:31.892+0000 7fc8b359e700 1 --2- 172.21.15.86:0/2978633836 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc8b40f51a0 0x7fc8b41065e0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7fc8a80089f0 tx=0x7fc8a800b040).stop 2024-02-19T22:01:31.892 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:31.892+0000 7fc8b359e700 1 -- 172.21.15.86:0/2978633836 shutdown_connections 2024-02-19T22:01:31.892 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:31.892+0000 7fc8b359e700 1 --2- 172.21.15.86:0/2978633836 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fc89c02ff60 0x7fc89c032420 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:01:31.892 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:31.892+0000 7fc8b359e700 1 --2- 172.21.15.86:0/2978633836 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc8b40f51a0 0x7fc8b41065e0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:01:31.892 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:31.892+0000 7fc8b359e700 1 -- 172.21.15.86:0/2978633836 >> 172.21.15.86:0/2978633836 conn(0x7fc8b40f09a0 msgr2=0x7fc8b405fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:01:31.892 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:31.892+0000 7fc8b359e700 1 -- 172.21.15.86:0/2978633836 shutdown_connections 2024-02-19T22:01:31.892 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:31.892+0000 7fc8b359e700 1 -- 172.21.15.86:0/2978633836 wait complete. 2024-02-19T22:01:31.892 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T22:01:32.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:01:32 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:01:31.481971+0000 mgr.smithi053.fyhcdi (mgr.14162) 149 : cluster [DBG] pgmap v142: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:01:32.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:01:32 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:01:31.891335+0000 mon.smithi053 (mon.0) 194 : audit [DBG] from='client.? 172.21.15.86:0/2978633836' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:01:33.401 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:01:33.401 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:01:34.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:01:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:01:33.482298+0000 mgr.smithi053.fyhcdi (mgr.14162) 150 : cluster [DBG] pgmap v143: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:01:35.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:35.321+0000 7f678bf30700 1 -- 172.21.15.86:0/3391102284 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f67840f5410 msgr2=0x7f67840f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:01:35.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:35.321+0000 7f678bf30700 1 --2- 172.21.15.86:0/3391102284 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f67840f5410 0x7f67840f5830 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 rx=0x7f6778009fb0 tx=0x7f6778009b30).stop 2024-02-19T22:01:35.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:35.321+0000 7f678bf30700 1 -- 172.21.15.86:0/3391102284 shutdown_connections 2024-02-19T22:01:35.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:35.321+0000 7f678bf30700 1 --2- 172.21.15.86:0/3391102284 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f67840f5410 0x7f67840f5830 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:01:35.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:35.321+0000 7f678bf30700 1 -- 172.21.15.86:0/3391102284 >> 172.21.15.86:0/3391102284 conn(0x7f67840f0b70 msgr2=0x7f67840f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:01:35.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:35.322+0000 7f678bf30700 1 -- 172.21.15.86:0/3391102284 shutdown_connections 2024-02-19T22:01:35.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:35.322+0000 7f678bf30700 1 -- 172.21.15.86:0/3391102284 wait complete. 2024-02-19T22:01:35.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:35.322+0000 7f678bf30700 1 Processor -- start 2024-02-19T22:01:35.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:35.322+0000 7f678bf30700 1 -- start start 2024-02-19T22:01:35.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:35.322+0000 7f678bf30700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f67840f5410 0x7f67841056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:01:35.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:35.322+0000 7f678bf30700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6784105bd0 con 0x7f67840f5410 2024-02-19T22:01:35.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:35.323+0000 7f6789ccc700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f67840f5410 0x7f67841056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:01:35.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:35.323+0000 7f6789ccc700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f67840f5410 0x7f67841056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:38702/0 (socket says 172.21.15.86:38702) 2024-02-19T22:01:35.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:35.323+0000 7f6789ccc700 1 -- 172.21.15.86:0/2502967403 learned_addr learned my addr 172.21.15.86:0/2502967403 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T22:01:35.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:35.323+0000 7f6789ccc700 1 -- 172.21.15.86:0/2502967403 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6778005040 con 0x7f67840f5410 2024-02-19T22:01:35.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:35.323+0000 7f6789ccc700 1 --2- 172.21.15.86:0/2502967403 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f67840f5410 0x7f67841056e0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7f6778005010 tx=0x7f677800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:01:35.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:35.323+0000 7f67767fc700 1 -- 172.21.15.86:0/2502967403 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6778009320 con 0x7f67840f5410 2024-02-19T22:01:35.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:35.323+0000 7f67767fc700 1 -- 172.21.15.86:0/2502967403 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6778013380 con 0x7f67840f5410 2024-02-19T22:01:35.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:35.323+0000 7f678bf30700 1 -- 172.21.15.86:0/2502967403 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6784105dd0 con 0x7f67840f5410 2024-02-19T22:01:35.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:35.323+0000 7f67767fc700 1 -- 172.21.15.86:0/2502967403 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6778014e50 con 0x7f67840f5410 2024-02-19T22:01:35.333 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:35.323+0000 7f678bf30700 1 -- 172.21.15.86:0/2502967403 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6784106260 con 0x7f67840f5410 2024-02-19T22:01:35.333 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:35.324+0000 7f67767fc700 1 -- 172.21.15.86:0/2502967403 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f67780095f0 con 0x7f67840f5410 2024-02-19T22:01:35.333 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:35.324+0000 7f678bf30700 1 -- 172.21.15.86:0/2502967403 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6784045ad0 con 0x7f67840f5410 2024-02-19T22:01:35.333 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:35.324+0000 7f67767fc700 1 --2- 172.21.15.86:0/2502967403 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f677002ff10 0x7f67700323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:01:35.333 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:35.324+0000 7f67767fc700 1 -- 172.21.15.86:0/2502967403 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6778040e90 con 0x7f67840f5410 2024-02-19T22:01:35.333 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:35.324+0000 7f67894cb700 1 --2- 172.21.15.86:0/2502967403 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f677002ff10 0x7f67700323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:01:35.333 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:35.325+0000 7f67894cb700 1 --2- 172.21.15.86:0/2502967403 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f677002ff10 0x7f67700323d0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f67800096a0 tx=0x7f6780006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:01:35.333 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:35.329+0000 7f67767fc700 1 -- 172.21.15.86:0/2502967403 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6778014420 con 0x7f67840f5410 2024-02-19T22:01:35.532 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:35.533+0000 7f678bf30700 1 -- 172.21.15.86:0/2502967403 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6784107110 con 0x7f67840f5410 2024-02-19T22:01:35.533 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:35.534+0000 7f67767fc700 1 -- 172.21.15.86:0/2502967403 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6778010230 con 0x7f67840f5410 2024-02-19T22:01:35.533 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T22:01:35.533 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:01:35.535 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:35.536+0000 7f678bf30700 1 -- 172.21.15.86:0/2502967403 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f677002ff10 msgr2=0x7f67700323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:01:35.535 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:35.536+0000 7f678bf30700 1 --2- 172.21.15.86:0/2502967403 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f677002ff10 0x7f67700323d0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f67800096a0 tx=0x7f6780006b40).stop 2024-02-19T22:01:35.535 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:35.536+0000 7f678bf30700 1 -- 172.21.15.86:0/2502967403 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f67840f5410 msgr2=0x7f67841056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:01:35.535 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:35.536+0000 7f678bf30700 1 --2- 172.21.15.86:0/2502967403 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f67840f5410 0x7f67841056e0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7f6778005010 tx=0x7f677800b040).stop 2024-02-19T22:01:35.535 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:35.536+0000 7f678bf30700 1 -- 172.21.15.86:0/2502967403 shutdown_connections 2024-02-19T22:01:35.535 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:35.536+0000 7f678bf30700 1 --2- 172.21.15.86:0/2502967403 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f677002ff10 0x7f67700323d0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:01:35.535 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:35.536+0000 7f678bf30700 1 --2- 172.21.15.86:0/2502967403 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f67840f5410 0x7f67841056e0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:01:35.535 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:35.536+0000 7f678bf30700 1 -- 172.21.15.86:0/2502967403 >> 172.21.15.86:0/2502967403 conn(0x7f67840f0b70 msgr2=0x7f67840f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:01:35.535 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:35.536+0000 7f678bf30700 1 -- 172.21.15.86:0/2502967403 shutdown_connections 2024-02-19T22:01:35.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:35.536+0000 7f678bf30700 1 -- 172.21.15.86:0/2502967403 wait complete. 2024-02-19T22:01:35.536 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T22:01:35.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:01:35 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:01:35.534453+0000 mon.smithi053 (mon.0) 195 : audit [DBG] from='client.? 172.21.15.86:0/2502967403' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:01:36.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:01:36 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:01:35.482581+0000 mgr.smithi053.fyhcdi (mgr.14162) 151 : cluster [DBG] pgmap v144: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:01:36.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:01:36 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:01:35.601164+0000 mon.smithi053 (mon.0) 196 : audit [INF] from='mgr.14162 172.21.15.53:0/2860358250' entity='mgr.smithi053.fyhcdi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi053.fyhcdi/mirror_snapshot_schedule"}]: dispatch 2024-02-19T22:01:36.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:01:36 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:01:35.604451+0000 mon.smithi053 (mon.0) 197 : audit [INF] from='mgr.14162 172.21.15.53:0/2860358250' entity='mgr.smithi053.fyhcdi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi053.fyhcdi/trash_purge_schedule"}]: dispatch 2024-02-19T22:01:37.234 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:01:37.234 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:01:38.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:01:38 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:01:37.482841+0000 mgr.smithi053.fyhcdi (mgr.14162) 152 : cluster [DBG] pgmap v145: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:01:39.136 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:39.134+0000 7f01d61bc700 1 -- 172.21.15.86:0/1446814400 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f01d00f6510 msgr2=0x7f01d00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:01:39.137 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:39.134+0000 7f01d61bc700 1 --2- 172.21.15.86:0/1446814400 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f01d00f6510 0x7f01d00f6930 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 rx=0x7f01c4009fb0 tx=0x7f01c4009b30).stop 2024-02-19T22:01:39.137 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:39.134+0000 7f01d61bc700 1 -- 172.21.15.86:0/1446814400 shutdown_connections 2024-02-19T22:01:39.137 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:39.134+0000 7f01d61bc700 1 --2- 172.21.15.86:0/1446814400 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f01d00f6510 0x7f01d00f6930 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:01:39.137 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:39.134+0000 7f01d61bc700 1 -- 172.21.15.86:0/1446814400 >> 172.21.15.86:0/1446814400 conn(0x7f01d00f1c70 msgr2=0x7f01d00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:01:39.137 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:39.134+0000 7f01d61bc700 1 -- 172.21.15.86:0/1446814400 shutdown_connections 2024-02-19T22:01:39.137 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:39.134+0000 7f01d61bc700 1 -- 172.21.15.86:0/1446814400 wait complete. 2024-02-19T22:01:39.137 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:39.134+0000 7f01d61bc700 1 Processor -- start 2024-02-19T22:01:39.137 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:39.135+0000 7f01d61bc700 1 -- start start 2024-02-19T22:01:39.137 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:39.135+0000 7f01d61bc700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f01d00f6510 0x7f01d01067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:01:39.137 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:39.135+0000 7f01d61bc700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01d0106cd0 con 0x7f01d00f6510 2024-02-19T22:01:39.137 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:39.135+0000 7f01cf7fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f01d00f6510 0x7f01d01067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:01:39.137 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:39.135+0000 7f01cf7fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f01d00f6510 0x7f01d01067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:38712/0 (socket says 172.21.15.86:38712) 2024-02-19T22:01:39.137 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:39.135+0000 7f01cf7fe700 1 -- 172.21.15.86:0/2914571344 learned_addr learned my addr 172.21.15.86:0/2914571344 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T22:01:39.138 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:39.135+0000 7f01cf7fe700 1 -- 172.21.15.86:0/2914571344 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01c4005040 con 0x7f01d00f6510 2024-02-19T22:01:39.138 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:39.136+0000 7f01cf7fe700 1 --2- 172.21.15.86:0/2914571344 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f01d00f6510 0x7f01d01067e0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7f01c4005010 tx=0x7f01c400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:01:39.138 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:39.136+0000 7f01b7fff700 1 -- 172.21.15.86:0/2914571344 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f01c4009320 con 0x7f01d00f6510 2024-02-19T22:01:39.138 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:39.136+0000 7f01d61bc700 1 -- 172.21.15.86:0/2914571344 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01d0106ed0 con 0x7f01d00f6510 2024-02-19T22:01:39.138 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:39.136+0000 7f01b7fff700 1 -- 172.21.15.86:0/2914571344 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f01c4013380 con 0x7f01d00f6510 2024-02-19T22:01:39.138 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:39.136+0000 7f01b7fff700 1 -- 172.21.15.86:0/2914571344 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f01c4014e50 con 0x7f01d00f6510 2024-02-19T22:01:39.138 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:39.136+0000 7f01d61bc700 1 -- 172.21.15.86:0/2914571344 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01d0107360 con 0x7f01d00f6510 2024-02-19T22:01:39.138 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:39.136+0000 7f01b7fff700 1 -- 172.21.15.86:0/2914571344 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f01c4014420 con 0x7f01d00f6510 2024-02-19T22:01:39.138 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:39.137+0000 7f01d61bc700 1 -- 172.21.15.86:0/2914571344 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01d0045ad0 con 0x7f01d00f6510 2024-02-19T22:01:39.138 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:39.137+0000 7f01b7fff700 1 --2- 172.21.15.86:0/2914571344 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f01b802ff10 0x7f01b80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:01:39.138 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:39.137+0000 7f01b7fff700 1 -- 172.21.15.86:0/2914571344 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f01c4040f70 con 0x7f01d00f6510 2024-02-19T22:01:39.138 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:39.137+0000 7f01ceffd700 1 --2- 172.21.15.86:0/2914571344 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f01b802ff10 0x7f01b80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:01:39.138 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:39.137+0000 7f01ceffd700 1 --2- 172.21.15.86:0/2914571344 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f01b802ff10 0x7f01b80323d0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f01c00096a0 tx=0x7f01c0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:01:39.140 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:39.142+0000 7f01b7fff700 1 -- 172.21.15.86:0/2914571344 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f01c40095f0 con 0x7f01d00f6510 2024-02-19T22:01:39.344 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:39.346+0000 7f01d61bc700 1 -- 172.21.15.86:0/2914571344 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f01d01081b0 con 0x7f01d00f6510 2024-02-19T22:01:39.345 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:39.346+0000 7f01b7fff700 1 -- 172.21.15.86:0/2914571344 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f01c401db70 con 0x7f01d00f6510 2024-02-19T22:01:39.345 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T22:01:39.345 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:01:39.347 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:39.348+0000 7f01d61bc700 1 -- 172.21.15.86:0/2914571344 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f01b802ff10 msgr2=0x7f01b80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:01:39.347 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:39.348+0000 7f01d61bc700 1 --2- 172.21.15.86:0/2914571344 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f01b802ff10 0x7f01b80323d0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f01c00096a0 tx=0x7f01c0006b40).stop 2024-02-19T22:01:39.347 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:39.348+0000 7f01d61bc700 1 -- 172.21.15.86:0/2914571344 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f01d00f6510 msgr2=0x7f01d01067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:01:39.347 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:39.348+0000 7f01d61bc700 1 --2- 172.21.15.86:0/2914571344 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f01d00f6510 0x7f01d01067e0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7f01c4005010 tx=0x7f01c400b040).stop 2024-02-19T22:01:39.347 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:39.348+0000 7f01d61bc700 1 -- 172.21.15.86:0/2914571344 shutdown_connections 2024-02-19T22:01:39.347 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:39.348+0000 7f01d61bc700 1 --2- 172.21.15.86:0/2914571344 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f01b802ff10 0x7f01b80323d0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:01:39.347 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:39.348+0000 7f01d61bc700 1 --2- 172.21.15.86:0/2914571344 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f01d00f6510 0x7f01d01067e0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:01:39.347 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:39.348+0000 7f01d61bc700 1 -- 172.21.15.86:0/2914571344 >> 172.21.15.86:0/2914571344 conn(0x7f01d00f1c70 msgr2=0x7f01d00f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:01:39.347 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:39.348+0000 7f01d61bc700 1 -- 172.21.15.86:0/2914571344 shutdown_connections 2024-02-19T22:01:39.347 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:39.348+0000 7f01d61bc700 1 -- 172.21.15.86:0/2914571344 wait complete. 2024-02-19T22:01:39.348 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T22:01:39.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:01:39 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:01:39.346607+0000 mon.smithi053 (mon.0) 198 : audit [DBG] from='client.? 172.21.15.86:0/2914571344' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:01:40.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:01:40 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:01:39.483152+0000 mgr.smithi053.fyhcdi (mgr.14162) 153 : cluster [DBG] pgmap v146: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:01:41.044 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:01:41.044 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:01:42.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:01:42 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:01:41.483439+0000 mgr.smithi053.fyhcdi (mgr.14162) 154 : cluster [DBG] pgmap v147: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:01:42.987 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:42.986+0000 7fb67bd2f700 1 -- 172.21.15.86:0/2835204014 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb6740f53f0 msgr2=0x7fb6740f5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:01:42.987 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:42.986+0000 7fb67bd2f700 1 --2- 172.21.15.86:0/2835204014 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb6740f53f0 0x7fb6740f5810 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 rx=0x7fb664009fb0 tx=0x7fb664009b30).stop 2024-02-19T22:01:42.987 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:42.986+0000 7fb67bd2f700 1 -- 172.21.15.86:0/2835204014 shutdown_connections 2024-02-19T22:01:42.987 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:42.986+0000 7fb67bd2f700 1 --2- 172.21.15.86:0/2835204014 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb6740f53f0 0x7fb6740f5810 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:01:42.987 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:42.986+0000 7fb67bd2f700 1 -- 172.21.15.86:0/2835204014 >> 172.21.15.86:0/2835204014 conn(0x7fb6740f0b70 msgr2=0x7fb6740f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:01:42.987 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:42.986+0000 7fb67bd2f700 1 -- 172.21.15.86:0/2835204014 shutdown_connections 2024-02-19T22:01:42.987 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:42.986+0000 7fb67bd2f700 1 -- 172.21.15.86:0/2835204014 wait complete. 2024-02-19T22:01:42.987 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:42.987+0000 7fb67bd2f700 1 Processor -- start 2024-02-19T22:01:42.987 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:42.987+0000 7fb67bd2f700 1 -- start start 2024-02-19T22:01:42.987 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:42.987+0000 7fb67bd2f700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb6740f53f0 0x7fb6741056c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:01:42.987 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:42.987+0000 7fb67bd2f700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb674105bb0 con 0x7fb6740f53f0 2024-02-19T22:01:42.989 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:42.987+0000 7fb679acb700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb6740f53f0 0x7fb6741056c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:01:42.989 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:42.987+0000 7fb679acb700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb6740f53f0 0x7fb6741056c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:33522/0 (socket says 172.21.15.86:33522) 2024-02-19T22:01:42.989 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:42.987+0000 7fb679acb700 1 -- 172.21.15.86:0/945690464 learned_addr learned my addr 172.21.15.86:0/945690464 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T22:01:42.989 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:42.988+0000 7fb679acb700 1 -- 172.21.15.86:0/945690464 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb664005040 con 0x7fb6740f53f0 2024-02-19T22:01:42.989 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:42.988+0000 7fb679acb700 1 --2- 172.21.15.86:0/945690464 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb6740f53f0 0x7fb6741056c0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7fb6640089f0 tx=0x7fb66400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:01:42.989 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:42.988+0000 7fb66a7fc700 1 -- 172.21.15.86:0/945690464 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb664009320 con 0x7fb6740f53f0 2024-02-19T22:01:42.990 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:42.988+0000 7fb67bd2f700 1 -- 172.21.15.86:0/945690464 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb674105db0 con 0x7fb6740f53f0 2024-02-19T22:01:42.990 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:42.988+0000 7fb66a7fc700 1 -- 172.21.15.86:0/945690464 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb664013380 con 0x7fb6740f53f0 2024-02-19T22:01:42.990 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:42.988+0000 7fb66a7fc700 1 -- 172.21.15.86:0/945690464 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb66401ed80 con 0x7fb6740f53f0 2024-02-19T22:01:42.990 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:42.988+0000 7fb67bd2f700 1 -- 172.21.15.86:0/945690464 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb674106240 con 0x7fb6740f53f0 2024-02-19T22:01:42.990 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:42.989+0000 7fb66a7fc700 1 -- 172.21.15.86:0/945690464 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb66401e420 con 0x7fb6740f53f0 2024-02-19T22:01:42.990 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:42.989+0000 7fb67bd2f700 1 -- 172.21.15.86:0/945690464 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb674045ad0 con 0x7fb6740f53f0 2024-02-19T22:01:42.990 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:42.989+0000 7fb66a7fc700 1 --2- 172.21.15.86:0/945690464 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fb66002ff60 0x7fb660032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:01:42.990 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:42.989+0000 7fb66a7fc700 1 -- 172.21.15.86:0/945690464 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb6640410c0 con 0x7fb6740f53f0 2024-02-19T22:01:42.990 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:42.989+0000 7fb6792ca700 1 --2- 172.21.15.86:0/945690464 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fb66002ff60 0x7fb660032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:01:42.990 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:42.990+0000 7fb6792ca700 1 --2- 172.21.15.86:0/945690464 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fb66002ff60 0x7fb660032420 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7fb6700096a0 tx=0x7fb670006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:01:42.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:42.994+0000 7fb66a7fc700 1 -- 172.21.15.86:0/945690464 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb6640095f0 con 0x7fb6740f53f0 2024-02-19T22:01:43.197 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:43.197+0000 7fb67bd2f700 1 -- 172.21.15.86:0/945690464 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb6741070f0 con 0x7fb6740f53f0 2024-02-19T22:01:43.198 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:43.198+0000 7fb66a7fc700 1 -- 172.21.15.86:0/945690464 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb66401cb10 con 0x7fb6740f53f0 2024-02-19T22:01:43.198 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T22:01:43.198 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:01:43.199 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:43.199+0000 7fb67bd2f700 1 -- 172.21.15.86:0/945690464 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fb66002ff60 msgr2=0x7fb660032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:01:43.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:43.199+0000 7fb67bd2f700 1 --2- 172.21.15.86:0/945690464 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fb66002ff60 0x7fb660032420 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7fb6700096a0 tx=0x7fb670006b40).stop 2024-02-19T22:01:43.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:43.199+0000 7fb67bd2f700 1 -- 172.21.15.86:0/945690464 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb6740f53f0 msgr2=0x7fb6741056c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:01:43.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:43.199+0000 7fb67bd2f700 1 --2- 172.21.15.86:0/945690464 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb6740f53f0 0x7fb6741056c0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7fb6640089f0 tx=0x7fb66400b040).stop 2024-02-19T22:01:43.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:43.199+0000 7fb67bd2f700 1 -- 172.21.15.86:0/945690464 shutdown_connections 2024-02-19T22:01:43.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:43.199+0000 7fb67bd2f700 1 --2- 172.21.15.86:0/945690464 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fb66002ff60 0x7fb660032420 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:01:43.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:43.199+0000 7fb67bd2f700 1 --2- 172.21.15.86:0/945690464 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb6740f53f0 0x7fb6741056c0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:01:43.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:43.199+0000 7fb67bd2f700 1 -- 172.21.15.86:0/945690464 >> 172.21.15.86:0/945690464 conn(0x7fb6740f0b70 msgr2=0x7fb6740f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:01:43.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:43.200+0000 7fb67bd2f700 1 -- 172.21.15.86:0/945690464 shutdown_connections 2024-02-19T22:01:43.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:43.200+0000 7fb67bd2f700 1 -- 172.21.15.86:0/945690464 wait complete. 2024-02-19T22:01:43.201 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T22:01:43.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:01:43 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:01:43.199267+0000 mon.smithi053 (mon.0) 199 : audit [DBG] from='client.? 172.21.15.86:0/945690464' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:01:44.832 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:01:44.832 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:01:45.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:01:44 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:01:43.483730+0000 mgr.smithi053.fyhcdi (mgr.14162) 155 : cluster [DBG] pgmap v148: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:01:46.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:46.706+0000 7fdce0005700 1 -- 172.21.15.86:0/4133442061 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdcd80f5330 msgr2=0x7fdcd80f5750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:01:46.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:46.706+0000 7fdce0005700 1 --2- 172.21.15.86:0/4133442061 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdcd80f5330 0x7fdcd80f5750 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 rx=0x7fdcc8009fb0 tx=0x7fdcc8009b30).stop 2024-02-19T22:01:46.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:46.706+0000 7fdce0005700 1 -- 172.21.15.86:0/4133442061 shutdown_connections 2024-02-19T22:01:46.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:46.706+0000 7fdce0005700 1 --2- 172.21.15.86:0/4133442061 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdcd80f5330 0x7fdcd80f5750 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:01:46.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:46.706+0000 7fdce0005700 1 -- 172.21.15.86:0/4133442061 >> 172.21.15.86:0/4133442061 conn(0x7fdcd80f0a90 msgr2=0x7fdcd80f2eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:01:46.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:46.707+0000 7fdce0005700 1 -- 172.21.15.86:0/4133442061 shutdown_connections 2024-02-19T22:01:46.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:46.707+0000 7fdce0005700 1 -- 172.21.15.86:0/4133442061 wait complete. 2024-02-19T22:01:46.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:46.707+0000 7fdce0005700 1 Processor -- start 2024-02-19T22:01:46.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:46.707+0000 7fdce0005700 1 -- start start 2024-02-19T22:01:46.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:46.707+0000 7fdce0005700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdcd80f5330 0x7fdcd8106a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:01:46.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:46.707+0000 7fdce0005700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcd8106f70 con 0x7fdcd80f5330 2024-02-19T22:01:46.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:46.707+0000 7fdcddda1700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdcd80f5330 0x7fdcd8106a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:01:46.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:46.707+0000 7fdcddda1700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdcd80f5330 0x7fdcd8106a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:33530/0 (socket says 172.21.15.86:33530) 2024-02-19T22:01:46.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:46.707+0000 7fdcddda1700 1 -- 172.21.15.86:0/1595097725 learned_addr learned my addr 172.21.15.86:0/1595097725 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T22:01:46.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:46.708+0000 7fdcddda1700 1 -- 172.21.15.86:0/1595097725 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcc8005040 con 0x7fdcd80f5330 2024-02-19T22:01:46.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:46.708+0000 7fdcddda1700 1 --2- 172.21.15.86:0/1595097725 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdcd80f5330 0x7fdcd8106a80 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7fdcc8005010 tx=0x7fdcc800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:01:46.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:46.708+0000 7fdcce7fc700 1 -- 172.21.15.86:0/1595097725 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdcc8009320 con 0x7fdcd80f5330 2024-02-19T22:01:46.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:46.708+0000 7fdcce7fc700 1 -- 172.21.15.86:0/1595097725 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdcc8013380 con 0x7fdcd80f5330 2024-02-19T22:01:46.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:46.708+0000 7fdcce7fc700 1 -- 172.21.15.86:0/1595097725 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdcc8014e50 con 0x7fdcd80f5330 2024-02-19T22:01:46.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:46.708+0000 7fdce0005700 1 -- 172.21.15.86:0/1595097725 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcd8107170 con 0x7fdcd80f5330 2024-02-19T22:01:46.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:46.708+0000 7fdce0005700 1 -- 172.21.15.86:0/1595097725 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcd8107600 con 0x7fdcd80f5330 2024-02-19T22:01:46.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:46.709+0000 7fdcce7fc700 1 -- 172.21.15.86:0/1595097725 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdcc8014420 con 0x7fdcd80f5330 2024-02-19T22:01:46.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:46.709+0000 7fdce0005700 1 -- 172.21.15.86:0/1595097725 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdcd8045ad0 con 0x7fdcd80f5330 2024-02-19T22:01:46.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:46.709+0000 7fdcce7fc700 1 --2- 172.21.15.86:0/1595097725 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fdcc402ff10 0x7fdcc40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:01:46.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:46.709+0000 7fdcce7fc700 1 -- 172.21.15.86:0/1595097725 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdcc8040f90 con 0x7fdcd80f5330 2024-02-19T22:01:46.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:46.709+0000 7fdcdd5a0700 1 --2- 172.21.15.86:0/1595097725 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fdcc402ff10 0x7fdcc40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:01:46.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:46.709+0000 7fdcdd5a0700 1 --2- 172.21.15.86:0/1595097725 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fdcc402ff10 0x7fdcc40323d0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7fdcd40096a0 tx=0x7fdcd4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:01:46.713 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:46.713+0000 7fdcce7fc700 1 -- 172.21.15.86:0/1595097725 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdcc8014790 con 0x7fdcd80f5330 2024-02-19T22:01:46.917 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:46.917+0000 7fdce0005700 1 -- 172.21.15.86:0/1595097725 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdcd805ea50 con 0x7fdcd80f5330 2024-02-19T22:01:46.917 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:46.918+0000 7fdcce7fc700 1 -- 172.21.15.86:0/1595097725 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fdcc801d380 con 0x7fdcd80f5330 2024-02-19T22:01:46.918 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T22:01:46.918 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:01:46.919 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:46.919+0000 7fdce0005700 1 -- 172.21.15.86:0/1595097725 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fdcc402ff10 msgr2=0x7fdcc40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:01:46.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:46.919+0000 7fdce0005700 1 --2- 172.21.15.86:0/1595097725 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fdcc402ff10 0x7fdcc40323d0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7fdcd40096a0 tx=0x7fdcd4006b40).stop 2024-02-19T22:01:46.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:46.919+0000 7fdce0005700 1 -- 172.21.15.86:0/1595097725 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdcd80f5330 msgr2=0x7fdcd8106a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:01:46.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:46.919+0000 7fdce0005700 1 --2- 172.21.15.86:0/1595097725 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdcd80f5330 0x7fdcd8106a80 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7fdcc8005010 tx=0x7fdcc800b040).stop 2024-02-19T22:01:46.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:46.920+0000 7fdce0005700 1 -- 172.21.15.86:0/1595097725 shutdown_connections 2024-02-19T22:01:46.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:46.920+0000 7fdce0005700 1 --2- 172.21.15.86:0/1595097725 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fdcc402ff10 0x7fdcc40323d0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:01:46.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:46.920+0000 7fdce0005700 1 --2- 172.21.15.86:0/1595097725 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdcd80f5330 0x7fdcd8106a80 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:01:46.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:46.920+0000 7fdce0005700 1 -- 172.21.15.86:0/1595097725 >> 172.21.15.86:0/1595097725 conn(0x7fdcd80f0a90 msgr2=0x7fdcd805fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:01:46.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:46.920+0000 7fdce0005700 1 -- 172.21.15.86:0/1595097725 shutdown_connections 2024-02-19T22:01:46.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:46.920+0000 7fdce0005700 1 -- 172.21.15.86:0/1595097725 wait complete. 2024-02-19T22:01:46.921 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T22:01:46.967 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:01:46 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:01:45.484079+0000 mgr.smithi053.fyhcdi (mgr.14162) 156 : cluster [DBG] pgmap v149: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:01:46.967 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:01:46 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:01:45.803197+0000 mon.smithi053 (mon.0) 200 : audit [INF] from='mgr.14162 172.21.15.53:0/2860358250' entity='mgr.smithi053.fyhcdi' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi053", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-19T22:01:46.967 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:01:46 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:01:45.805640+0000 mon.smithi053 (mon.0) 201 : audit [INF] from='mgr.14162 172.21.15.53:0/2860358250' entity='mgr.smithi053.fyhcdi' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi053", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-02-19T22:01:46.967 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:01:46 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:01:45.806540+0000 mon.smithi053 (mon.0) 202 : audit [DBG] from='mgr.14162 172.21.15.53:0/2860358250' entity='mgr.smithi053.fyhcdi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T22:01:47.630 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:01:47 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cephadm 2024-02-19T22:01:45.801952+0000 mgr.smithi053.fyhcdi (mgr.14162) 157 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Deploy daemon alertmanager.smithi053 ... 2024-02-19T22:01:47.631 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:01:47 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi053 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/alertmanager 2024-02-19T22:01:47.631 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:01:47 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2024-02-19T22:01:47.631 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:01:47 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: stat: stderr time="2024-02-19T21:57:24Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-19T22:01:47.631 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:01:47 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: stat: stderr time="2024-02-19T21:58:01Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-19T22:01:47.631 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:01:47 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: stat: stderr time="2024-02-19T21:58:39Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-19T22:01:47.631 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:01:47 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-19T22:01:47.631 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:01:47 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi053 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/prometheus 2024-02-19T22:01:47.631 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:01:47 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2024-02-19T22:01:47.631 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:01:47 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: stat: stderr time="2024-02-19T21:59:53Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-19T22:01:47.631 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:01:47 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: stat: stderr time="2024-02-19T22:00:30Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-19T22:01:47.631 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:01:47 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: stat: stderr time="2024-02-19T22:01:08Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-19T22:01:47.631 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:01:47 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-19T22:01:47.631 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:01:47 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: Traceback (most recent call last): 2024-02-19T22:01:47.632 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:01:47 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: File "/var/lib/ceph/867b6b52-cf71-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-02-19T22:01:47.632 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:01:47 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: main() 2024-02-19T22:01:47.632 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:01:47 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: File "/var/lib/ceph/867b6b52-cf71-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-02-19T22:01:47.632 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:01:47 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: r = ctx.func(ctx) 2024-02-19T22:01:47.632 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:01:47 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: File "/var/lib/ceph/867b6b52-cf71-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-02-19T22:01:47.632 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:01:47 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: return func(ctx) 2024-02-19T22:01:47.632 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:01:47 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: File "/var/lib/ceph/867b6b52-cf71-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy 2024-02-19T22:01:47.632 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:01:47 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2024-02-19T22:01:47.632 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:01:47 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: File "/var/lib/ceph/867b6b52-cf71-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4191, in extract_uid_gid_monitoring 2024-02-19T22:01:47.632 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:01:47 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: uid, gid = extract_uid_gid(ctx, file_path=['/etc/alertmanager', '/etc/prometheus']) 2024-02-19T22:01:47.632 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:01:47 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: File "/var/lib/ceph/867b6b52-cf71-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid 2024-02-19T22:01:47.632 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:01:47 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: raise RuntimeError('uid/gid not found') 2024-02-19T22:01:47.632 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:01:47 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: RuntimeError: uid/gid not found 2024-02-19T22:01:47.632 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:01:47 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: Traceback (most recent call last): 2024-02-19T22:01:47.633 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:01:47 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1144, in _remote_connection 2024-02-19T22:01:47.633 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:01:47 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: yield (conn, connr) 2024-02-19T22:01:47.633 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:01:47 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1059, in _run_cephadm 2024-02-19T22:01:47.633 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:01:47 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: code, '\n'.join(err))) 2024-02-19T22:01:47.633 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:01:47 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Deploy daemon alertmanager.smithi053 ... 2024-02-19T22:01:47.633 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:01:47 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi053 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/alertmanager 2024-02-19T22:01:47.633 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:01:47 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2024-02-19T22:01:47.633 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:01:47 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: stat: stderr time="2024-02-19T21:57:24Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-19T22:01:47.633 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:01:47 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: stat: stderr time="2024-02-19T21:58:01Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-19T22:01:47.633 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:01:47 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: stat: stderr time="2024-02-19T21:58:39Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-19T22:01:47.633 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:01:47 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-19T22:01:47.633 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:01:47 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi053 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/prometheus 2024-02-19T22:01:47.633 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:01:47 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2024-02-19T22:01:47.633 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:01:47 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: stat: stderr time="2024-02-19T21:59:53Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-19T22:01:47.634 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:01:47 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: stat: stderr time="2024-02-19T22:00:30Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-19T22:01:47.634 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:01:47 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: stat: stderr time="2024-02-19T22:01:08Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-19T22:01:47.634 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:01:47 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-19T22:01:47.634 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:01:47 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: Traceback (most recent call last): 2024-02-19T22:01:47.634 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:01:47 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: File "/var/lib/ceph/867b6b52-cf71-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-02-19T22:01:47.634 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:01:47 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: main() 2024-02-19T22:01:47.634 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:01:47 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: File "/var/lib/ceph/867b6b52-cf71-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-02-19T22:01:47.634 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:01:47 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: r = ctx.func(ctx) 2024-02-19T22:01:47.634 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:01:47 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: File "/var/lib/ceph/867b6b52-cf71-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-02-19T22:01:47.634 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:01:47 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: return func(ctx) 2024-02-19T22:01:47.634 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:01:47 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: File "/var/lib/ceph/867b6b52-cf71-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy 2024-02-19T22:01:47.634 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:01:47 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2024-02-19T22:01:47.634 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:01:47 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: File "/var/lib/ceph/867b6b52-cf71-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4191, in extract_uid_gid_monitoring 2024-02-19T22:01:47.634 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:01:47 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: uid, gid = extract_uid_gid(ctx, file_path=['/etc/alertmanager', '/etc/prometheus']) 2024-02-19T22:01:47.635 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:01:47 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: File "/var/lib/ceph/867b6b52-cf71-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid 2024-02-19T22:01:47.635 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:01:47 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: raise RuntimeError('uid/gid not found') 2024-02-19T22:01:47.635 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:01:47 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: RuntimeError: uid/gid not found 2024-02-19T22:01:47.635 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:01:47 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cephadm 2024-02-19T22:01:45.807179+0000 mgr.smithi053.fyhcdi (mgr.14162) 158 : cephadm [INF] Deploying daemon crash.smithi053 on smithi053 2024-02-19T22:01:47.635 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:01:47 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:01:46.919065+0000 mon.smithi053 (mon.0) 203 : audit [DBG] from='client.? 172.21.15.86:0/1595097725' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:01:48.534 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:01:48.535 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:01:49.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:01:48 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:01:47.484343+0000 mgr.smithi053.fyhcdi (mgr.14162) 159 : cluster [DBG] pgmap v150: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:01:49.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:01:48 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:01:48.231342+0000 mon.smithi053 (mon.0) 204 : audit [INF] from='mgr.14162 172.21.15.53:0/2860358250' entity='mgr.smithi053.fyhcdi' 2024-02-19T22:01:49.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:01:48 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:01:48.344069+0000 mon.smithi053 (mon.0) 205 : audit [INF] from='mgr.14162 172.21.15.53:0/2860358250' entity='mgr.smithi053.fyhcdi' 2024-02-19T22:01:49.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:01:48 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:01:48.346413+0000 mon.smithi053 (mon.0) 206 : audit [INF] from='mgr.14162 172.21.15.53:0/2860358250' entity='mgr.smithi053.fyhcdi' 2024-02-19T22:01:49.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:01:48 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:01:48.346865+0000 mon.smithi053 (mon.0) 207 : audit [INF] from='mgr.14162 172.21.15.53:0/2860358250' entity='mgr.smithi053.fyhcdi' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-19T22:01:49.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:01:48 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:01:48.349567+0000 mon.smithi053 (mon.0) 208 : audit [INF] from='mgr.14162 172.21.15.53:0/2860358250' entity='mgr.smithi053.fyhcdi' 2024-02-19T22:01:50.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:01:49 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:01:48.347314+0000 mgr.smithi053.fyhcdi (mgr.14162) 160 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-19T22:01:50.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:01:49 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cephadm 2024-02-19T22:01:48.353164+0000 mgr.smithi053.fyhcdi (mgr.14162) 161 : cephadm [INF] Deploying daemon grafana.smithi053 on smithi053 2024-02-19T22:01:50.521 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:50.520+0000 7f6b1a518700 1 -- 172.21.15.86:0/384274717 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6b140f62a0 msgr2=0x7f6b140f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:01:50.521 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:50.520+0000 7f6b1a518700 1 --2- 172.21.15.86:0/384274717 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6b140f62a0 0x7f6b140f66c0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 rx=0x7f6b04009fb0 tx=0x7f6b04009b30).stop 2024-02-19T22:01:50.521 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:50.520+0000 7f6b1a518700 1 -- 172.21.15.86:0/384274717 shutdown_connections 2024-02-19T22:01:50.521 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:50.520+0000 7f6b1a518700 1 --2- 172.21.15.86:0/384274717 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6b140f62a0 0x7f6b140f66c0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:01:50.522 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:50.520+0000 7f6b1a518700 1 -- 172.21.15.86:0/384274717 >> 172.21.15.86:0/384274717 conn(0x7f6b140f1aa0 msgr2=0x7f6b140f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:01:50.522 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:50.520+0000 7f6b1a518700 1 -- 172.21.15.86:0/384274717 shutdown_connections 2024-02-19T22:01:50.522 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:50.520+0000 7f6b1a518700 1 -- 172.21.15.86:0/384274717 wait complete. 2024-02-19T22:01:50.522 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:50.520+0000 7f6b1a518700 1 Processor -- start 2024-02-19T22:01:50.522 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:50.520+0000 7f6b1a518700 1 -- start start 2024-02-19T22:01:50.522 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:50.520+0000 7f6b1a518700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6b140f62a0 0x7f6b141065d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:01:50.522 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:50.520+0000 7f6b1a518700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b14106ac0 con 0x7f6b140f62a0 2024-02-19T22:01:50.522 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:50.521+0000 7f6b19516700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6b140f62a0 0x7f6b141065d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:01:50.522 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:50.521+0000 7f6b19516700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6b140f62a0 0x7f6b141065d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:40626/0 (socket says 172.21.15.86:40626) 2024-02-19T22:01:50.522 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:50.521+0000 7f6b19516700 1 -- 172.21.15.86:0/3376195626 learned_addr learned my addr 172.21.15.86:0/3376195626 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T22:01:50.522 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:50.521+0000 7f6b19516700 1 -- 172.21.15.86:0/3376195626 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b04005040 con 0x7f6b140f62a0 2024-02-19T22:01:50.522 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:50.521+0000 7f6b19516700 1 --2- 172.21.15.86:0/3376195626 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6b140f62a0 0x7f6b141065d0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7f6b040089f0 tx=0x7f6b0400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:01:50.522 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:50.521+0000 7f6b09ffb700 1 -- 172.21.15.86:0/3376195626 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6b04009320 con 0x7f6b140f62a0 2024-02-19T22:01:50.522 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:50.521+0000 7f6b1a518700 1 -- 172.21.15.86:0/3376195626 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b14106cc0 con 0x7f6b140f62a0 2024-02-19T22:01:50.523 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:50.521+0000 7f6b09ffb700 1 -- 172.21.15.86:0/3376195626 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6b04013380 con 0x7f6b140f62a0 2024-02-19T22:01:50.523 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:50.521+0000 7f6b09ffb700 1 -- 172.21.15.86:0/3376195626 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6b04014e20 con 0x7f6b140f62a0 2024-02-19T22:01:50.523 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:50.521+0000 7f6b1a518700 1 -- 172.21.15.86:0/3376195626 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b14107150 con 0x7f6b140f62a0 2024-02-19T22:01:50.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:50.522+0000 7f6b09ffb700 1 -- 172.21.15.86:0/3376195626 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6b040095f0 con 0x7f6b140f62a0 2024-02-19T22:01:50.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:50.522+0000 7f6b1a518700 1 -- 172.21.15.86:0/3376195626 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b14045ad0 con 0x7f6b140f62a0 2024-02-19T22:01:50.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:50.522+0000 7f6b09ffb700 1 --2- 172.21.15.86:0/3376195626 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f6b0002ff10 0x7f6b000323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:01:50.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:50.522+0000 7f6b09ffb700 1 -- 172.21.15.86:0/3376195626 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6b04040f50 con 0x7f6b140f62a0 2024-02-19T22:01:50.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:50.523+0000 7f6b18d15700 1 --2- 172.21.15.86:0/3376195626 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f6b0002ff10 0x7f6b000323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:01:50.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:50.523+0000 7f6b18d15700 1 --2- 172.21.15.86:0/3376195626 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f6b0002ff10 0x7f6b000323d0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f6b100096a0 tx=0x7f6b10006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:01:50.526 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:50.527+0000 7f6b09ffb700 1 -- 172.21.15.86:0/3376195626 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6b04014420 con 0x7f6b140f62a0 2024-02-19T22:01:50.734 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:50.734+0000 7f6b1a518700 1 -- 172.21.15.86:0/3376195626 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6b14108000 con 0x7f6b140f62a0 2024-02-19T22:01:50.734 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:50.734+0000 7f6b09ffb700 1 -- 172.21.15.86:0/3376195626 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6b0401d980 con 0x7f6b140f62a0 2024-02-19T22:01:50.734 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T22:01:50.734 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:01:50.736 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:50.736+0000 7f6b1a518700 1 -- 172.21.15.86:0/3376195626 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f6b0002ff10 msgr2=0x7f6b000323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:01:50.736 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:50.736+0000 7f6b1a518700 1 --2- 172.21.15.86:0/3376195626 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f6b0002ff10 0x7f6b000323d0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f6b100096a0 tx=0x7f6b10006b40).stop 2024-02-19T22:01:50.736 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:50.736+0000 7f6b1a518700 1 -- 172.21.15.86:0/3376195626 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6b140f62a0 msgr2=0x7f6b141065d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:01:50.736 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:50.736+0000 7f6b1a518700 1 --2- 172.21.15.86:0/3376195626 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6b140f62a0 0x7f6b141065d0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7f6b040089f0 tx=0x7f6b0400b040).stop 2024-02-19T22:01:50.736 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:50.736+0000 7f6b1a518700 1 -- 172.21.15.86:0/3376195626 shutdown_connections 2024-02-19T22:01:50.736 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:50.736+0000 7f6b1a518700 1 --2- 172.21.15.86:0/3376195626 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f6b0002ff10 0x7f6b000323d0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:01:50.736 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:50.736+0000 7f6b1a518700 1 --2- 172.21.15.86:0/3376195626 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6b140f62a0 0x7f6b141065d0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:01:50.736 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:50.736+0000 7f6b1a518700 1 -- 172.21.15.86:0/3376195626 >> 172.21.15.86:0/3376195626 conn(0x7f6b140f1aa0 msgr2=0x7f6b140f2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:01:50.737 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:50.736+0000 7f6b1a518700 1 -- 172.21.15.86:0/3376195626 shutdown_connections 2024-02-19T22:01:50.737 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:50.736+0000 7f6b1a518700 1 -- 172.21.15.86:0/3376195626 wait complete. 2024-02-19T22:01:50.737 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T22:01:51.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:01:50 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:01:49.484752+0000 mgr.smithi053.fyhcdi (mgr.14162) 162 : cluster [DBG] pgmap v151: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:01:52.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:01:51 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:01:50.735617+0000 mon.smithi053 (mon.0) 209 : audit [DBG] from='client.? 172.21.15.86:0/3376195626' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:01:52.355 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:01:52.356 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:01:53.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:01:52 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:01:51.485176+0000 mgr.smithi053.fyhcdi (mgr.14162) 163 : cluster [DBG] pgmap v152: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:01:54.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:54.143+0000 7fe9e1066700 1 -- 172.21.15.86:0/2219537899 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe9dc0f6510 msgr2=0x7fe9dc0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:01:54.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:54.143+0000 7fe9e1066700 1 --2- 172.21.15.86:0/2219537899 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe9dc0f6510 0x7fe9dc0f6930 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 rx=0x7fe9d0009fb0 tx=0x7fe9d0009b30).stop 2024-02-19T22:01:54.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:54.143+0000 7fe9e1066700 1 -- 172.21.15.86:0/2219537899 shutdown_connections 2024-02-19T22:01:54.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:54.143+0000 7fe9e1066700 1 --2- 172.21.15.86:0/2219537899 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe9dc0f6510 0x7fe9dc0f6930 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:01:54.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:54.143+0000 7fe9e1066700 1 -- 172.21.15.86:0/2219537899 >> 172.21.15.86:0/2219537899 conn(0x7fe9dc0f1c70 msgr2=0x7fe9dc0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:01:54.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:54.143+0000 7fe9e1066700 1 -- 172.21.15.86:0/2219537899 shutdown_connections 2024-02-19T22:01:54.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:54.143+0000 7fe9e1066700 1 -- 172.21.15.86:0/2219537899 wait complete. 2024-02-19T22:01:54.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:54.143+0000 7fe9e1066700 1 Processor -- start 2024-02-19T22:01:54.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:54.144+0000 7fe9e1066700 1 -- start start 2024-02-19T22:01:54.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:54.144+0000 7fe9e1066700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe9dc0f6510 0x7fe9dc06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:01:54.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:54.144+0000 7fe9e1066700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9dc069e20 con 0x7fe9dc0f6510 2024-02-19T22:01:54.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:54.144+0000 7fe9dad9d700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe9dc0f6510 0x7fe9dc06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:01:54.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:54.144+0000 7fe9dad9d700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe9dc0f6510 0x7fe9dc06b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:40634/0 (socket says 172.21.15.86:40634) 2024-02-19T22:01:54.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:54.144+0000 7fe9dad9d700 1 -- 172.21.15.86:0/66928809 learned_addr learned my addr 172.21.15.86:0/66928809 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T22:01:54.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:54.144+0000 7fe9dad9d700 1 -- 172.21.15.86:0/66928809 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9d0005040 con 0x7fe9dc0f6510 2024-02-19T22:01:54.147 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:54.144+0000 7fe9dad9d700 1 --2- 172.21.15.86:0/66928809 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe9dc0f6510 0x7fe9dc06b9a0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7fe9d0005010 tx=0x7fe9d000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:01:54.147 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:54.145+0000 7fe9c37fe700 1 -- 172.21.15.86:0/66928809 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe9d0009320 con 0x7fe9dc0f6510 2024-02-19T22:01:54.147 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:54.145+0000 7fe9c37fe700 1 -- 172.21.15.86:0/66928809 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe9d0013380 con 0x7fe9dc0f6510 2024-02-19T22:01:54.147 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:54.145+0000 7fe9e1066700 1 -- 172.21.15.86:0/66928809 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9dc069fc0 con 0x7fe9dc0f6510 2024-02-19T22:01:54.147 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:54.145+0000 7fe9c37fe700 1 -- 172.21.15.86:0/66928809 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe9d0014e50 con 0x7fe9dc0f6510 2024-02-19T22:01:54.147 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:54.145+0000 7fe9e1066700 1 -- 172.21.15.86:0/66928809 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9dc06a450 con 0x7fe9dc0f6510 2024-02-19T22:01:54.147 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:54.145+0000 7fe9e1066700 1 -- 172.21.15.86:0/66928809 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9dc059100 con 0x7fe9dc0f6510 2024-02-19T22:01:54.147 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:54.145+0000 7fe9c37fe700 1 -- 172.21.15.86:0/66928809 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe9d00095f0 con 0x7fe9dc0f6510 2024-02-19T22:01:54.147 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:54.146+0000 7fe9c37fe700 1 --2- 172.21.15.86:0/66928809 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fe9c402ff10 0x7fe9c40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:01:54.147 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:54.146+0000 7fe9c37fe700 1 -- 172.21.15.86:0/66928809 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe9d0040e70 con 0x7fe9dc0f6510 2024-02-19T22:01:54.147 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:54.146+0000 7fe9da59c700 1 --2- 172.21.15.86:0/66928809 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fe9c402ff10 0x7fe9c40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:01:54.147 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:54.146+0000 7fe9da59c700 1 --2- 172.21.15.86:0/66928809 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fe9c402ff10 0x7fe9c40323d0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7fe9cc0096a0 tx=0x7fe9cc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:01:54.150 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:54.150+0000 7fe9c37fe700 1 -- 172.21.15.86:0/66928809 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe9d0014420 con 0x7fe9dc0f6510 2024-02-19T22:01:54.355 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:54.355+0000 7fe9e1066700 1 -- 172.21.15.86:0/66928809 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe9dc0fc0a0 con 0x7fe9dc0f6510 2024-02-19T22:01:54.355 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:54.355+0000 7fe9c37fe700 1 -- 172.21.15.86:0/66928809 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe9d0010230 con 0x7fe9dc0f6510 2024-02-19T22:01:54.356 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T22:01:54.356 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:01:54.357 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:54.357+0000 7fe9e1066700 1 -- 172.21.15.86:0/66928809 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fe9c402ff10 msgr2=0x7fe9c40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:01:54.357 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:54.357+0000 7fe9e1066700 1 --2- 172.21.15.86:0/66928809 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fe9c402ff10 0x7fe9c40323d0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7fe9cc0096a0 tx=0x7fe9cc006b40).stop 2024-02-19T22:01:54.357 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:54.357+0000 7fe9e1066700 1 -- 172.21.15.86:0/66928809 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe9dc0f6510 msgr2=0x7fe9dc06b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:01:54.357 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:54.357+0000 7fe9e1066700 1 --2- 172.21.15.86:0/66928809 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe9dc0f6510 0x7fe9dc06b9a0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7fe9d0005010 tx=0x7fe9d000b040).stop 2024-02-19T22:01:54.357 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:54.357+0000 7fe9e1066700 1 -- 172.21.15.86:0/66928809 shutdown_connections 2024-02-19T22:01:54.357 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:54.357+0000 7fe9e1066700 1 --2- 172.21.15.86:0/66928809 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fe9c402ff10 0x7fe9c40323d0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:01:54.357 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:54.357+0000 7fe9e1066700 1 --2- 172.21.15.86:0/66928809 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe9dc0f6510 0x7fe9dc06b9a0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:01:54.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:54.357+0000 7fe9e1066700 1 -- 172.21.15.86:0/66928809 >> 172.21.15.86:0/66928809 conn(0x7fe9dc0f1c70 msgr2=0x7fe9dc0f26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:01:54.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:54.357+0000 7fe9e1066700 1 -- 172.21.15.86:0/66928809 shutdown_connections 2024-02-19T22:01:54.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:54.357+0000 7fe9e1066700 1 -- 172.21.15.86:0/66928809 wait complete. 2024-02-19T22:01:54.358 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T22:01:55.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:01:54 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:01:53.485518+0000 mgr.smithi053.fyhcdi (mgr.14162) 164 : cluster [DBG] pgmap v153: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:01:55.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:01:54 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:01:54.356817+0000 mon.smithi053 (mon.0) 210 : audit [DBG] from='client.? 172.21.15.86:0/66928809' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:01:55.983 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:01:55.983 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:01:57.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:01:56 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:01:55.485861+0000 mgr.smithi053.fyhcdi (mgr.14162) 165 : cluster [DBG] pgmap v154: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:01:57.896 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:57.895+0000 7f7532c1e700 1 -- 172.21.15.86:0/610821626 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f752c060200 msgr2=0x7f752c0fb570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:01:57.896 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:57.895+0000 7f7532c1e700 1 --2- 172.21.15.86:0/610821626 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f752c060200 0x7f752c0fb570 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 rx=0x7f7520009fb0 tx=0x7f7520009b30).stop 2024-02-19T22:01:57.896 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:57.895+0000 7f7532c1e700 1 -- 172.21.15.86:0/610821626 shutdown_connections 2024-02-19T22:01:57.896 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:57.895+0000 7f7532c1e700 1 --2- 172.21.15.86:0/610821626 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f752c060200 0x7f752c0fb570 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:01:57.896 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:57.895+0000 7f7532c1e700 1 -- 172.21.15.86:0/610821626 >> 172.21.15.86:0/610821626 conn(0x7f752c0f0b70 msgr2=0x7f752c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:01:57.896 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:57.895+0000 7f7532c1e700 1 -- 172.21.15.86:0/610821626 shutdown_connections 2024-02-19T22:01:57.897 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:57.895+0000 7f7532c1e700 1 -- 172.21.15.86:0/610821626 wait complete. 2024-02-19T22:01:57.897 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:57.895+0000 7f7532c1e700 1 Processor -- start 2024-02-19T22:01:57.897 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:57.895+0000 7f7532c1e700 1 -- start start 2024-02-19T22:01:57.897 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:57.895+0000 7f7532c1e700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f752c060200 0x7f752c1089a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:01:57.897 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:57.895+0000 7f7532c1e700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f752c108e90 con 0x7f752c060200 2024-02-19T22:01:57.897 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:57.896+0000 7f75309ba700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f752c060200 0x7f752c1089a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:01:57.897 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:57.896+0000 7f75309ba700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f752c060200 0x7f752c1089a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:40656/0 (socket says 172.21.15.86:40656) 2024-02-19T22:01:57.897 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:57.896+0000 7f75309ba700 1 -- 172.21.15.86:0/3783476989 learned_addr learned my addr 172.21.15.86:0/3783476989 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T22:01:57.897 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:57.896+0000 7f75309ba700 1 -- 172.21.15.86:0/3783476989 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7520005040 con 0x7f752c060200 2024-02-19T22:01:57.897 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:57.896+0000 7f75309ba700 1 --2- 172.21.15.86:0/3783476989 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f752c060200 0x7f752c1089a0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7f75200089f0 tx=0x7f752000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:01:57.897 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:57.896+0000 7f75297fa700 1 -- 172.21.15.86:0/3783476989 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7520009320 con 0x7f752c060200 2024-02-19T22:01:57.897 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:57.896+0000 7f75297fa700 1 -- 172.21.15.86:0/3783476989 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7520013380 con 0x7f752c060200 2024-02-19T22:01:57.897 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:57.896+0000 7f7532c1e700 1 -- 172.21.15.86:0/3783476989 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f752c109090 con 0x7f752c060200 2024-02-19T22:01:57.897 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:57.896+0000 7f75297fa700 1 -- 172.21.15.86:0/3783476989 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7520014e20 con 0x7f752c060200 2024-02-19T22:01:57.897 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:57.896+0000 7f7532c1e700 1 -- 172.21.15.86:0/3783476989 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f752c109520 con 0x7f752c060200 2024-02-19T22:01:57.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:57.897+0000 7f75297fa700 1 -- 172.21.15.86:0/3783476989 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f75200263a0 con 0x7f752c060200 2024-02-19T22:01:57.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:57.897+0000 7f75297fa700 1 --2- 172.21.15.86:0/3783476989 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f751c02ff60 0x7f751c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:01:57.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:57.897+0000 7f7532c1e700 1 -- 172.21.15.86:0/3783476989 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f752c045ad0 con 0x7f752c060200 2024-02-19T22:01:57.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:57.897+0000 7f75297fa700 1 -- 172.21.15.86:0/3783476989 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7520041230 con 0x7f752c060200 2024-02-19T22:01:57.899 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:57.897+0000 7f752bfff700 1 --2- 172.21.15.86:0/3783476989 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f751c02ff60 0x7f751c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:01:57.899 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:57.897+0000 7f752bfff700 1 --2- 172.21.15.86:0/3783476989 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f751c02ff60 0x7f751c032420 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f75180096a0 tx=0x7f7518006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:01:57.902 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:57.902+0000 7f75297fa700 1 -- 172.21.15.86:0/3783476989 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7520012510 con 0x7f752c060200 2024-02-19T22:01:58.106 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:58.106+0000 7f7532c1e700 1 -- 172.21.15.86:0/3783476989 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f752c10a3d0 con 0x7f752c060200 2024-02-19T22:01:58.106 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:58.107+0000 7f75297fa700 1 -- 172.21.15.86:0/3783476989 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7520012330 con 0x7f752c060200 2024-02-19T22:01:58.107 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T22:01:58.107 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:01:58.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:58.108+0000 7f7532c1e700 1 -- 172.21.15.86:0/3783476989 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f751c02ff60 msgr2=0x7f751c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:01:58.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:58.108+0000 7f7532c1e700 1 --2- 172.21.15.86:0/3783476989 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f751c02ff60 0x7f751c032420 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f75180096a0 tx=0x7f7518006b40).stop 2024-02-19T22:01:58.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:58.108+0000 7f7532c1e700 1 -- 172.21.15.86:0/3783476989 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f752c060200 msgr2=0x7f752c1089a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:01:58.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:58.108+0000 7f7532c1e700 1 --2- 172.21.15.86:0/3783476989 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f752c060200 0x7f752c1089a0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7f75200089f0 tx=0x7f752000b040).stop 2024-02-19T22:01:58.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:58.108+0000 7f7532c1e700 1 -- 172.21.15.86:0/3783476989 shutdown_connections 2024-02-19T22:01:58.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:58.108+0000 7f7532c1e700 1 --2- 172.21.15.86:0/3783476989 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f751c02ff60 0x7f751c032420 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:01:58.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:58.108+0000 7f7532c1e700 1 --2- 172.21.15.86:0/3783476989 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f752c060200 0x7f752c1089a0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:01:58.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:58.108+0000 7f7532c1e700 1 -- 172.21.15.86:0/3783476989 >> 172.21.15.86:0/3783476989 conn(0x7f752c0f0b70 msgr2=0x7f752c0f15c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:01:58.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:58.108+0000 7f7532c1e700 1 -- 172.21.15.86:0/3783476989 shutdown_connections 2024-02-19T22:01:58.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:01:58.108+0000 7f7532c1e700 1 -- 172.21.15.86:0/3783476989 wait complete. 2024-02-19T22:01:58.109 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T22:01:59.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:01:58 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:01:57.486237+0000 mgr.smithi053.fyhcdi (mgr.14162) 166 : cluster [DBG] pgmap v155: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:01:59.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:01:58 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:01:58.108081+0000 mon.smithi053 (mon.0) 211 : audit [DBG] from='client.? 172.21.15.86:0/3783476989' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:01:59.801 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:01:59.802 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:02:00.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:02:00 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:01:59.486491+0000 mgr.smithi053.fyhcdi (mgr.14162) 167 : cluster [DBG] pgmap v156: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:02:01.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:01.553+0000 7f7df629e700 1 -- 172.21.15.86:0/4273781213 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7df00f5410 msgr2=0x7f7df00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:02:01.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:01.553+0000 7f7df629e700 1 --2- 172.21.15.86:0/4273781213 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7df00f5410 0x7f7df00f5830 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 rx=0x7f7de4009fb0 tx=0x7f7de4009b30).stop 2024-02-19T22:02:01.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:01.553+0000 7f7df629e700 1 -- 172.21.15.86:0/4273781213 shutdown_connections 2024-02-19T22:02:01.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:01.553+0000 7f7df629e700 1 --2- 172.21.15.86:0/4273781213 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7df00f5410 0x7f7df00f5830 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:02:01.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:01.553+0000 7f7df629e700 1 -- 172.21.15.86:0/4273781213 >> 172.21.15.86:0/4273781213 conn(0x7f7df00f0b70 msgr2=0x7f7df00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:02:01.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:01.553+0000 7f7df629e700 1 -- 172.21.15.86:0/4273781213 shutdown_connections 2024-02-19T22:02:01.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:01.553+0000 7f7df629e700 1 -- 172.21.15.86:0/4273781213 wait complete. 2024-02-19T22:02:01.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:01.554+0000 7f7df629e700 1 Processor -- start 2024-02-19T22:02:01.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:01.554+0000 7f7df629e700 1 -- start start 2024-02-19T22:02:01.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:01.554+0000 7f7df629e700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7df00f5410 0x7f7df0109500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:02:01.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:01.554+0000 7f7df629e700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7df0060220 con 0x7f7df00f5410 2024-02-19T22:02:01.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:01.554+0000 7f7deffff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7df00f5410 0x7f7df0109500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:02:01.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:01.554+0000 7f7deffff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7df00f5410 0x7f7df0109500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:57820/0 (socket says 172.21.15.86:57820) 2024-02-19T22:02:01.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:01.554+0000 7f7deffff700 1 -- 172.21.15.86:0/848475843 learned_addr learned my addr 172.21.15.86:0/848475843 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T22:02:01.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:01.554+0000 7f7deffff700 1 -- 172.21.15.86:0/848475843 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7de4005040 con 0x7f7df00f5410 2024-02-19T22:02:01.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:01.555+0000 7f7deffff700 1 --2- 172.21.15.86:0/848475843 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7df00f5410 0x7f7df0109500 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7f7de4005010 tx=0x7f7de400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:02:01.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:01.555+0000 7f7decff9700 1 -- 172.21.15.86:0/848475843 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7de4009320 con 0x7f7df00f5410 2024-02-19T22:02:01.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:01.555+0000 7f7df629e700 1 -- 172.21.15.86:0/848475843 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7df0060420 con 0x7f7df00f5410 2024-02-19T22:02:01.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:01.555+0000 7f7decff9700 1 -- 172.21.15.86:0/848475843 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7de4013380 con 0x7f7df00f5410 2024-02-19T22:02:01.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:01.555+0000 7f7decff9700 1 -- 172.21.15.86:0/848475843 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7de4014e50 con 0x7f7df00f5410 2024-02-19T22:02:01.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:01.555+0000 7f7df629e700 1 -- 172.21.15.86:0/848475843 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7df0109bb0 con 0x7f7df00f5410 2024-02-19T22:02:01.557 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:01.555+0000 7f7decff9700 1 -- 172.21.15.86:0/848475843 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7de4014420 con 0x7f7df00f5410 2024-02-19T22:02:01.557 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:01.556+0000 7f7df629e700 1 -- 172.21.15.86:0/848475843 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7df0059100 con 0x7f7df00f5410 2024-02-19T22:02:01.557 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:01.556+0000 7f7decff9700 1 --2- 172.21.15.86:0/848475843 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f7dd802ff10 0x7f7dd80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:02:01.557 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:01.556+0000 7f7decff9700 1 -- 172.21.15.86:0/848475843 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7de4040ea0 con 0x7f7df00f5410 2024-02-19T22:02:01.557 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:01.556+0000 7f7def7fe700 1 --2- 172.21.15.86:0/848475843 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f7dd802ff10 0x7f7dd80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:02:01.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:01.556+0000 7f7def7fe700 1 --2- 172.21.15.86:0/848475843 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f7dd802ff10 0x7f7dd80323d0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f7de00096a0 tx=0x7f7de0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:02:01.561 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:01.561+0000 7f7decff9700 1 -- 172.21.15.86:0/848475843 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7de40095f0 con 0x7f7df00f5410 2024-02-19T22:02:01.765 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:01.766+0000 7f7df629e700 1 -- 172.21.15.86:0/848475843 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7df0045ad0 con 0x7f7df00f5410 2024-02-19T22:02:01.766 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:01.766+0000 7f7decff9700 1 -- 172.21.15.86:0/848475843 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7de401db70 con 0x7f7df00f5410 2024-02-19T22:02:01.766 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T22:02:01.766 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:02:01.767 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:01.768+0000 7f7df629e700 1 -- 172.21.15.86:0/848475843 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f7dd802ff10 msgr2=0x7f7dd80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:02:01.767 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:01.768+0000 7f7df629e700 1 --2- 172.21.15.86:0/848475843 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f7dd802ff10 0x7f7dd80323d0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f7de00096a0 tx=0x7f7de0006b40).stop 2024-02-19T22:02:01.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:01.768+0000 7f7df629e700 1 -- 172.21.15.86:0/848475843 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7df00f5410 msgr2=0x7f7df0109500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:02:01.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:01.768+0000 7f7df629e700 1 --2- 172.21.15.86:0/848475843 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7df00f5410 0x7f7df0109500 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7f7de4005010 tx=0x7f7de400b040).stop 2024-02-19T22:02:01.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:01.768+0000 7f7df629e700 1 -- 172.21.15.86:0/848475843 shutdown_connections 2024-02-19T22:02:01.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:01.768+0000 7f7df629e700 1 --2- 172.21.15.86:0/848475843 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f7dd802ff10 0x7f7dd80323d0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:02:01.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:01.768+0000 7f7df629e700 1 --2- 172.21.15.86:0/848475843 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7df00f5410 0x7f7df0109500 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:02:01.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:01.768+0000 7f7df629e700 1 -- 172.21.15.86:0/848475843 >> 172.21.15.86:0/848475843 conn(0x7f7df00f0b70 msgr2=0x7f7df00f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:02:01.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:01.768+0000 7f7df629e700 1 -- 172.21.15.86:0/848475843 shutdown_connections 2024-02-19T22:02:01.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:01.768+0000 7f7df629e700 1 -- 172.21.15.86:0/848475843 wait complete. 2024-02-19T22:02:01.768 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T22:02:02.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:02:02 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:02:01.486737+0000 mgr.smithi053.fyhcdi (mgr.14162) 168 : cluster [DBG] pgmap v157: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:02:02.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:02:02 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:02:01.767699+0000 mon.smithi053 (mon.0) 212 : audit [DBG] from='client.? 172.21.15.86:0/848475843' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:02:03.443 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:02:03.443 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:02:04.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:02:04 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:02:03.487020+0000 mgr.smithi053.fyhcdi (mgr.14162) 169 : cluster [DBG] pgmap v158: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:02:05.211 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:05.210+0000 7f8e45e67700 1 -- 172.21.15.86:0/2207692740 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8e400f6510 msgr2=0x7f8e400f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:02:05.211 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:05.210+0000 7f8e45e67700 1 --2- 172.21.15.86:0/2207692740 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8e400f6510 0x7f8e400f6930 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 rx=0x7f8e30009fb0 tx=0x7f8e30009b30).stop 2024-02-19T22:02:05.211 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:05.210+0000 7f8e45e67700 1 -- 172.21.15.86:0/2207692740 shutdown_connections 2024-02-19T22:02:05.211 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:05.210+0000 7f8e45e67700 1 --2- 172.21.15.86:0/2207692740 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8e400f6510 0x7f8e400f6930 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:02:05.211 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:05.210+0000 7f8e45e67700 1 -- 172.21.15.86:0/2207692740 >> 172.21.15.86:0/2207692740 conn(0x7f8e400f1c70 msgr2=0x7f8e400f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:02:05.212 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:05.210+0000 7f8e45e67700 1 -- 172.21.15.86:0/2207692740 shutdown_connections 2024-02-19T22:02:05.212 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:05.210+0000 7f8e45e67700 1 -- 172.21.15.86:0/2207692740 wait complete. 2024-02-19T22:02:05.212 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:05.210+0000 7f8e45e67700 1 Processor -- start 2024-02-19T22:02:05.212 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:05.210+0000 7f8e45e67700 1 -- start start 2024-02-19T22:02:05.212 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:05.211+0000 7f8e45e67700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8e400f6510 0x7f8e401067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:02:05.212 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:05.211+0000 7f8e45e67700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e40106cd0 con 0x7f8e400f6510 2024-02-19T22:02:05.212 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:05.211+0000 7f8e3f7fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8e400f6510 0x7f8e401067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:02:05.212 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:05.211+0000 7f8e3f7fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8e400f6510 0x7f8e401067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:57848/0 (socket says 172.21.15.86:57848) 2024-02-19T22:02:05.212 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:05.211+0000 7f8e3f7fe700 1 -- 172.21.15.86:0/3039401718 learned_addr learned my addr 172.21.15.86:0/3039401718 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T22:02:05.212 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:05.211+0000 7f8e3f7fe700 1 -- 172.21.15.86:0/3039401718 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e30005040 con 0x7f8e400f6510 2024-02-19T22:02:05.212 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:05.211+0000 7f8e3f7fe700 1 --2- 172.21.15.86:0/3039401718 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8e400f6510 0x7f8e401067e0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7f8e30005010 tx=0x7f8e3000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:02:05.212 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:05.211+0000 7f8e27fff700 1 -- 172.21.15.86:0/3039401718 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8e30009320 con 0x7f8e400f6510 2024-02-19T22:02:05.212 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:05.211+0000 7f8e45e67700 1 -- 172.21.15.86:0/3039401718 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e40106ed0 con 0x7f8e400f6510 2024-02-19T22:02:05.212 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:05.211+0000 7f8e27fff700 1 -- 172.21.15.86:0/3039401718 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8e30013380 con 0x7f8e400f6510 2024-02-19T22:02:05.213 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:05.211+0000 7f8e27fff700 1 -- 172.21.15.86:0/3039401718 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8e30014e50 con 0x7f8e400f6510 2024-02-19T22:02:05.213 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:05.211+0000 7f8e45e67700 1 -- 172.21.15.86:0/3039401718 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e40107360 con 0x7f8e400f6510 2024-02-19T22:02:05.213 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:05.212+0000 7f8e27fff700 1 -- 172.21.15.86:0/3039401718 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8e300095f0 con 0x7f8e400f6510 2024-02-19T22:02:05.214 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:05.212+0000 7f8e45e67700 1 -- 172.21.15.86:0/3039401718 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e40045ad0 con 0x7f8e400f6510 2024-02-19T22:02:05.214 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:05.212+0000 7f8e27fff700 1 --2- 172.21.15.86:0/3039401718 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f8e2802ff10 0x7f8e280323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:02:05.214 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:05.212+0000 7f8e27fff700 1 -- 172.21.15.86:0/3039401718 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8e30040fa0 con 0x7f8e400f6510 2024-02-19T22:02:05.214 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:05.213+0000 7f8e3effd700 1 --2- 172.21.15.86:0/3039401718 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f8e2802ff10 0x7f8e280323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:02:05.214 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:05.213+0000 7f8e3effd700 1 --2- 172.21.15.86:0/3039401718 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f8e2802ff10 0x7f8e280323d0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f8e340096a0 tx=0x7f8e34006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:02:05.216 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:05.217+0000 7f8e27fff700 1 -- 172.21.15.86:0/3039401718 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8e30014420 con 0x7f8e400f6510 2024-02-19T22:02:05.420 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:05.420+0000 7f8e45e67700 1 -- 172.21.15.86:0/3039401718 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8e40108210 con 0x7f8e400f6510 2024-02-19T22:02:05.421 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:05.421+0000 7f8e27fff700 1 -- 172.21.15.86:0/3039401718 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8e30010230 con 0x7f8e400f6510 2024-02-19T22:02:05.421 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T22:02:05.421 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:02:05.422 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:05.422+0000 7f8e45e67700 1 -- 172.21.15.86:0/3039401718 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f8e2802ff10 msgr2=0x7f8e280323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:02:05.423 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:05.422+0000 7f8e45e67700 1 --2- 172.21.15.86:0/3039401718 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f8e2802ff10 0x7f8e280323d0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f8e340096a0 tx=0x7f8e34006b40).stop 2024-02-19T22:02:05.423 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:05.422+0000 7f8e45e67700 1 -- 172.21.15.86:0/3039401718 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8e400f6510 msgr2=0x7f8e401067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:02:05.423 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:05.422+0000 7f8e45e67700 1 --2- 172.21.15.86:0/3039401718 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8e400f6510 0x7f8e401067e0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7f8e30005010 tx=0x7f8e3000b040).stop 2024-02-19T22:02:05.423 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:05.423+0000 7f8e45e67700 1 -- 172.21.15.86:0/3039401718 shutdown_connections 2024-02-19T22:02:05.423 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:05.423+0000 7f8e45e67700 1 --2- 172.21.15.86:0/3039401718 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f8e2802ff10 0x7f8e280323d0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:02:05.423 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:05.423+0000 7f8e45e67700 1 --2- 172.21.15.86:0/3039401718 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8e400f6510 0x7f8e401067e0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:02:05.423 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:05.423+0000 7f8e45e67700 1 -- 172.21.15.86:0/3039401718 >> 172.21.15.86:0/3039401718 conn(0x7f8e400f1c70 msgr2=0x7f8e400f26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:02:05.423 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:05.423+0000 7f8e45e67700 1 -- 172.21.15.86:0/3039401718 shutdown_connections 2024-02-19T22:02:05.423 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:05.423+0000 7f8e45e67700 1 -- 172.21.15.86:0/3039401718 wait complete. 2024-02-19T22:02:05.423 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T22:02:05.764 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:02:05 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:02:05.422420+0000 mon.smithi053 (mon.0) 213 : audit [DBG] from='client.? 172.21.15.86:0/3039401718' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:02:06.736 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:02:06 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:02:05.487309+0000 mgr.smithi053.fyhcdi (mgr.14162) 170 : cluster [DBG] pgmap v159: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:02:06.980 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:02:06.980 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:02:08.800 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:08.799+0000 7ffa448d8700 1 -- 172.21.15.86:0/2945880708 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffa400f47c0 msgr2=0x7ffa400f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:02:08.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:08.799+0000 7ffa448d8700 1 --2- 172.21.15.86:0/2945880708 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffa400f47c0 0x7ffa400f4be0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 rx=0x7ffa28009fb0 tx=0x7ffa28009b30).stop 2024-02-19T22:02:08.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:08.799+0000 7ffa448d8700 1 -- 172.21.15.86:0/2945880708 shutdown_connections 2024-02-19T22:02:08.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:08.799+0000 7ffa448d8700 1 --2- 172.21.15.86:0/2945880708 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffa400f47c0 0x7ffa400f4be0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:02:08.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:08.799+0000 7ffa448d8700 1 -- 172.21.15.86:0/2945880708 >> 172.21.15.86:0/2945880708 conn(0x7ffa400f02e0 msgr2=0x7ffa400f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:02:08.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:08.799+0000 7ffa448d8700 1 -- 172.21.15.86:0/2945880708 shutdown_connections 2024-02-19T22:02:08.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:08.799+0000 7ffa448d8700 1 -- 172.21.15.86:0/2945880708 wait complete. 2024-02-19T22:02:08.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:08.800+0000 7ffa448d8700 1 Processor -- start 2024-02-19T22:02:08.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:08.800+0000 7ffa448d8700 1 -- start start 2024-02-19T22:02:08.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:08.800+0000 7ffa448d8700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffa400f47c0 0x7ffa401067d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:02:08.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:08.800+0000 7ffa448d8700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa40106cc0 con 0x7ffa400f47c0 2024-02-19T22:02:08.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:08.800+0000 7ffa3e59c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffa400f47c0 0x7ffa401067d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:02:08.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:08.800+0000 7ffa3e59c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffa400f47c0 0x7ffa401067d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:57866/0 (socket says 172.21.15.86:57866) 2024-02-19T22:02:08.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:08.800+0000 7ffa3e59c700 1 -- 172.21.15.86:0/2014335566 learned_addr learned my addr 172.21.15.86:0/2014335566 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T22:02:08.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:08.800+0000 7ffa3e59c700 1 -- 172.21.15.86:0/2014335566 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa28005040 con 0x7ffa400f47c0 2024-02-19T22:02:08.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:08.801+0000 7ffa3e59c700 1 --2- 172.21.15.86:0/2014335566 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffa400f47c0 0x7ffa401067d0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7ffa28005010 tx=0x7ffa2800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:02:08.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:08.801+0000 7ffa36ffd700 1 -- 172.21.15.86:0/2014335566 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffa28009320 con 0x7ffa400f47c0 2024-02-19T22:02:08.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:08.801+0000 7ffa448d8700 1 -- 172.21.15.86:0/2014335566 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa40106ec0 con 0x7ffa400f47c0 2024-02-19T22:02:08.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:08.801+0000 7ffa36ffd700 1 -- 172.21.15.86:0/2014335566 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ffa28013380 con 0x7ffa400f47c0 2024-02-19T22:02:08.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:08.801+0000 7ffa36ffd700 1 -- 172.21.15.86:0/2014335566 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffa28014e50 con 0x7ffa400f47c0 2024-02-19T22:02:08.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:08.801+0000 7ffa448d8700 1 -- 172.21.15.86:0/2014335566 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa40107350 con 0x7ffa400f47c0 2024-02-19T22:02:08.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:08.802+0000 7ffa36ffd700 1 -- 172.21.15.86:0/2014335566 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ffa280095f0 con 0x7ffa400f47c0 2024-02-19T22:02:08.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:08.802+0000 7ffa36ffd700 1 --2- 172.21.15.86:0/2014335566 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7ffa2c02ff10 0x7ffa2c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:02:08.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:08.802+0000 7ffa448d8700 1 -- 172.21.15.86:0/2014335566 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa40045ad0 con 0x7ffa400f47c0 2024-02-19T22:02:08.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:08.802+0000 7ffa36ffd700 1 -- 172.21.15.86:0/2014335566 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ffa28040e90 con 0x7ffa400f47c0 2024-02-19T22:02:08.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:08.802+0000 7ffa3dd9b700 1 --2- 172.21.15.86:0/2014335566 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7ffa2c02ff10 0x7ffa2c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:02:08.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:08.802+0000 7ffa3dd9b700 1 --2- 172.21.15.86:0/2014335566 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7ffa2c02ff10 0x7ffa2c0323d0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7ffa300096a0 tx=0x7ffa30006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:02:08.806 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:08.806+0000 7ffa36ffd700 1 -- 172.21.15.86:0/2014335566 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ffa28014420 con 0x7ffa400f47c0 2024-02-19T22:02:09.011 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:09.011+0000 7ffa448d8700 1 -- 172.21.15.86:0/2014335566 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ffa40108200 con 0x7ffa400f47c0 2024-02-19T22:02:09.011 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:09.012+0000 7ffa36ffd700 1 -- 172.21.15.86:0/2014335566 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ffa28010230 con 0x7ffa400f47c0 2024-02-19T22:02:09.012 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T22:02:09.012 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:02:09.013 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:09.013+0000 7ffa448d8700 1 -- 172.21.15.86:0/2014335566 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7ffa2c02ff10 msgr2=0x7ffa2c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:02:09.014 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:09.013+0000 7ffa448d8700 1 --2- 172.21.15.86:0/2014335566 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7ffa2c02ff10 0x7ffa2c0323d0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7ffa300096a0 tx=0x7ffa30006b40).stop 2024-02-19T22:02:09.014 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:09.013+0000 7ffa448d8700 1 -- 172.21.15.86:0/2014335566 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffa400f47c0 msgr2=0x7ffa401067d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:02:09.014 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:09.013+0000 7ffa448d8700 1 --2- 172.21.15.86:0/2014335566 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffa400f47c0 0x7ffa401067d0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7ffa28005010 tx=0x7ffa2800b040).stop 2024-02-19T22:02:09.014 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:09.013+0000 7ffa448d8700 1 -- 172.21.15.86:0/2014335566 shutdown_connections 2024-02-19T22:02:09.014 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:09.013+0000 7ffa448d8700 1 --2- 172.21.15.86:0/2014335566 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7ffa2c02ff10 0x7ffa2c0323d0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:02:09.014 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:09.013+0000 7ffa448d8700 1 --2- 172.21.15.86:0/2014335566 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffa400f47c0 0x7ffa401067d0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:02:09.014 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:09.013+0000 7ffa448d8700 1 -- 172.21.15.86:0/2014335566 >> 172.21.15.86:0/2014335566 conn(0x7ffa400f02e0 msgr2=0x7ffa400f0d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:02:09.014 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:09.014+0000 7ffa448d8700 1 -- 172.21.15.86:0/2014335566 shutdown_connections 2024-02-19T22:02:09.014 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:09.014+0000 7ffa448d8700 1 -- 172.21.15.86:0/2014335566 wait complete. 2024-02-19T22:02:09.014 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T22:02:09.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:02:08 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:02:07.487547+0000 mgr.smithi053.fyhcdi (mgr.14162) 171 : cluster [DBG] pgmap v160: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:02:09.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:02:08 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:02:07.574568+0000 mon.smithi053 (mon.0) 214 : audit [INF] from='mgr.14162 172.21.15.53:0/2860358250' entity='mgr.smithi053.fyhcdi' 2024-02-19T22:02:10.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:02:09 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cephadm 2024-02-19T22:02:07.575885+0000 mgr.smithi053.fyhcdi (mgr.14162) 172 : cephadm [INF] Deploying daemon node-exporter.smithi053 on smithi053 2024-02-19T22:02:10.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:02:09 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:02:09.013030+0000 mon.smithi053 (mon.0) 215 : audit [DBG] from='client.? 172.21.15.86:0/2014335566' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:02:10.794 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:02:10.794 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:02:11.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:02:10 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:02:09.487819+0000 mgr.smithi053.fyhcdi (mgr.14162) 173 : cluster [DBG] pgmap v161: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:02:12.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:12.805+0000 7ff83df80700 1 -- 172.21.15.86:0/4164337074 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff8380f6510 msgr2=0x7ff8380f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:02:12.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:12.805+0000 7ff83df80700 1 --2- 172.21.15.86:0/4164337074 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff8380f6510 0x7ff8380f6930 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 rx=0x7ff82c009fb0 tx=0x7ff82c009b30).stop 2024-02-19T22:02:12.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:12.805+0000 7ff83df80700 1 -- 172.21.15.86:0/4164337074 shutdown_connections 2024-02-19T22:02:12.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:12.805+0000 7ff83df80700 1 --2- 172.21.15.86:0/4164337074 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff8380f6510 0x7ff8380f6930 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:02:12.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:12.805+0000 7ff83df80700 1 -- 172.21.15.86:0/4164337074 >> 172.21.15.86:0/4164337074 conn(0x7ff8380f1c70 msgr2=0x7ff8380f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:02:12.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:12.805+0000 7ff83df80700 1 -- 172.21.15.86:0/4164337074 shutdown_connections 2024-02-19T22:02:12.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:12.805+0000 7ff83df80700 1 -- 172.21.15.86:0/4164337074 wait complete. 2024-02-19T22:02:12.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:12.806+0000 7ff83df80700 1 Processor -- start 2024-02-19T22:02:12.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:12.806+0000 7ff83df80700 1 -- start start 2024-02-19T22:02:12.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:12.806+0000 7ff83df80700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff8380f6510 0x7ff8381067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:02:12.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:12.806+0000 7ff83df80700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff838106cd0 con 0x7ff8380f6510 2024-02-19T22:02:12.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:12.806+0000 7ff8377fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff8380f6510 0x7ff8381067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:02:12.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:12.806+0000 7ff8377fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff8380f6510 0x7ff8381067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:53698/0 (socket says 172.21.15.86:53698) 2024-02-19T22:02:12.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:12.806+0000 7ff8377fe700 1 -- 172.21.15.86:0/2661279153 learned_addr learned my addr 172.21.15.86:0/2661279153 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T22:02:12.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:12.807+0000 7ff8377fe700 1 -- 172.21.15.86:0/2661279153 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff82c005040 con 0x7ff8380f6510 2024-02-19T22:02:12.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:12.807+0000 7ff8377fe700 1 --2- 172.21.15.86:0/2661279153 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff8380f6510 0x7ff8381067e0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7ff82c005010 tx=0x7ff82c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:02:12.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:12.807+0000 7ff81ffff700 1 -- 172.21.15.86:0/2661279153 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff82c009320 con 0x7ff8380f6510 2024-02-19T22:02:12.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:12.807+0000 7ff83df80700 1 -- 172.21.15.86:0/2661279153 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff838106ed0 con 0x7ff8380f6510 2024-02-19T22:02:12.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:12.807+0000 7ff81ffff700 1 -- 172.21.15.86:0/2661279153 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff82c013380 con 0x7ff8380f6510 2024-02-19T22:02:12.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:12.807+0000 7ff81ffff700 1 -- 172.21.15.86:0/2661279153 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff82c014e50 con 0x7ff8380f6510 2024-02-19T22:02:12.809 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:12.807+0000 7ff83df80700 1 -- 172.21.15.86:0/2661279153 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff838107360 con 0x7ff8380f6510 2024-02-19T22:02:12.809 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:12.808+0000 7ff81ffff700 1 -- 172.21.15.86:0/2661279153 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff82c014420 con 0x7ff8380f6510 2024-02-19T22:02:12.810 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:12.808+0000 7ff83df80700 1 -- 172.21.15.86:0/2661279153 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff838045ad0 con 0x7ff8380f6510 2024-02-19T22:02:12.810 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:12.808+0000 7ff81ffff700 1 --2- 172.21.15.86:0/2661279153 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7ff82002ff10 0x7ff8200323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:02:12.810 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:12.808+0000 7ff81ffff700 1 -- 172.21.15.86:0/2661279153 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff82c040f70 con 0x7ff8380f6510 2024-02-19T22:02:12.810 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:12.808+0000 7ff836ffd700 1 --2- 172.21.15.86:0/2661279153 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7ff82002ff10 0x7ff8200323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:02:12.810 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:12.808+0000 7ff836ffd700 1 --2- 172.21.15.86:0/2661279153 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7ff82002ff10 0x7ff8200323d0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7ff8280096a0 tx=0x7ff828006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:02:12.812 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:12.813+0000 7ff81ffff700 1 -- 172.21.15.86:0/2661279153 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff82c0095f0 con 0x7ff8380f6510 2024-02-19T22:02:13.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:13.023+0000 7ff83df80700 1 -- 172.21.15.86:0/2661279153 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff838108210 con 0x7ff8380f6510 2024-02-19T22:02:13.024 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:13.024+0000 7ff81ffff700 1 -- 172.21.15.86:0/2661279153 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff82c01db70 con 0x7ff8380f6510 2024-02-19T22:02:13.024 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T22:02:13.024 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:02:13.026 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:13.026+0000 7ff83df80700 1 -- 172.21.15.86:0/2661279153 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7ff82002ff10 msgr2=0x7ff8200323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:02:13.026 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:13.026+0000 7ff83df80700 1 --2- 172.21.15.86:0/2661279153 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7ff82002ff10 0x7ff8200323d0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7ff8280096a0 tx=0x7ff828006b40).stop 2024-02-19T22:02:13.026 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:13.026+0000 7ff83df80700 1 -- 172.21.15.86:0/2661279153 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff8380f6510 msgr2=0x7ff8381067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:02:13.027 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:13.026+0000 7ff83df80700 1 --2- 172.21.15.86:0/2661279153 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff8380f6510 0x7ff8381067e0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7ff82c005010 tx=0x7ff82c00b040).stop 2024-02-19T22:02:13.027 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:13.026+0000 7ff83df80700 1 -- 172.21.15.86:0/2661279153 shutdown_connections 2024-02-19T22:02:13.027 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:13.026+0000 7ff83df80700 1 --2- 172.21.15.86:0/2661279153 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7ff82002ff10 0x7ff8200323d0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:02:13.027 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:13.026+0000 7ff83df80700 1 --2- 172.21.15.86:0/2661279153 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff8380f6510 0x7ff8381067e0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:02:13.027 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:13.026+0000 7ff83df80700 1 -- 172.21.15.86:0/2661279153 >> 172.21.15.86:0/2661279153 conn(0x7ff8380f1c70 msgr2=0x7ff8380f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:02:13.027 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:13.026+0000 7ff83df80700 1 -- 172.21.15.86:0/2661279153 shutdown_connections 2024-02-19T22:02:13.027 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:13.026+0000 7ff83df80700 1 -- 172.21.15.86:0/2661279153 wait complete. 2024-02-19T22:02:13.028 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T22:02:13.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:02:12 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:02:11.488054+0000 mgr.smithi053.fyhcdi (mgr.14162) 174 : cluster [DBG] pgmap v162: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:02:14.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:02:13 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:02:13.025352+0000 mon.smithi053 (mon.0) 216 : audit [DBG] from='client.? 172.21.15.86:0/2661279153' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:02:14.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:02:13 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:02:13.488297+0000 mgr.smithi053.fyhcdi (mgr.14162) 175 : cluster [DBG] pgmap v163: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:02:14.671 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:02:14.671 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:02:16.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:02:16 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:02:15.488604+0000 mgr.smithi053.fyhcdi (mgr.14162) 176 : cluster [DBG] pgmap v164: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:02:16.826 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:16.826+0000 7f414dbb7700 1 -- 172.21.15.86:0/3710257161 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f41480f78e0 msgr2=0x7f41480f7d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:02:16.826 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:16.826+0000 7f414dbb7700 1 --2- 172.21.15.86:0/3710257161 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f41480f78e0 0x7f41480f7d00 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 rx=0x7f413c009fb0 tx=0x7f413c009b30).stop 2024-02-19T22:02:16.826 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:16.826+0000 7f414dbb7700 1 -- 172.21.15.86:0/3710257161 shutdown_connections 2024-02-19T22:02:16.826 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:16.826+0000 7f414dbb7700 1 --2- 172.21.15.86:0/3710257161 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f41480f78e0 0x7f41480f7d00 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:02:16.826 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:16.826+0000 7f414dbb7700 1 -- 172.21.15.86:0/3710257161 >> 172.21.15.86:0/3710257161 conn(0x7f41480f0120 msgr2=0x7f41480f2540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:02:16.827 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:16.826+0000 7f414dbb7700 1 -- 172.21.15.86:0/3710257161 shutdown_connections 2024-02-19T22:02:16.827 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:16.826+0000 7f414dbb7700 1 -- 172.21.15.86:0/3710257161 wait complete. 2024-02-19T22:02:16.827 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:16.826+0000 7f414dbb7700 1 Processor -- start 2024-02-19T22:02:16.827 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:16.826+0000 7f414dbb7700 1 -- start start 2024-02-19T22:02:16.827 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:16.827+0000 7f414dbb7700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f41480f78e0 0x7f41481065e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:02:16.827 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:16.827+0000 7f414dbb7700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4148106ad0 con 0x7f41480f78e0 2024-02-19T22:02:16.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:16.827+0000 7f414cbb5700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f41480f78e0 0x7f41481065e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:02:16.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:16.827+0000 7f414cbb5700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f41480f78e0 0x7f41481065e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:53712/0 (socket says 172.21.15.86:53712) 2024-02-19T22:02:16.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:16.827+0000 7f414cbb5700 1 -- 172.21.15.86:0/541543252 learned_addr learned my addr 172.21.15.86:0/541543252 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T22:02:16.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:16.827+0000 7f414cbb5700 1 -- 172.21.15.86:0/541543252 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f413c005040 con 0x7f41480f78e0 2024-02-19T22:02:16.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:16.827+0000 7f414cbb5700 1 --2- 172.21.15.86:0/541543252 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f41480f78e0 0x7f41481065e0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7f413c0089f0 tx=0x7f413c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:02:16.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:16.827+0000 7f41457fa700 1 -- 172.21.15.86:0/541543252 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f413c009320 con 0x7f41480f78e0 2024-02-19T22:02:16.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:16.827+0000 7f414dbb7700 1 -- 172.21.15.86:0/541543252 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4148106cd0 con 0x7f41480f78e0 2024-02-19T22:02:16.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:16.827+0000 7f41457fa700 1 -- 172.21.15.86:0/541543252 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f413c013380 con 0x7f41480f78e0 2024-02-19T22:02:16.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:16.827+0000 7f41457fa700 1 -- 172.21.15.86:0/541543252 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f413c014e20 con 0x7f41480f78e0 2024-02-19T22:02:16.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:16.827+0000 7f414dbb7700 1 -- 172.21.15.86:0/541543252 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4148107160 con 0x7f41480f78e0 2024-02-19T22:02:16.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:16.828+0000 7f41457fa700 1 -- 172.21.15.86:0/541543252 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f413c014420 con 0x7f41480f78e0 2024-02-19T22:02:16.830 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:16.828+0000 7f414dbb7700 1 -- 172.21.15.86:0/541543252 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4148045ad0 con 0x7f41480f78e0 2024-02-19T22:02:16.830 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:16.828+0000 7f41457fa700 1 --2- 172.21.15.86:0/541543252 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f413802ff10 0x7f41380323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:02:16.830 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:16.828+0000 7f41457fa700 1 -- 172.21.15.86:0/541543252 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f413c040fc0 con 0x7f41480f78e0 2024-02-19T22:02:16.830 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:16.829+0000 7f4147fff700 1 --2- 172.21.15.86:0/541543252 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f413802ff10 0x7f41380323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:02:16.830 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:16.829+0000 7f4147fff700 1 --2- 172.21.15.86:0/541543252 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f413802ff10 0x7f41380323d0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f41340096a0 tx=0x7f4134006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:02:16.833 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:16.833+0000 7f41457fa700 1 -- 172.21.15.86:0/541543252 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f413c06d2b0 con 0x7f41480f78e0 2024-02-19T22:02:17.039 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:17.039+0000 7f414dbb7700 1 -- 172.21.15.86:0/541543252 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4148108010 con 0x7f41480f78e0 2024-02-19T22:02:17.040 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:17.040+0000 7f41457fa700 1 -- 172.21.15.86:0/541543252 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f413c0095f0 con 0x7f41480f78e0 2024-02-19T22:02:17.040 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T22:02:17.040 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:02:17.041 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:17.041+0000 7f414dbb7700 1 -- 172.21.15.86:0/541543252 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f413802ff10 msgr2=0x7f41380323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:02:17.041 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:17.041+0000 7f414dbb7700 1 --2- 172.21.15.86:0/541543252 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f413802ff10 0x7f41380323d0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f41340096a0 tx=0x7f4134006b40).stop 2024-02-19T22:02:17.041 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:17.041+0000 7f414dbb7700 1 -- 172.21.15.86:0/541543252 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f41480f78e0 msgr2=0x7f41481065e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:02:17.042 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:17.041+0000 7f414dbb7700 1 --2- 172.21.15.86:0/541543252 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f41480f78e0 0x7f41481065e0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7f413c0089f0 tx=0x7f413c00b040).stop 2024-02-19T22:02:17.042 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:17.042+0000 7f414dbb7700 1 -- 172.21.15.86:0/541543252 shutdown_connections 2024-02-19T22:02:17.042 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:17.042+0000 7f414dbb7700 1 --2- 172.21.15.86:0/541543252 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f413802ff10 0x7f41380323d0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:02:17.042 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:17.042+0000 7f414dbb7700 1 --2- 172.21.15.86:0/541543252 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f41480f78e0 0x7f41481065e0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:02:17.042 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:17.042+0000 7f414dbb7700 1 -- 172.21.15.86:0/541543252 >> 172.21.15.86:0/541543252 conn(0x7f41480f0120 msgr2=0x7f41480f0b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:02:17.042 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:17.042+0000 7f414dbb7700 1 -- 172.21.15.86:0/541543252 shutdown_connections 2024-02-19T22:02:17.042 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:17.042+0000 7f414dbb7700 1 -- 172.21.15.86:0/541543252 wait complete. 2024-02-19T22:02:17.042 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T22:02:17.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:02:17 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:02:17.041275+0000 mon.smithi053 (mon.0) 217 : audit [DBG] from='client.? 172.21.15.86:0/541543252' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:02:18.722 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:02:18.722 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:02:18.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:02:18 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:02:17.488957+0000 mgr.smithi053.fyhcdi (mgr.14162) 177 : cluster [DBG] pgmap v165: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:02:20.648 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:20.646+0000 7f71297d1700 1 -- 172.21.15.86:0/506019072 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f71240f47c0 msgr2=0x7f71240f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:02:20.649 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:20.646+0000 7f71297d1700 1 --2- 172.21.15.86:0/506019072 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f71240f47c0 0x7f71240f4be0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 rx=0x7f7118009fb0 tx=0x7f7118009b30).stop 2024-02-19T22:02:20.649 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:20.646+0000 7f71297d1700 1 -- 172.21.15.86:0/506019072 shutdown_connections 2024-02-19T22:02:20.649 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:20.646+0000 7f71297d1700 1 --2- 172.21.15.86:0/506019072 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f71240f47c0 0x7f71240f4be0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:02:20.649 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:20.646+0000 7f71297d1700 1 -- 172.21.15.86:0/506019072 >> 172.21.15.86:0/506019072 conn(0x7f71240f02e0 msgr2=0x7f71240f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:02:20.649 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:20.646+0000 7f71297d1700 1 -- 172.21.15.86:0/506019072 shutdown_connections 2024-02-19T22:02:20.650 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:20.646+0000 7f71297d1700 1 -- 172.21.15.86:0/506019072 wait complete. 2024-02-19T22:02:20.650 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:20.647+0000 7f71297d1700 1 Processor -- start 2024-02-19T22:02:20.650 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:20.647+0000 7f71297d1700 1 -- start start 2024-02-19T22:02:20.650 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:20.647+0000 7f71297d1700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f71240f47c0 0x7f71241067d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:02:20.650 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:20.647+0000 7f71297d1700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7124106cc0 con 0x7f71240f47c0 2024-02-19T22:02:20.650 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:20.647+0000 7f7122ffd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f71240f47c0 0x7f71241067d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:02:20.650 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:20.647+0000 7f7122ffd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f71240f47c0 0x7f71241067d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:42740/0 (socket says 172.21.15.86:42740) 2024-02-19T22:02:20.651 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:20.647+0000 7f7122ffd700 1 -- 172.21.15.86:0/584984037 learned_addr learned my addr 172.21.15.86:0/584984037 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T22:02:20.651 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:20.648+0000 7f7122ffd700 1 -- 172.21.15.86:0/584984037 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7118005040 con 0x7f71240f47c0 2024-02-19T22:02:20.651 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:20.648+0000 7f7122ffd700 1 --2- 172.21.15.86:0/584984037 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f71240f47c0 0x7f71241067d0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7f7118005010 tx=0x7f711800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:02:20.651 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:20.648+0000 7f710b7fe700 1 -- 172.21.15.86:0/584984037 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7118009320 con 0x7f71240f47c0 2024-02-19T22:02:20.651 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:20.648+0000 7f71297d1700 1 -- 172.21.15.86:0/584984037 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7124106ec0 con 0x7f71240f47c0 2024-02-19T22:02:20.651 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:20.648+0000 7f710b7fe700 1 -- 172.21.15.86:0/584984037 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7118013380 con 0x7f71240f47c0 2024-02-19T22:02:20.651 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:20.648+0000 7f710b7fe700 1 -- 172.21.15.86:0/584984037 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7118014e50 con 0x7f71240f47c0 2024-02-19T22:02:20.651 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:20.648+0000 7f71297d1700 1 -- 172.21.15.86:0/584984037 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7124107350 con 0x7f71240f47c0 2024-02-19T22:02:20.652 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:20.649+0000 7f710b7fe700 1 -- 172.21.15.86:0/584984037 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f71180095f0 con 0x7f71240f47c0 2024-02-19T22:02:20.652 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:20.649+0000 7f71297d1700 1 -- 172.21.15.86:0/584984037 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7124045ad0 con 0x7f71240f47c0 2024-02-19T22:02:20.652 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:20.649+0000 7f710b7fe700 1 --2- 172.21.15.86:0/584984037 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f710c02ff10 0x7f710c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:02:20.652 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:20.649+0000 7f710b7fe700 1 -- 172.21.15.86:0/584984037 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7118040e90 con 0x7f71240f47c0 2024-02-19T22:02:20.652 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:20.649+0000 7f71227fc700 1 --2- 172.21.15.86:0/584984037 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f710c02ff10 0x7f710c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:02:20.652 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:20.649+0000 7f71227fc700 1 --2- 172.21.15.86:0/584984037 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f710c02ff10 0x7f710c0323d0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f71140096a0 tx=0x7f7114006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:02:20.653 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:20.654+0000 7f710b7fe700 1 -- 172.21.15.86:0/584984037 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7118014420 con 0x7f71240f47c0 2024-02-19T22:02:20.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:02:20 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:02:19.489341+0000 mgr.smithi053.fyhcdi (mgr.14162) 178 : cluster [DBG] pgmap v166: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:02:20.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:20.860+0000 7f71297d1700 1 -- 172.21.15.86:0/584984037 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7124108200 con 0x7f71240f47c0 2024-02-19T22:02:20.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:20.861+0000 7f710b7fe700 1 -- 172.21.15.86:0/584984037 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7118010230 con 0x7f71240f47c0 2024-02-19T22:02:20.860 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T22:02:20.860 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:02:20.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:20.862+0000 7f71297d1700 1 -- 172.21.15.86:0/584984037 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f710c02ff10 msgr2=0x7f710c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:02:20.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:20.862+0000 7f71297d1700 1 --2- 172.21.15.86:0/584984037 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f710c02ff10 0x7f710c0323d0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f71140096a0 tx=0x7f7114006b40).stop 2024-02-19T22:02:20.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:20.862+0000 7f71297d1700 1 -- 172.21.15.86:0/584984037 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f71240f47c0 msgr2=0x7f71241067d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:02:20.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:20.862+0000 7f71297d1700 1 --2- 172.21.15.86:0/584984037 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f71240f47c0 0x7f71241067d0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7f7118005010 tx=0x7f711800b040).stop 2024-02-19T22:02:20.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:20.862+0000 7f71297d1700 1 -- 172.21.15.86:0/584984037 shutdown_connections 2024-02-19T22:02:20.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:20.862+0000 7f71297d1700 1 --2- 172.21.15.86:0/584984037 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f710c02ff10 0x7f710c0323d0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:02:20.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:20.862+0000 7f71297d1700 1 --2- 172.21.15.86:0/584984037 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f71240f47c0 0x7f71241067d0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:02:20.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:20.862+0000 7f71297d1700 1 -- 172.21.15.86:0/584984037 >> 172.21.15.86:0/584984037 conn(0x7f71240f02e0 msgr2=0x7f71240f0d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:02:20.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:20.862+0000 7f71297d1700 1 -- 172.21.15.86:0/584984037 shutdown_connections 2024-02-19T22:02:20.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:20.862+0000 7f71297d1700 1 -- 172.21.15.86:0/584984037 wait complete. 2024-02-19T22:02:20.862 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T22:02:21.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:02:21 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:02:20.860809+0000 mon.smithi053 (mon.0) 218 : audit [DBG] from='client.? 172.21.15.86:0/584984037' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:02:22.566 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:02:22.566 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:02:22.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:02:22 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:02:21.489562+0000 mgr.smithi053.fyhcdi (mgr.14162) 179 : cluster [DBG] pgmap v167: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:02:24.658 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:24.621+0000 7f3cce97b700 1 -- 172.21.15.86:0/1198330819 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3cc80f51a0 msgr2=0x7f3cc80f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:02:24.658 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:24.621+0000 7f3cce97b700 1 --2- 172.21.15.86:0/1198330819 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3cc80f51a0 0x7f3cc80f55c0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 rx=0x7f3cc4009fb0 tx=0x7f3cc4009b30).stop 2024-02-19T22:02:24.658 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:24.621+0000 7f3cce97b700 1 -- 172.21.15.86:0/1198330819 shutdown_connections 2024-02-19T22:02:24.658 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:24.621+0000 7f3cce97b700 1 --2- 172.21.15.86:0/1198330819 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3cc80f51a0 0x7f3cc80f55c0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:02:24.658 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:24.621+0000 7f3cce97b700 1 -- 172.21.15.86:0/1198330819 >> 172.21.15.86:0/1198330819 conn(0x7f3cc80f09a0 msgr2=0x7f3cc80f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:02:24.658 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:24.621+0000 7f3cce97b700 1 -- 172.21.15.86:0/1198330819 shutdown_connections 2024-02-19T22:02:24.658 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:24.621+0000 7f3cce97b700 1 -- 172.21.15.86:0/1198330819 wait complete. 2024-02-19T22:02:24.658 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:24.621+0000 7f3cce97b700 1 Processor -- start 2024-02-19T22:02:24.658 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:24.621+0000 7f3cce97b700 1 -- start start 2024-02-19T22:02:24.658 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:24.621+0000 7f3cce97b700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3cc80f51a0 0x7f3cc80fd270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:02:24.658 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:24.622+0000 7f3cce97b700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cc80fb6f0 con 0x7f3cc80f51a0 2024-02-19T22:02:24.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:24.622+0000 7f3ccd979700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3cc80f51a0 0x7f3cc80fd270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:02:24.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:24.622+0000 7f3ccd979700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3cc80f51a0 0x7f3cc80fd270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:42758/0 (socket says 172.21.15.86:42758) 2024-02-19T22:02:24.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:24.622+0000 7f3ccd979700 1 -- 172.21.15.86:0/321235737 learned_addr learned my addr 172.21.15.86:0/321235737 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T22:02:24.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:24.622+0000 7f3ccd979700 1 -- 172.21.15.86:0/321235737 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cc4005040 con 0x7f3cc80f51a0 2024-02-19T22:02:24.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:24.622+0000 7f3ccd979700 1 --2- 172.21.15.86:0/321235737 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3cc80f51a0 0x7f3cc80fd270 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7f3cc4009f80 tx=0x7f3cc400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:02:24.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:24.622+0000 7f3cbe7fc700 1 -- 172.21.15.86:0/321235737 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3cc40095f0 con 0x7f3cc80f51a0 2024-02-19T22:02:24.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:24.622+0000 7f3cce97b700 1 -- 172.21.15.86:0/321235737 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cc80fb890 con 0x7f3cc80f51a0 2024-02-19T22:02:24.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:24.622+0000 7f3cbe7fc700 1 -- 172.21.15.86:0/321235737 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3cc4007e90 con 0x7f3cc80f51a0 2024-02-19T22:02:24.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:24.622+0000 7f3cbe7fc700 1 -- 172.21.15.86:0/321235737 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3cc4014d80 con 0x7f3cc80f51a0 2024-02-19T22:02:24.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:24.622+0000 7f3cce97b700 1 -- 172.21.15.86:0/321235737 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cc80fbd20 con 0x7f3cc80f51a0 2024-02-19T22:02:24.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:24.623+0000 7f3cbe7fc700 1 -- 172.21.15.86:0/321235737 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3cc4025070 con 0x7f3cc80f51a0 2024-02-19T22:02:24.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:24.623+0000 7f3cbe7fc700 1 --2- 172.21.15.86:0/321235737 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f3cb402ff60 0x7f3cb4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:02:24.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:24.623+0000 7f3cce97b700 1 -- 172.21.15.86:0/321235737 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3cc8059100 con 0x7f3cc80f51a0 2024-02-19T22:02:24.660 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:24.623+0000 7f3cbe7fc700 1 -- 172.21.15.86:0/321235737 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3cc40408f0 con 0x7f3cc80f51a0 2024-02-19T22:02:24.660 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:24.624+0000 7f3ccd178700 1 --2- 172.21.15.86:0/321235737 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f3cb402ff60 0x7f3cb4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:02:24.660 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:24.624+0000 7f3ccd178700 1 --2- 172.21.15.86:0/321235737 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f3cb402ff60 0x7f3cb4032420 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f3cb80096a0 tx=0x7f3cb8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:02:24.660 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:24.628+0000 7f3cbe7fc700 1 -- 172.21.15.86:0/321235737 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3cc400f4e0 con 0x7f3cc80f51a0 2024-02-19T22:02:24.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:02:24 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:02:23.489847+0000 mgr.smithi053.fyhcdi (mgr.14162) 180 : cluster [DBG] pgmap v168: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:02:24.832 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:24.833+0000 7f3cce97b700 1 -- 172.21.15.86:0/321235737 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3cc80fad30 con 0x7f3cc80f51a0 2024-02-19T22:02:24.832 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:24.834+0000 7f3cbe7fc700 1 -- 172.21.15.86:0/321235737 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3cc400f300 con 0x7f3cc80f51a0 2024-02-19T22:02:24.833 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T22:02:24.833 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:02:24.834 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:24.835+0000 7f3cce97b700 1 -- 172.21.15.86:0/321235737 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f3cb402ff60 msgr2=0x7f3cb4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:02:24.834 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:24.835+0000 7f3cce97b700 1 --2- 172.21.15.86:0/321235737 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f3cb402ff60 0x7f3cb4032420 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f3cb80096a0 tx=0x7f3cb8006b40).stop 2024-02-19T22:02:24.834 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:24.835+0000 7f3cce97b700 1 -- 172.21.15.86:0/321235737 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3cc80f51a0 msgr2=0x7f3cc80fd270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:02:24.835 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:24.835+0000 7f3cce97b700 1 --2- 172.21.15.86:0/321235737 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3cc80f51a0 0x7f3cc80fd270 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7f3cc4009f80 tx=0x7f3cc400b040).stop 2024-02-19T22:02:24.835 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:24.835+0000 7f3cce97b700 1 -- 172.21.15.86:0/321235737 shutdown_connections 2024-02-19T22:02:24.835 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:24.835+0000 7f3cce97b700 1 --2- 172.21.15.86:0/321235737 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f3cb402ff60 0x7f3cb4032420 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:02:24.835 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:24.835+0000 7f3cce97b700 1 --2- 172.21.15.86:0/321235737 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3cc80f51a0 0x7f3cc80fd270 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:02:24.835 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:24.835+0000 7f3cce97b700 1 -- 172.21.15.86:0/321235737 >> 172.21.15.86:0/321235737 conn(0x7f3cc80f09a0 msgr2=0x7f3cc805fe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:02:24.835 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:24.835+0000 7f3cce97b700 1 -- 172.21.15.86:0/321235737 shutdown_connections 2024-02-19T22:02:24.835 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:24.835+0000 7f3cce97b700 1 -- 172.21.15.86:0/321235737 wait complete. 2024-02-19T22:02:24.835 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T22:02:25.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:02:25 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:02:24.834068+0000 mon.smithi053 (mon.0) 219 : audit [DBG] from='client.? 172.21.15.86:0/321235737' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:02:26.714 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:02:26.714 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:02:27.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:02:26 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:02:25.490146+0000 mgr.smithi053.fyhcdi (mgr.14162) 181 : cluster [DBG] pgmap v169: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:02:28.656 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:28.654+0000 7ffa7c5b3700 1 -- 172.21.15.86:0/1950912633 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffa740f6510 msgr2=0x7ffa740f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:02:28.656 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:28.654+0000 7ffa7c5b3700 1 --2- 172.21.15.86:0/1950912633 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffa740f6510 0x7ffa740f6930 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 rx=0x7ffa70009fb0 tx=0x7ffa70009b30).stop 2024-02-19T22:02:28.656 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:28.655+0000 7ffa7c5b3700 1 -- 172.21.15.86:0/1950912633 shutdown_connections 2024-02-19T22:02:28.656 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:28.655+0000 7ffa7c5b3700 1 --2- 172.21.15.86:0/1950912633 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffa740f6510 0x7ffa740f6930 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:02:28.657 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:28.655+0000 7ffa7c5b3700 1 -- 172.21.15.86:0/1950912633 >> 172.21.15.86:0/1950912633 conn(0x7ffa740f1c70 msgr2=0x7ffa740f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:02:28.657 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:28.655+0000 7ffa7c5b3700 1 -- 172.21.15.86:0/1950912633 shutdown_connections 2024-02-19T22:02:28.657 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:28.655+0000 7ffa7c5b3700 1 -- 172.21.15.86:0/1950912633 wait complete. 2024-02-19T22:02:28.657 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:28.655+0000 7ffa7c5b3700 1 Processor -- start 2024-02-19T22:02:28.657 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:28.655+0000 7ffa7c5b3700 1 -- start start 2024-02-19T22:02:28.657 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:28.655+0000 7ffa7c5b3700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffa740f6510 0x7ffa741067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:02:28.657 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:28.655+0000 7ffa7c5b3700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa74106cd0 con 0x7ffa740f6510 2024-02-19T22:02:28.657 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:28.655+0000 7ffa7a34f700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffa740f6510 0x7ffa741067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:02:28.657 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:28.656+0000 7ffa7a34f700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffa740f6510 0x7ffa741067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:42782/0 (socket says 172.21.15.86:42782) 2024-02-19T22:02:28.657 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:28.656+0000 7ffa7a34f700 1 -- 172.21.15.86:0/115644374 learned_addr learned my addr 172.21.15.86:0/115644374 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T22:02:28.657 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:28.656+0000 7ffa7a34f700 1 -- 172.21.15.86:0/115644374 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa70005040 con 0x7ffa740f6510 2024-02-19T22:02:28.657 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:28.656+0000 7ffa7a34f700 1 --2- 172.21.15.86:0/115644374 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffa740f6510 0x7ffa741067e0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7ffa700089f0 tx=0x7ffa7000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:02:28.657 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:28.656+0000 7ffa6affd700 1 -- 172.21.15.86:0/115644374 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffa70009320 con 0x7ffa740f6510 2024-02-19T22:02:28.657 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:28.656+0000 7ffa7c5b3700 1 -- 172.21.15.86:0/115644374 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa74106ed0 con 0x7ffa740f6510 2024-02-19T22:02:28.658 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:28.656+0000 7ffa6affd700 1 -- 172.21.15.86:0/115644374 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ffa70013380 con 0x7ffa740f6510 2024-02-19T22:02:28.658 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:28.656+0000 7ffa6affd700 1 -- 172.21.15.86:0/115644374 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffa7001ed80 con 0x7ffa740f6510 2024-02-19T22:02:28.658 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:28.656+0000 7ffa7c5b3700 1 -- 172.21.15.86:0/115644374 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa74107360 con 0x7ffa740f6510 2024-02-19T22:02:28.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:28.657+0000 7ffa6affd700 1 -- 172.21.15.86:0/115644374 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ffa7001e420 con 0x7ffa740f6510 2024-02-19T22:02:28.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:28.657+0000 7ffa7c5b3700 1 -- 172.21.15.86:0/115644374 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa74045ad0 con 0x7ffa740f6510 2024-02-19T22:02:28.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:28.657+0000 7ffa6affd700 1 --2- 172.21.15.86:0/115644374 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7ffa6002ff60 0x7ffa60032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:02:28.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:28.657+0000 7ffa6affd700 1 -- 172.21.15.86:0/115644374 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ffa70041090 con 0x7ffa740f6510 2024-02-19T22:02:28.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:28.657+0000 7ffa79b4e700 1 --2- 172.21.15.86:0/115644374 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7ffa6002ff60 0x7ffa60032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:02:28.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:28.658+0000 7ffa79b4e700 1 --2- 172.21.15.86:0/115644374 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7ffa6002ff60 0x7ffa60032420 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7ffa640096a0 tx=0x7ffa64006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:02:28.661 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:28.662+0000 7ffa6affd700 1 -- 172.21.15.86:0/115644374 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ffa700095f0 con 0x7ffa740f6510 2024-02-19T22:02:28.865 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:28.866+0000 7ffa7c5b3700 1 -- 172.21.15.86:0/115644374 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ffa74108210 con 0x7ffa740f6510 2024-02-19T22:02:28.866 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:28.866+0000 7ffa6affd700 1 -- 172.21.15.86:0/115644374 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ffa70004490 con 0x7ffa740f6510 2024-02-19T22:02:28.866 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T22:02:28.866 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:02:28.868 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:28.868+0000 7ffa7c5b3700 1 -- 172.21.15.86:0/115644374 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7ffa6002ff60 msgr2=0x7ffa60032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:02:28.868 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:28.868+0000 7ffa7c5b3700 1 --2- 172.21.15.86:0/115644374 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7ffa6002ff60 0x7ffa60032420 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7ffa640096a0 tx=0x7ffa64006b40).stop 2024-02-19T22:02:28.868 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:28.868+0000 7ffa7c5b3700 1 -- 172.21.15.86:0/115644374 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffa740f6510 msgr2=0x7ffa741067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:02:28.868 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:28.868+0000 7ffa7c5b3700 1 --2- 172.21.15.86:0/115644374 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffa740f6510 0x7ffa741067e0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7ffa700089f0 tx=0x7ffa7000b040).stop 2024-02-19T22:02:28.868 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:28.868+0000 7ffa7c5b3700 1 -- 172.21.15.86:0/115644374 shutdown_connections 2024-02-19T22:02:28.868 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:28.868+0000 7ffa7c5b3700 1 --2- 172.21.15.86:0/115644374 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7ffa6002ff60 0x7ffa60032420 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:02:28.868 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:28.868+0000 7ffa7c5b3700 1 --2- 172.21.15.86:0/115644374 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffa740f6510 0x7ffa741067e0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:02:28.868 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:28.868+0000 7ffa7c5b3700 1 -- 172.21.15.86:0/115644374 >> 172.21.15.86:0/115644374 conn(0x7ffa740f1c70 msgr2=0x7ffa740f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:02:28.868 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:28.868+0000 7ffa7c5b3700 1 -- 172.21.15.86:0/115644374 shutdown_connections 2024-02-19T22:02:28.869 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:28.868+0000 7ffa7c5b3700 1 -- 172.21.15.86:0/115644374 wait complete. 2024-02-19T22:02:28.869 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T22:02:29.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:02:28 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:02:27.490411+0000 mgr.smithi053.fyhcdi (mgr.14162) 182 : cluster [DBG] pgmap v170: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:02:30.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:02:29 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:02:28.867791+0000 mon.smithi053 (mon.0) 220 : audit [DBG] from='client.? 172.21.15.86:0/115644374' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:02:30.463 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:02:30.463 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:02:31.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:02:30 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:02:29.490660+0000 mgr.smithi053.fyhcdi (mgr.14162) 183 : cluster [DBG] pgmap v171: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:02:32.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:32.444+0000 7f3fdad93700 1 -- 172.21.15.86:0/588494528 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3fd40f6510 msgr2=0x7f3fd40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:02:32.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:32.444+0000 7f3fdad93700 1 --2- 172.21.15.86:0/588494528 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3fd40f6510 0x7f3fd40f6930 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 rx=0x7f3fc8009fb0 tx=0x7f3fc8009b30).stop 2024-02-19T22:02:32.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:32.444+0000 7f3fdad93700 1 -- 172.21.15.86:0/588494528 shutdown_connections 2024-02-19T22:02:32.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:32.444+0000 7f3fdad93700 1 --2- 172.21.15.86:0/588494528 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3fd40f6510 0x7f3fd40f6930 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:02:32.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:32.444+0000 7f3fdad93700 1 -- 172.21.15.86:0/588494528 >> 172.21.15.86:0/588494528 conn(0x7f3fd40f1c70 msgr2=0x7f3fd40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:02:32.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:32.444+0000 7f3fdad93700 1 -- 172.21.15.86:0/588494528 shutdown_connections 2024-02-19T22:02:32.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:32.444+0000 7f3fdad93700 1 -- 172.21.15.86:0/588494528 wait complete. 2024-02-19T22:02:32.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:32.445+0000 7f3fdad93700 1 Processor -- start 2024-02-19T22:02:32.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:32.445+0000 7f3fdad93700 1 -- start start 2024-02-19T22:02:32.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:32.445+0000 7f3fdad93700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3fd40f6510 0x7f3fd406b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:02:32.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:32.445+0000 7f3fdad93700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fd4069e20 con 0x7f3fd40f6510 2024-02-19T22:02:32.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:32.445+0000 7f3fd8b2f700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3fd40f6510 0x7f3fd406b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:02:32.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:32.445+0000 7f3fd8b2f700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3fd40f6510 0x7f3fd406b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:49236/0 (socket says 172.21.15.86:49236) 2024-02-19T22:02:32.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:32.445+0000 7f3fd8b2f700 1 -- 172.21.15.86:0/1993387448 learned_addr learned my addr 172.21.15.86:0/1993387448 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T22:02:32.447 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:32.446+0000 7f3fd8b2f700 1 -- 172.21.15.86:0/1993387448 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fc8005040 con 0x7f3fd40f6510 2024-02-19T22:02:32.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:32.446+0000 7f3fd8b2f700 1 --2- 172.21.15.86:0/1993387448 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3fd40f6510 0x7f3fd406b9a0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7f3fc8005010 tx=0x7f3fc800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:02:32.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:32.446+0000 7f3fd17fa700 1 -- 172.21.15.86:0/1993387448 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3fc8009320 con 0x7f3fd40f6510 2024-02-19T22:02:32.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:32.446+0000 7f3fdad93700 1 -- 172.21.15.86:0/1993387448 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fd4069fc0 con 0x7f3fd40f6510 2024-02-19T22:02:32.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:32.446+0000 7f3fd17fa700 1 -- 172.21.15.86:0/1993387448 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3fc8013380 con 0x7f3fd40f6510 2024-02-19T22:02:32.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:32.446+0000 7f3fd17fa700 1 -- 172.21.15.86:0/1993387448 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3fc8014e50 con 0x7f3fd40f6510 2024-02-19T22:02:32.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:32.446+0000 7f3fdad93700 1 -- 172.21.15.86:0/1993387448 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fd406a450 con 0x7f3fd40f6510 2024-02-19T22:02:32.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:32.447+0000 7f3fd17fa700 1 -- 172.21.15.86:0/1993387448 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3fc80095f0 con 0x7f3fd40f6510 2024-02-19T22:02:32.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:32.447+0000 7f3fdad93700 1 -- 172.21.15.86:0/1993387448 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fd4059100 con 0x7f3fd40f6510 2024-02-19T22:02:32.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:32.447+0000 7f3fd17fa700 1 --2- 172.21.15.86:0/1993387448 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f3fc402ff10 0x7f3fc40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:02:32.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:32.447+0000 7f3fd17fa700 1 -- 172.21.15.86:0/1993387448 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3fc8040e90 con 0x7f3fd40f6510 2024-02-19T22:02:32.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:32.448+0000 7f3fd3fff700 1 --2- 172.21.15.86:0/1993387448 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f3fc402ff10 0x7f3fc40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:02:32.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:32.448+0000 7f3fd3fff700 1 --2- 172.21.15.86:0/1993387448 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f3fc402ff10 0x7f3fc40323d0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f3fc00096a0 tx=0x7f3fc0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:02:32.451 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:32.452+0000 7f3fd17fa700 1 -- 172.21.15.86:0/1993387448 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3fc8014420 con 0x7f3fd40f6510 2024-02-19T22:02:32.656 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:32.657+0000 7f3fdad93700 1 -- 172.21.15.86:0/1993387448 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3fd40fc0a0 con 0x7f3fd40f6510 2024-02-19T22:02:32.657 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:32.657+0000 7f3fd17fa700 1 -- 172.21.15.86:0/1993387448 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3fc8010230 con 0x7f3fd40f6510 2024-02-19T22:02:32.657 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T22:02:32.657 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:02:32.658 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:32.659+0000 7f3fdad93700 1 -- 172.21.15.86:0/1993387448 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f3fc402ff10 msgr2=0x7f3fc40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:02:32.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:32.659+0000 7f3fdad93700 1 --2- 172.21.15.86:0/1993387448 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f3fc402ff10 0x7f3fc40323d0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f3fc00096a0 tx=0x7f3fc0006b40).stop 2024-02-19T22:02:32.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:32.659+0000 7f3fdad93700 1 -- 172.21.15.86:0/1993387448 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3fd40f6510 msgr2=0x7f3fd406b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:02:32.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:32.659+0000 7f3fdad93700 1 --2- 172.21.15.86:0/1993387448 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3fd40f6510 0x7f3fd406b9a0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7f3fc8005010 tx=0x7f3fc800b040).stop 2024-02-19T22:02:32.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:32.659+0000 7f3fdad93700 1 -- 172.21.15.86:0/1993387448 shutdown_connections 2024-02-19T22:02:32.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:32.659+0000 7f3fdad93700 1 --2- 172.21.15.86:0/1993387448 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f3fc402ff10 0x7f3fc40323d0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:02:32.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:32.659+0000 7f3fdad93700 1 --2- 172.21.15.86:0/1993387448 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3fd40f6510 0x7f3fd406b9a0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:02:32.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:32.659+0000 7f3fdad93700 1 -- 172.21.15.86:0/1993387448 >> 172.21.15.86:0/1993387448 conn(0x7f3fd40f1c70 msgr2=0x7f3fd40f26f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:02:32.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:32.659+0000 7f3fdad93700 1 -- 172.21.15.86:0/1993387448 shutdown_connections 2024-02-19T22:02:32.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:32.659+0000 7f3fdad93700 1 -- 172.21.15.86:0/1993387448 wait complete. 2024-02-19T22:02:32.660 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T22:02:33.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:02:32 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:02:31.490934+0000 mgr.smithi053.fyhcdi (mgr.14162) 184 : cluster [DBG] pgmap v172: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:02:34.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:02:33 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:02:32.658623+0000 mon.smithi053 (mon.0) 221 : audit [DBG] from='client.? 172.21.15.86:0/1993387448' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:02:34.352 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:02:34.353 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:02:35.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:02:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:02:33.491254+0000 mgr.smithi053.fyhcdi (mgr.14162) 185 : cluster [DBG] pgmap v173: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:02:36.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:36.218+0000 7fd937c15700 1 -- 172.21.15.86:0/3692615693 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd9300f53f0 msgr2=0x7fd9300f5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:02:36.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:36.218+0000 7fd937c15700 1 --2- 172.21.15.86:0/3692615693 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd9300f53f0 0x7fd9300f5810 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 rx=0x7fd920009fb0 tx=0x7fd920009b30).stop 2024-02-19T22:02:36.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:36.218+0000 7fd937c15700 1 -- 172.21.15.86:0/3692615693 shutdown_connections 2024-02-19T22:02:36.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:36.218+0000 7fd937c15700 1 --2- 172.21.15.86:0/3692615693 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd9300f53f0 0x7fd9300f5810 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:02:36.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:36.218+0000 7fd937c15700 1 -- 172.21.15.86:0/3692615693 >> 172.21.15.86:0/3692615693 conn(0x7fd9300f0b70 msgr2=0x7fd9300f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:02:36.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:36.218+0000 7fd937c15700 1 -- 172.21.15.86:0/3692615693 shutdown_connections 2024-02-19T22:02:36.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:36.219+0000 7fd937c15700 1 -- 172.21.15.86:0/3692615693 wait complete. 2024-02-19T22:02:36.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:36.219+0000 7fd937c15700 1 Processor -- start 2024-02-19T22:02:36.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:36.219+0000 7fd937c15700 1 -- start start 2024-02-19T22:02:36.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:36.219+0000 7fd937c15700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd9300f53f0 0x7fd9301056c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:02:36.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:36.219+0000 7fd937c15700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd930105bb0 con 0x7fd9300f53f0 2024-02-19T22:02:36.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:36.219+0000 7fd9359b1700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd9300f53f0 0x7fd9301056c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:02:36.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:36.219+0000 7fd9359b1700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd9300f53f0 0x7fd9301056c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:49250/0 (socket says 172.21.15.86:49250) 2024-02-19T22:02:36.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:36.219+0000 7fd9359b1700 1 -- 172.21.15.86:0/4167171106 learned_addr learned my addr 172.21.15.86:0/4167171106 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T22:02:36.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:36.220+0000 7fd9359b1700 1 -- 172.21.15.86:0/4167171106 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd920005040 con 0x7fd9300f53f0 2024-02-19T22:02:36.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:36.220+0000 7fd9359b1700 1 --2- 172.21.15.86:0/4167171106 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd9300f53f0 0x7fd9301056c0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7fd920005010 tx=0x7fd92000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:02:36.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:36.220+0000 7fd9267fc700 1 -- 172.21.15.86:0/4167171106 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd920009320 con 0x7fd9300f53f0 2024-02-19T22:02:36.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:36.220+0000 7fd937c15700 1 -- 172.21.15.86:0/4167171106 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd930105db0 con 0x7fd9300f53f0 2024-02-19T22:02:36.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:36.220+0000 7fd9267fc700 1 -- 172.21.15.86:0/4167171106 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd920013380 con 0x7fd9300f53f0 2024-02-19T22:02:36.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:36.220+0000 7fd9267fc700 1 -- 172.21.15.86:0/4167171106 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd920014e50 con 0x7fd9300f53f0 2024-02-19T22:02:36.222 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:36.220+0000 7fd937c15700 1 -- 172.21.15.86:0/4167171106 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd930106240 con 0x7fd9300f53f0 2024-02-19T22:02:36.222 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:36.221+0000 7fd9267fc700 1 -- 172.21.15.86:0/4167171106 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd9200095f0 con 0x7fd9300f53f0 2024-02-19T22:02:36.222 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:36.221+0000 7fd937c15700 1 -- 172.21.15.86:0/4167171106 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd930045ad0 con 0x7fd9300f53f0 2024-02-19T22:02:36.222 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:36.221+0000 7fd9267fc700 1 --2- 172.21.15.86:0/4167171106 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fd91c02ff10 0x7fd91c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:02:36.222 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:36.221+0000 7fd9267fc700 1 -- 172.21.15.86:0/4167171106 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd920040fa0 con 0x7fd9300f53f0 2024-02-19T22:02:36.222 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:36.221+0000 7fd9351b0700 1 --2- 172.21.15.86:0/4167171106 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fd91c02ff10 0x7fd91c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:02:36.222 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:36.222+0000 7fd9351b0700 1 --2- 172.21.15.86:0/4167171106 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fd91c02ff10 0x7fd91c0323d0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7fd92c0096a0 tx=0x7fd92c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:02:36.225 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:36.226+0000 7fd9267fc700 1 -- 172.21.15.86:0/4167171106 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd920014420 con 0x7fd9300f53f0 2024-02-19T22:02:36.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:36.433+0000 7fd937c15700 1 -- 172.21.15.86:0/4167171106 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd9301070f0 con 0x7fd9300f53f0 2024-02-19T22:02:36.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:36.433+0000 7fd9267fc700 1 -- 172.21.15.86:0/4167171106 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd920010230 con 0x7fd9300f53f0 2024-02-19T22:02:36.433 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T22:02:36.433 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:02:36.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:36.435+0000 7fd937c15700 1 -- 172.21.15.86:0/4167171106 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fd91c02ff10 msgr2=0x7fd91c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:02:36.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:36.435+0000 7fd937c15700 1 --2- 172.21.15.86:0/4167171106 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fd91c02ff10 0x7fd91c0323d0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7fd92c0096a0 tx=0x7fd92c006b40).stop 2024-02-19T22:02:36.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:36.435+0000 7fd937c15700 1 -- 172.21.15.86:0/4167171106 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd9300f53f0 msgr2=0x7fd9301056c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:02:36.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:36.435+0000 7fd937c15700 1 --2- 172.21.15.86:0/4167171106 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd9300f53f0 0x7fd9301056c0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7fd920005010 tx=0x7fd92000b040).stop 2024-02-19T22:02:36.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:36.435+0000 7fd937c15700 1 -- 172.21.15.86:0/4167171106 shutdown_connections 2024-02-19T22:02:36.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:36.435+0000 7fd937c15700 1 --2- 172.21.15.86:0/4167171106 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fd91c02ff10 0x7fd91c0323d0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:02:36.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:36.435+0000 7fd937c15700 1 --2- 172.21.15.86:0/4167171106 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd9300f53f0 0x7fd9301056c0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:02:36.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:36.435+0000 7fd937c15700 1 -- 172.21.15.86:0/4167171106 >> 172.21.15.86:0/4167171106 conn(0x7fd9300f0b70 msgr2=0x7fd9300f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:02:36.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:36.435+0000 7fd937c15700 1 -- 172.21.15.86:0/4167171106 shutdown_connections 2024-02-19T22:02:36.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:36.435+0000 7fd937c15700 1 -- 172.21.15.86:0/4167171106 wait complete. 2024-02-19T22:02:36.435 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T22:02:37.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:02:36 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:02:35.491589+0000 mgr.smithi053.fyhcdi (mgr.14162) 186 : cluster [DBG] pgmap v174: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:02:37.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:02:36 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:02:35.602168+0000 mon.smithi053 (mon.0) 222 : audit [INF] from='mgr.14162 172.21.15.53:0/2860358250' entity='mgr.smithi053.fyhcdi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi053.fyhcdi/mirror_snapshot_schedule"}]: dispatch 2024-02-19T22:02:37.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:02:36 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:02:35.605359+0000 mon.smithi053 (mon.0) 223 : audit [INF] from='mgr.14162 172.21.15.53:0/2860358250' entity='mgr.smithi053.fyhcdi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi053.fyhcdi/trash_purge_schedule"}]: dispatch 2024-02-19T22:02:37.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:02:36 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:02:36.434693+0000 mon.smithi053 (mon.0) 224 : audit [DBG] from='client.? 172.21.15.86:0/4167171106' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:02:38.135 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:02:38.136 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:02:39.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:02:38 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:02:37.491834+0000 mgr.smithi053.fyhcdi (mgr.14162) 187 : cluster [DBG] pgmap v175: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:02:39.962 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:39.961+0000 7efd86998700 1 -- 172.21.15.86:0/125292441 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efd800f6a80 msgr2=0x7efd800f6ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:02:39.962 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:39.961+0000 7efd86998700 1 --2- 172.21.15.86:0/125292441 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efd800f6a80 0x7efd800f6ea0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 rx=0x7efd74009fb0 tx=0x7efd74009b30).stop 2024-02-19T22:02:39.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:39.962+0000 7efd86998700 1 -- 172.21.15.86:0/125292441 shutdown_connections 2024-02-19T22:02:39.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:39.962+0000 7efd86998700 1 --2- 172.21.15.86:0/125292441 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efd800f6a80 0x7efd800f6ea0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:02:39.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:39.962+0000 7efd86998700 1 -- 172.21.15.86:0/125292441 >> 172.21.15.86:0/125292441 conn(0x7efd800f02e0 msgr2=0x7efd800f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:02:39.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:39.962+0000 7efd86998700 1 -- 172.21.15.86:0/125292441 shutdown_connections 2024-02-19T22:02:39.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:39.962+0000 7efd86998700 1 -- 172.21.15.86:0/125292441 wait complete. 2024-02-19T22:02:39.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:39.962+0000 7efd86998700 1 Processor -- start 2024-02-19T22:02:39.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:39.962+0000 7efd86998700 1 -- start start 2024-02-19T22:02:39.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:39.962+0000 7efd86998700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efd800f6a80 0x7efd80106780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:02:39.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:39.962+0000 7efd86998700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd80106c70 con 0x7efd800f6a80 2024-02-19T22:02:39.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:39.963+0000 7efd7ffff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efd800f6a80 0x7efd80106780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:02:39.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:39.963+0000 7efd7ffff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efd800f6a80 0x7efd80106780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:41204/0 (socket says 172.21.15.86:41204) 2024-02-19T22:02:39.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:39.963+0000 7efd7ffff700 1 -- 172.21.15.86:0/3590448092 learned_addr learned my addr 172.21.15.86:0/3590448092 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T22:02:39.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:39.963+0000 7efd7ffff700 1 -- 172.21.15.86:0/3590448092 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd74005040 con 0x7efd800f6a80 2024-02-19T22:02:39.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:39.963+0000 7efd7ffff700 1 --2- 172.21.15.86:0/3590448092 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efd800f6a80 0x7efd80106780 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 rx=0x7efd740089f0 tx=0x7efd7400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:02:39.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:39.963+0000 7efd7cff9700 1 -- 172.21.15.86:0/3590448092 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efd74009320 con 0x7efd800f6a80 2024-02-19T22:02:39.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:39.963+0000 7efd86998700 1 -- 172.21.15.86:0/3590448092 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd80106e70 con 0x7efd800f6a80 2024-02-19T22:02:39.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:39.963+0000 7efd7cff9700 1 -- 172.21.15.86:0/3590448092 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efd74013380 con 0x7efd800f6a80 2024-02-19T22:02:39.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:39.963+0000 7efd86998700 1 -- 172.21.15.86:0/3590448092 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd80107300 con 0x7efd800f6a80 2024-02-19T22:02:39.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:39.963+0000 7efd7cff9700 1 -- 172.21.15.86:0/3590448092 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efd74014e20 con 0x7efd800f6a80 2024-02-19T22:02:39.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:39.964+0000 7efd7cff9700 1 -- 172.21.15.86:0/3590448092 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7efd740095f0 con 0x7efd800f6a80 2024-02-19T22:02:39.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:39.964+0000 7efd86998700 1 -- 172.21.15.86:0/3590448092 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd80045ad0 con 0x7efd800f6a80 2024-02-19T22:02:39.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:39.964+0000 7efd7cff9700 1 --2- 172.21.15.86:0/3590448092 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7efd6802ff10 0x7efd680323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:02:39.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:39.964+0000 7efd7cff9700 1 -- 172.21.15.86:0/3590448092 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7efd74040ea0 con 0x7efd800f6a80 2024-02-19T22:02:39.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:39.965+0000 7efd7f7fe700 1 --2- 172.21.15.86:0/3590448092 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7efd6802ff10 0x7efd680323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:02:39.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:39.965+0000 7efd7f7fe700 1 --2- 172.21.15.86:0/3590448092 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7efd6802ff10 0x7efd680323d0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7efd700096a0 tx=0x7efd70006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:02:39.968 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:39.969+0000 7efd7cff9700 1 -- 172.21.15.86:0/3590448092 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7efd74014420 con 0x7efd800f6a80 2024-02-19T22:02:40.174 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:40.174+0000 7efd86998700 1 -- 172.21.15.86:0/3590448092 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7efd80108150 con 0x7efd800f6a80 2024-02-19T22:02:40.174 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:40.175+0000 7efd7cff9700 1 -- 172.21.15.86:0/3590448092 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7efd74010230 con 0x7efd800f6a80 2024-02-19T22:02:40.174 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T22:02:40.174 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:02:40.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:40.176+0000 7efd86998700 1 -- 172.21.15.86:0/3590448092 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7efd6802ff10 msgr2=0x7efd680323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:02:40.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:40.176+0000 7efd86998700 1 --2- 172.21.15.86:0/3590448092 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7efd6802ff10 0x7efd680323d0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7efd700096a0 tx=0x7efd70006b40).stop 2024-02-19T22:02:40.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:40.176+0000 7efd86998700 1 -- 172.21.15.86:0/3590448092 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efd800f6a80 msgr2=0x7efd80106780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:02:40.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:40.176+0000 7efd86998700 1 --2- 172.21.15.86:0/3590448092 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efd800f6a80 0x7efd80106780 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 rx=0x7efd740089f0 tx=0x7efd7400b040).stop 2024-02-19T22:02:40.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:40.176+0000 7efd86998700 1 -- 172.21.15.86:0/3590448092 shutdown_connections 2024-02-19T22:02:40.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:40.176+0000 7efd86998700 1 --2- 172.21.15.86:0/3590448092 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7efd6802ff10 0x7efd680323d0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:02:40.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:40.176+0000 7efd86998700 1 --2- 172.21.15.86:0/3590448092 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efd800f6a80 0x7efd80106780 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:02:40.177 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:40.176+0000 7efd86998700 1 -- 172.21.15.86:0/3590448092 >> 172.21.15.86:0/3590448092 conn(0x7efd800f02e0 msgr2=0x7efd800f4ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:02:40.177 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:40.176+0000 7efd86998700 1 -- 172.21.15.86:0/3590448092 shutdown_connections 2024-02-19T22:02:40.177 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:40.176+0000 7efd86998700 1 -- 172.21.15.86:0/3590448092 wait complete. 2024-02-19T22:02:40.177 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T22:02:40.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:02:40 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:02:39.492063+0000 mgr.smithi053.fyhcdi (mgr.14162) 188 : cluster [DBG] pgmap v176: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:02:40.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:02:40 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:02:40.175887+0000 mon.smithi053 (mon.0) 225 : audit [DBG] from='client.? 172.21.15.86:0/3590448092' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:02:41.931 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:02:41.932 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:02:42.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:02:42 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:02:41.492409+0000 mgr.smithi053.fyhcdi (mgr.14162) 189 : cluster [DBG] pgmap v177: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:02:43.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:43.875+0000 7fac329c7700 1 -- 172.21.15.86:0/3926197671 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fac2c0f5410 msgr2=0x7fac2c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:02:43.876 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:43.875+0000 7fac329c7700 1 --2- 172.21.15.86:0/3926197671 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fac2c0f5410 0x7fac2c0f5830 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 rx=0x7fac1c009fb0 tx=0x7fac1c009b30).stop 2024-02-19T22:02:43.876 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:43.875+0000 7fac329c7700 1 -- 172.21.15.86:0/3926197671 shutdown_connections 2024-02-19T22:02:43.876 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:43.875+0000 7fac329c7700 1 --2- 172.21.15.86:0/3926197671 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fac2c0f5410 0x7fac2c0f5830 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:02:43.876 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:43.875+0000 7fac329c7700 1 -- 172.21.15.86:0/3926197671 >> 172.21.15.86:0/3926197671 conn(0x7fac2c0f0b70 msgr2=0x7fac2c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:02:43.876 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:43.875+0000 7fac329c7700 1 -- 172.21.15.86:0/3926197671 shutdown_connections 2024-02-19T22:02:43.876 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:43.875+0000 7fac329c7700 1 -- 172.21.15.86:0/3926197671 wait complete. 2024-02-19T22:02:43.876 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:43.875+0000 7fac329c7700 1 Processor -- start 2024-02-19T22:02:43.876 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:43.875+0000 7fac329c7700 1 -- start start 2024-02-19T22:02:43.876 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:43.876+0000 7fac329c7700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fac2c0f5410 0x7fac2c106810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:02:43.876 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:43.876+0000 7fac329c7700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac2c106d00 con 0x7fac2c0f5410 2024-02-19T22:02:43.877 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:43.876+0000 7fac2bfff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fac2c0f5410 0x7fac2c106810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:02:43.877 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:43.876+0000 7fac2bfff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fac2c0f5410 0x7fac2c106810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:41220/0 (socket says 172.21.15.86:41220) 2024-02-19T22:02:43.877 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:43.876+0000 7fac2bfff700 1 -- 172.21.15.86:0/231195749 learned_addr learned my addr 172.21.15.86:0/231195749 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T22:02:43.877 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:43.876+0000 7fac2bfff700 1 -- 172.21.15.86:0/231195749 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac1c005040 con 0x7fac2c0f5410 2024-02-19T22:02:43.878 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:43.876+0000 7fac2bfff700 1 --2- 172.21.15.86:0/231195749 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fac2c0f5410 0x7fac2c106810 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7fac1c005010 tx=0x7fac1c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:02:43.878 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:43.876+0000 7fac28ff9700 1 -- 172.21.15.86:0/231195749 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fac1c009320 con 0x7fac2c0f5410 2024-02-19T22:02:43.878 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:43.876+0000 7fac28ff9700 1 -- 172.21.15.86:0/231195749 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fac1c013380 con 0x7fac2c0f5410 2024-02-19T22:02:43.879 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:43.876+0000 7fac329c7700 1 -- 172.21.15.86:0/231195749 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac2c106f00 con 0x7fac2c0f5410 2024-02-19T22:02:43.879 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:43.876+0000 7fac28ff9700 1 -- 172.21.15.86:0/231195749 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fac1c014e50 con 0x7fac2c0f5410 2024-02-19T22:02:43.879 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:43.877+0000 7fac329c7700 1 -- 172.21.15.86:0/231195749 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac2c107390 con 0x7fac2c0f5410 2024-02-19T22:02:43.879 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:43.877+0000 7fac28ff9700 1 -- 172.21.15.86:0/231195749 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fac1c0095f0 con 0x7fac2c0f5410 2024-02-19T22:02:43.879 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:43.877+0000 7fac329c7700 1 -- 172.21.15.86:0/231195749 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac2c045ad0 con 0x7fac2c0f5410 2024-02-19T22:02:43.879 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:43.877+0000 7fac28ff9700 1 --2- 172.21.15.86:0/231195749 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fac1402ff10 0x7fac140323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:02:43.879 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:43.877+0000 7fac28ff9700 1 -- 172.21.15.86:0/231195749 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fac1c040e90 con 0x7fac2c0f5410 2024-02-19T22:02:43.879 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:43.878+0000 7fac2b7fe700 1 --2- 172.21.15.86:0/231195749 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fac1402ff10 0x7fac140323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:02:43.879 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:43.878+0000 7fac2b7fe700 1 --2- 172.21.15.86:0/231195749 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fac1402ff10 0x7fac140323d0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7fac200096a0 tx=0x7fac20006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:02:43.881 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:43.882+0000 7fac28ff9700 1 -- 172.21.15.86:0/231195749 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fac1c014420 con 0x7fac2c0f5410 2024-02-19T22:02:44.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:44.086+0000 7fac329c7700 1 -- 172.21.15.86:0/231195749 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fac2c108240 con 0x7fac2c0f5410 2024-02-19T22:02:44.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:44.086+0000 7fac28ff9700 1 -- 172.21.15.86:0/231195749 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fac1c010230 con 0x7fac2c0f5410 2024-02-19T22:02:44.086 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T22:02:44.086 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:02:44.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:44.088+0000 7fac329c7700 1 -- 172.21.15.86:0/231195749 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fac1402ff10 msgr2=0x7fac140323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:02:44.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:44.088+0000 7fac329c7700 1 --2- 172.21.15.86:0/231195749 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fac1402ff10 0x7fac140323d0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7fac200096a0 tx=0x7fac20006b40).stop 2024-02-19T22:02:44.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:44.088+0000 7fac329c7700 1 -- 172.21.15.86:0/231195749 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fac2c0f5410 msgr2=0x7fac2c106810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:02:44.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:44.088+0000 7fac329c7700 1 --2- 172.21.15.86:0/231195749 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fac2c0f5410 0x7fac2c106810 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7fac1c005010 tx=0x7fac1c00b040).stop 2024-02-19T22:02:44.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:44.088+0000 7fac329c7700 1 -- 172.21.15.86:0/231195749 shutdown_connections 2024-02-19T22:02:44.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:44.088+0000 7fac329c7700 1 --2- 172.21.15.86:0/231195749 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fac1402ff10 0x7fac140323d0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:02:44.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:44.088+0000 7fac329c7700 1 --2- 172.21.15.86:0/231195749 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fac2c0f5410 0x7fac2c106810 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:02:44.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:44.088+0000 7fac329c7700 1 -- 172.21.15.86:0/231195749 >> 172.21.15.86:0/231195749 conn(0x7fac2c0f0b70 msgr2=0x7fac2c05fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:02:44.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:44.088+0000 7fac329c7700 1 -- 172.21.15.86:0/231195749 shutdown_connections 2024-02-19T22:02:44.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:44.088+0000 7fac329c7700 1 -- 172.21.15.86:0/231195749 wait complete. 2024-02-19T22:02:44.089 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T22:02:45.020 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:02:44 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:02:43.492663+0000 mgr.smithi053.fyhcdi (mgr.14162) 190 : cluster [DBG] pgmap v178: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:02:45.020 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:02:44 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:02:44.087797+0000 mon.smithi053 (mon.0) 226 : audit [DBG] from='client.? 172.21.15.86:0/231195749' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:02:45.694 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:02:45.694 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:02:47.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:02:46 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:02:45.492945+0000 mgr.smithi053.fyhcdi (mgr.14162) 191 : cluster [DBG] pgmap v179: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:02:47.626 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:47.625+0000 7f51cc451700 1 -- 172.21.15.86:0/2088702869 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f51c40f6510 msgr2=0x7f51c40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:02:47.626 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:47.625+0000 7f51cc451700 1 --2- 172.21.15.86:0/2088702869 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f51c40f6510 0x7f51c40f6930 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 rx=0x7f51c0009fb0 tx=0x7f51c0009b30).stop 2024-02-19T22:02:47.626 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:47.625+0000 7f51cc451700 1 -- 172.21.15.86:0/2088702869 shutdown_connections 2024-02-19T22:02:47.626 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:47.625+0000 7f51cc451700 1 --2- 172.21.15.86:0/2088702869 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f51c40f6510 0x7f51c40f6930 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:02:47.626 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:47.625+0000 7f51cc451700 1 -- 172.21.15.86:0/2088702869 >> 172.21.15.86:0/2088702869 conn(0x7f51c40f1c70 msgr2=0x7f51c40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:02:47.626 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:47.625+0000 7f51cc451700 1 -- 172.21.15.86:0/2088702869 shutdown_connections 2024-02-19T22:02:47.626 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:47.625+0000 7f51cc451700 1 -- 172.21.15.86:0/2088702869 wait complete. 2024-02-19T22:02:47.626 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:47.626+0000 7f51cc451700 1 Processor -- start 2024-02-19T22:02:47.626 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:47.626+0000 7f51cc451700 1 -- start start 2024-02-19T22:02:47.626 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:47.626+0000 7f51cc451700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f51c40f6510 0x7f51c406b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:02:47.626 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:47.626+0000 7f51cc451700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51c4069e20 con 0x7f51c40f6510 2024-02-19T22:02:47.627 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:47.626+0000 7f51ca1ed700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f51c40f6510 0x7f51c406b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:02:47.627 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:47.626+0000 7f51ca1ed700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f51c40f6510 0x7f51c406b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:41248/0 (socket says 172.21.15.86:41248) 2024-02-19T22:02:47.627 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:47.626+0000 7f51ca1ed700 1 -- 172.21.15.86:0/2123381871 learned_addr learned my addr 172.21.15.86:0/2123381871 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T22:02:47.628 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:47.626+0000 7f51ca1ed700 1 -- 172.21.15.86:0/2123381871 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51c0005040 con 0x7f51c40f6510 2024-02-19T22:02:47.628 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:47.627+0000 7f51ca1ed700 1 --2- 172.21.15.86:0/2123381871 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f51c40f6510 0x7f51c406b9a0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 rx=0x7f51c00089f0 tx=0x7f51c000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:02:47.628 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:47.627+0000 7f51baffd700 1 -- 172.21.15.86:0/2123381871 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f51c0009320 con 0x7f51c40f6510 2024-02-19T22:02:47.629 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:47.627+0000 7f51cc451700 1 -- 172.21.15.86:0/2123381871 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51c4069fc0 con 0x7f51c40f6510 2024-02-19T22:02:47.629 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:47.627+0000 7f51baffd700 1 -- 172.21.15.86:0/2123381871 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f51c0013380 con 0x7f51c40f6510 2024-02-19T22:02:47.629 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:47.627+0000 7f51baffd700 1 -- 172.21.15.86:0/2123381871 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f51c001ed80 con 0x7f51c40f6510 2024-02-19T22:02:47.629 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:47.627+0000 7f51cc451700 1 -- 172.21.15.86:0/2123381871 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51c406a450 con 0x7f51c40f6510 2024-02-19T22:02:47.629 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:47.628+0000 7f51baffd700 1 -- 172.21.15.86:0/2123381871 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f51c001e420 con 0x7f51c40f6510 2024-02-19T22:02:47.629 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:47.628+0000 7f51cc451700 1 -- 172.21.15.86:0/2123381871 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51c4059100 con 0x7f51c40f6510 2024-02-19T22:02:47.629 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:47.628+0000 7f51baffd700 1 --2- 172.21.15.86:0/2123381871 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f51b002ff60 0x7f51b0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:02:47.629 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:47.628+0000 7f51baffd700 1 -- 172.21.15.86:0/2123381871 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f51c0041120 con 0x7f51c40f6510 2024-02-19T22:02:47.629 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:47.628+0000 7f51c99ec700 1 --2- 172.21.15.86:0/2123381871 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f51b002ff60 0x7f51b0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:02:47.629 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:47.628+0000 7f51c99ec700 1 --2- 172.21.15.86:0/2123381871 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f51b002ff60 0x7f51b0032420 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f51b40096a0 tx=0x7f51b4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:02:47.632 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:47.633+0000 7f51baffd700 1 -- 172.21.15.86:0/2123381871 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f51c001cb10 con 0x7f51c40f6510 2024-02-19T22:02:47.838 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:47.839+0000 7f51cc451700 1 -- 172.21.15.86:0/2123381871 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f51c40fc0a0 con 0x7f51c40f6510 2024-02-19T22:02:47.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:47.839+0000 7f51baffd700 1 -- 172.21.15.86:0/2123381871 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f51c002c2d0 con 0x7f51c40f6510 2024-02-19T22:02:47.839 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T22:02:47.839 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:02:47.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:47.841+0000 7f51cc451700 1 -- 172.21.15.86:0/2123381871 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f51b002ff60 msgr2=0x7f51b0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:02:47.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:47.841+0000 7f51cc451700 1 --2- 172.21.15.86:0/2123381871 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f51b002ff60 0x7f51b0032420 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f51b40096a0 tx=0x7f51b4006b40).stop 2024-02-19T22:02:47.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:47.841+0000 7f51cc451700 1 -- 172.21.15.86:0/2123381871 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f51c40f6510 msgr2=0x7f51c406b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:02:47.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:47.841+0000 7f51cc451700 1 --2- 172.21.15.86:0/2123381871 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f51c40f6510 0x7f51c406b9a0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 rx=0x7f51c00089f0 tx=0x7f51c000b040).stop 2024-02-19T22:02:47.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:47.841+0000 7f51cc451700 1 -- 172.21.15.86:0/2123381871 shutdown_connections 2024-02-19T22:02:47.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:47.841+0000 7f51cc451700 1 --2- 172.21.15.86:0/2123381871 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f51b002ff60 0x7f51b0032420 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:02:47.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:47.841+0000 7f51cc451700 1 --2- 172.21.15.86:0/2123381871 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f51c40f6510 0x7f51c406b9a0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:02:47.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:47.841+0000 7f51cc451700 1 -- 172.21.15.86:0/2123381871 >> 172.21.15.86:0/2123381871 conn(0x7f51c40f1c70 msgr2=0x7f51c40f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:02:47.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:47.841+0000 7f51cc451700 1 -- 172.21.15.86:0/2123381871 shutdown_connections 2024-02-19T22:02:47.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:47.841+0000 7f51cc451700 1 -- 172.21.15.86:0/2123381871 wait complete. 2024-02-19T22:02:47.842 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T22:02:49.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:02:48 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:02:47.493296+0000 mgr.smithi053.fyhcdi (mgr.14162) 192 : cluster [DBG] pgmap v180: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:02:49.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:02:48 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:02:47.840586+0000 mon.smithi053 (mon.0) 227 : audit [DBG] from='client.? 172.21.15.86:0/2123381871' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:02:49.481 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:02:49.481 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:02:50.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:02:50 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:02:49.493576+0000 mgr.smithi053.fyhcdi (mgr.14162) 193 : cluster [DBG] pgmap v181: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:02:51.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:51.530+0000 7f1ee07aa700 1 -- 172.21.15.86:0/2113677216 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1ed80f6510 msgr2=0x7f1ed80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:02:51.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:51.530+0000 7f1ee07aa700 1 --2- 172.21.15.86:0/2113677216 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1ed80f6510 0x7f1ed80f6930 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 rx=0x7f1ec8009fb0 tx=0x7f1ec8009b30).stop 2024-02-19T22:02:51.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:51.530+0000 7f1ee07aa700 1 -- 172.21.15.86:0/2113677216 shutdown_connections 2024-02-19T22:02:51.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:51.530+0000 7f1ee07aa700 1 --2- 172.21.15.86:0/2113677216 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1ed80f6510 0x7f1ed80f6930 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:02:51.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:51.530+0000 7f1ee07aa700 1 -- 172.21.15.86:0/2113677216 >> 172.21.15.86:0/2113677216 conn(0x7f1ed80f1c70 msgr2=0x7f1ed80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:02:51.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:51.530+0000 7f1ee07aa700 1 -- 172.21.15.86:0/2113677216 shutdown_connections 2024-02-19T22:02:51.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:51.530+0000 7f1ee07aa700 1 -- 172.21.15.86:0/2113677216 wait complete. 2024-02-19T22:02:51.532 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:51.531+0000 7f1ee07aa700 1 Processor -- start 2024-02-19T22:02:51.532 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:51.531+0000 7f1ee07aa700 1 -- start start 2024-02-19T22:02:51.532 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:51.531+0000 7f1ee07aa700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1ed80f6510 0x7f1ed81067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:02:51.532 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:51.531+0000 7f1ee07aa700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ed8106cd0 con 0x7f1ed80f6510 2024-02-19T22:02:51.532 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:51.531+0000 7f1ede546700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1ed80f6510 0x7f1ed81067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:02:51.532 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:51.531+0000 7f1ede546700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1ed80f6510 0x7f1ed81067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:55084/0 (socket says 172.21.15.86:55084) 2024-02-19T22:02:51.532 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:51.531+0000 7f1ede546700 1 -- 172.21.15.86:0/2793382728 learned_addr learned my addr 172.21.15.86:0/2793382728 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T22:02:51.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:51.532+0000 7f1ede546700 1 -- 172.21.15.86:0/2793382728 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ec8005040 con 0x7f1ed80f6510 2024-02-19T22:02:51.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:51.532+0000 7f1ede546700 1 --2- 172.21.15.86:0/2793382728 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1ed80f6510 0x7f1ed81067e0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7f1ec8005010 tx=0x7f1ec800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:02:51.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:51.532+0000 7f1eceffd700 1 -- 172.21.15.86:0/2793382728 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1ec8009320 con 0x7f1ed80f6510 2024-02-19T22:02:51.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:51.532+0000 7f1ee07aa700 1 -- 172.21.15.86:0/2793382728 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ed8106ed0 con 0x7f1ed80f6510 2024-02-19T22:02:51.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:51.532+0000 7f1eceffd700 1 -- 172.21.15.86:0/2793382728 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1ec8013380 con 0x7f1ed80f6510 2024-02-19T22:02:51.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:51.532+0000 7f1eceffd700 1 -- 172.21.15.86:0/2793382728 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1ec8014e50 con 0x7f1ed80f6510 2024-02-19T22:02:51.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:51.532+0000 7f1ee07aa700 1 -- 172.21.15.86:0/2793382728 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ed8107360 con 0x7f1ed80f6510 2024-02-19T22:02:51.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:51.533+0000 7f1eceffd700 1 -- 172.21.15.86:0/2793382728 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1ec80095f0 con 0x7f1ed80f6510 2024-02-19T22:02:51.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:51.533+0000 7f1ee07aa700 1 -- 172.21.15.86:0/2793382728 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ed8045ad0 con 0x7f1ed80f6510 2024-02-19T22:02:51.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:51.533+0000 7f1eceffd700 1 --2- 172.21.15.86:0/2793382728 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f1ec402ff10 0x7f1ec40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:02:51.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:51.533+0000 7f1eceffd700 1 -- 172.21.15.86:0/2793382728 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1ec8040e90 con 0x7f1ed80f6510 2024-02-19T22:02:51.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:51.533+0000 7f1eddd45700 1 --2- 172.21.15.86:0/2793382728 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f1ec402ff10 0x7f1ec40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:02:51.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:51.534+0000 7f1eddd45700 1 --2- 172.21.15.86:0/2793382728 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f1ec402ff10 0x7f1ec40323d0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f1ed40096a0 tx=0x7f1ed4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:02:51.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:51.538+0000 7f1eceffd700 1 -- 172.21.15.86:0/2793382728 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1ec8014420 con 0x7f1ed80f6510 2024-02-19T22:02:51.747 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:51.748+0000 7f1ee07aa700 1 -- 172.21.15.86:0/2793382728 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1ed8108210 con 0x7f1ed80f6510 2024-02-19T22:02:51.748 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:51.748+0000 7f1eceffd700 1 -- 172.21.15.86:0/2793382728 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1ec8010230 con 0x7f1ed80f6510 2024-02-19T22:02:51.748 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T22:02:51.749 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:02:51.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:51.750+0000 7f1ee07aa700 1 -- 172.21.15.86:0/2793382728 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f1ec402ff10 msgr2=0x7f1ec40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:02:51.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:51.750+0000 7f1ee07aa700 1 --2- 172.21.15.86:0/2793382728 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f1ec402ff10 0x7f1ec40323d0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f1ed40096a0 tx=0x7f1ed4006b40).stop 2024-02-19T22:02:51.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:51.750+0000 7f1ee07aa700 1 -- 172.21.15.86:0/2793382728 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1ed80f6510 msgr2=0x7f1ed81067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:02:51.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:51.750+0000 7f1ee07aa700 1 --2- 172.21.15.86:0/2793382728 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1ed80f6510 0x7f1ed81067e0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7f1ec8005010 tx=0x7f1ec800b040).stop 2024-02-19T22:02:51.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:51.750+0000 7f1ee07aa700 1 -- 172.21.15.86:0/2793382728 shutdown_connections 2024-02-19T22:02:51.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:51.750+0000 7f1ee07aa700 1 --2- 172.21.15.86:0/2793382728 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f1ec402ff10 0x7f1ec40323d0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:02:51.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:51.750+0000 7f1ee07aa700 1 --2- 172.21.15.86:0/2793382728 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1ed80f6510 0x7f1ed81067e0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:02:51.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:51.750+0000 7f1ee07aa700 1 -- 172.21.15.86:0/2793382728 >> 172.21.15.86:0/2793382728 conn(0x7f1ed80f1c70 msgr2=0x7f1ed80f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:02:51.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:51.750+0000 7f1ee07aa700 1 -- 172.21.15.86:0/2793382728 shutdown_connections 2024-02-19T22:02:51.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:51.750+0000 7f1ee07aa700 1 -- 172.21.15.86:0/2793382728 wait complete. 2024-02-19T22:02:51.751 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T22:02:53.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:02:52 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:02:51.493926+0000 mgr.smithi053.fyhcdi (mgr.14162) 194 : cluster [DBG] pgmap v182: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:02:53.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:02:52 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:02:51.749712+0000 mon.smithi053 (mon.0) 228 : audit [DBG] from='client.? 172.21.15.86:0/2793382728' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:02:53.532 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:02:53.532 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:02:55.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:02:54 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:02:53.494312+0000 mgr.smithi053.fyhcdi (mgr.14162) 195 : cluster [DBG] pgmap v183: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:02:55.486 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:55.485+0000 7f1622128700 1 -- 172.21.15.86:0/798126123 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f161c0f6510 msgr2=0x7f161c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:02:55.486 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:55.485+0000 7f1622128700 1 --2- 172.21.15.86:0/798126123 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f161c0f6510 0x7f161c0f6930 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 rx=0x7f1610009fb0 tx=0x7f1610009b30).stop 2024-02-19T22:02:55.486 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:55.485+0000 7f1622128700 1 -- 172.21.15.86:0/798126123 shutdown_connections 2024-02-19T22:02:55.486 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:55.485+0000 7f1622128700 1 --2- 172.21.15.86:0/798126123 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f161c0f6510 0x7f161c0f6930 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:02:55.486 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:55.485+0000 7f1622128700 1 -- 172.21.15.86:0/798126123 >> 172.21.15.86:0/798126123 conn(0x7f161c0f1c70 msgr2=0x7f161c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:02:55.486 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:55.485+0000 7f1622128700 1 -- 172.21.15.86:0/798126123 shutdown_connections 2024-02-19T22:02:55.486 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:55.485+0000 7f1622128700 1 -- 172.21.15.86:0/798126123 wait complete. 2024-02-19T22:02:55.487 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:55.486+0000 7f1622128700 1 Processor -- start 2024-02-19T22:02:55.487 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:55.486+0000 7f1622128700 1 -- start start 2024-02-19T22:02:55.487 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:55.486+0000 7f1622128700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f161c06b960 0x7f161c069e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:02:55.487 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:55.486+0000 7f1622128700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f161c06bd80 con 0x7f161c06b960 2024-02-19T22:02:55.488 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:55.486+0000 7f161b7fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f161c06b960 0x7f161c069e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:02:55.489 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:55.487+0000 7f161b7fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f161c06b960 0x7f161c069e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:55102/0 (socket says 172.21.15.86:55102) 2024-02-19T22:02:55.489 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:55.487+0000 7f161b7fe700 1 -- 172.21.15.86:0/117836201 learned_addr learned my addr 172.21.15.86:0/117836201 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T22:02:55.489 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:55.487+0000 7f161b7fe700 1 -- 172.21.15.86:0/117836201 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1610005040 con 0x7f161c06b960 2024-02-19T22:02:55.489 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:55.487+0000 7f161b7fe700 1 --2- 172.21.15.86:0/117836201 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f161c06b960 0x7f161c069e20 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7f1610009f80 tx=0x7f1610009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:02:55.489 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:55.487+0000 7f1603fff700 1 -- 172.21.15.86:0/117836201 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1610013380 con 0x7f161c06b960 2024-02-19T22:02:55.489 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:55.487+0000 7f1603fff700 1 -- 172.21.15.86:0/117836201 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1610013790 con 0x7f161c06b960 2024-02-19T22:02:55.489 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:55.487+0000 7f1622128700 1 -- 172.21.15.86:0/117836201 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f161c06a310 con 0x7f161c06b960 2024-02-19T22:02:55.489 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:55.487+0000 7f1603fff700 1 -- 172.21.15.86:0/117836201 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1610014420 con 0x7f161c06b960 2024-02-19T22:02:55.489 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:55.487+0000 7f1622128700 1 -- 172.21.15.86:0/117836201 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f161c06a800 con 0x7f161c06b960 2024-02-19T22:02:55.489 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:55.488+0000 7f1603fff700 1 -- 172.21.15.86:0/117836201 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1610025060 con 0x7f161c06b960 2024-02-19T22:02:55.489 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:55.488+0000 7f1622128700 1 -- 172.21.15.86:0/117836201 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f161c045ad0 con 0x7f161c06b960 2024-02-19T22:02:55.489 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:55.488+0000 7f1603fff700 1 --2- 172.21.15.86:0/117836201 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f160402ff60 0x7f1604032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:02:55.490 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:55.488+0000 7f1603fff700 1 -- 172.21.15.86:0/117836201 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1610040dd0 con 0x7f161c06b960 2024-02-19T22:02:55.490 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:55.488+0000 7f161affd700 1 --2- 172.21.15.86:0/117836201 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f160402ff60 0x7f1604032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:02:55.490 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:55.489+0000 7f161affd700 1 --2- 172.21.15.86:0/117836201 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f160402ff60 0x7f1604032420 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f160c0096a0 tx=0x7f160c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:02:55.492 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:55.493+0000 7f1603fff700 1 -- 172.21.15.86:0/117836201 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f161000f280 con 0x7f161c06b960 2024-02-19T22:02:55.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:55.694+0000 7f1622128700 1 -- 172.21.15.86:0/117836201 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f161c10cae0 con 0x7f161c06b960 2024-02-19T22:02:55.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:55.695+0000 7f1603fff700 1 -- 172.21.15.86:0/117836201 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f161002d2d0 con 0x7f161c06b960 2024-02-19T22:02:55.695 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T22:02:55.695 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:02:55.696 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:55.696+0000 7f1622128700 1 -- 172.21.15.86:0/117836201 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f160402ff60 msgr2=0x7f1604032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:02:55.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:55.696+0000 7f1622128700 1 --2- 172.21.15.86:0/117836201 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f160402ff60 0x7f1604032420 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f160c0096a0 tx=0x7f160c006b40).stop 2024-02-19T22:02:55.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:55.696+0000 7f1622128700 1 -- 172.21.15.86:0/117836201 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f161c06b960 msgr2=0x7f161c069e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:02:55.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:55.696+0000 7f1622128700 1 --2- 172.21.15.86:0/117836201 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f161c06b960 0x7f161c069e20 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7f1610009f80 tx=0x7f1610009b30).stop 2024-02-19T22:02:55.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:55.696+0000 7f1622128700 1 -- 172.21.15.86:0/117836201 shutdown_connections 2024-02-19T22:02:55.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:55.696+0000 7f1622128700 1 --2- 172.21.15.86:0/117836201 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f160402ff60 0x7f1604032420 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:02:55.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:55.696+0000 7f1622128700 1 --2- 172.21.15.86:0/117836201 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f161c06b960 0x7f161c069e20 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:02:55.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:55.696+0000 7f1622128700 1 -- 172.21.15.86:0/117836201 >> 172.21.15.86:0/117836201 conn(0x7f161c0f1c70 msgr2=0x7f161c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:02:55.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:55.697+0000 7f1622128700 1 -- 172.21.15.86:0/117836201 shutdown_connections 2024-02-19T22:02:55.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:55.697+0000 7f1622128700 1 -- 172.21.15.86:0/117836201 wait complete. 2024-02-19T22:02:55.697 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T22:02:57.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:02:56 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:02:55.494522+0000 mgr.smithi053.fyhcdi (mgr.14162) 196 : cluster [DBG] pgmap v184: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:02:57.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:02:56 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:02:55.696069+0000 mon.smithi053 (mon.0) 229 : audit [DBG] from='client.? 172.21.15.86:0/117836201' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:02:57.420 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:02:57.421 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:02:59.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:02:58 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:02:57.494873+0000 mgr.smithi053.fyhcdi (mgr.14162) 197 : cluster [DBG] pgmap v185: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:02:59.344 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:59.342+0000 7fe7a4006700 1 -- 172.21.15.86:0/1130838167 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe79c0f6510 msgr2=0x7fe79c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:02:59.344 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:59.342+0000 7fe7a4006700 1 --2- 172.21.15.86:0/1130838167 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe79c0f6510 0x7fe79c0f6930 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 rx=0x7fe78c009fb0 tx=0x7fe78c009b30).stop 2024-02-19T22:02:59.344 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:59.342+0000 7fe7a4006700 1 -- 172.21.15.86:0/1130838167 shutdown_connections 2024-02-19T22:02:59.344 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:59.342+0000 7fe7a4006700 1 --2- 172.21.15.86:0/1130838167 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe79c0f6510 0x7fe79c0f6930 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:02:59.344 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:59.342+0000 7fe7a4006700 1 -- 172.21.15.86:0/1130838167 >> 172.21.15.86:0/1130838167 conn(0x7fe79c0f1c70 msgr2=0x7fe79c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:02:59.344 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:59.342+0000 7fe7a4006700 1 -- 172.21.15.86:0/1130838167 shutdown_connections 2024-02-19T22:02:59.344 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:59.342+0000 7fe7a4006700 1 -- 172.21.15.86:0/1130838167 wait complete. 2024-02-19T22:02:59.345 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:59.343+0000 7fe7a4006700 1 Processor -- start 2024-02-19T22:02:59.345 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:59.343+0000 7fe7a4006700 1 -- start start 2024-02-19T22:02:59.345 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:59.343+0000 7fe7a4006700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe79c06b960 0x7fe79c069e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:02:59.345 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:59.343+0000 7fe7a4006700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe79c06bd80 con 0x7fe79c06b960 2024-02-19T22:02:59.345 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:59.343+0000 7fe7a1da2700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe79c06b960 0x7fe79c069e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:02:59.345 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:59.343+0000 7fe7a1da2700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe79c06b960 0x7fe79c069e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:44694/0 (socket says 172.21.15.86:44694) 2024-02-19T22:02:59.345 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:59.343+0000 7fe7a1da2700 1 -- 172.21.15.86:0/461316531 learned_addr learned my addr 172.21.15.86:0/461316531 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T22:02:59.345 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:59.344+0000 7fe7a1da2700 1 -- 172.21.15.86:0/461316531 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe78c005040 con 0x7fe79c06b960 2024-02-19T22:02:59.345 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:59.344+0000 7fe7a1da2700 1 --2- 172.21.15.86:0/461316531 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe79c06b960 0x7fe79c069e20 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0x7fe78c009f80 tx=0x7fe78c009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:02:59.345 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:59.344+0000 7fe7927fc700 1 -- 172.21.15.86:0/461316531 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe78c013380 con 0x7fe79c06b960 2024-02-19T22:02:59.345 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:59.344+0000 7fe7927fc700 1 -- 172.21.15.86:0/461316531 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe78c013790 con 0x7fe79c06b960 2024-02-19T22:02:59.345 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:59.344+0000 7fe7a4006700 1 -- 172.21.15.86:0/461316531 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe79c06a310 con 0x7fe79c06b960 2024-02-19T22:02:59.345 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:59.344+0000 7fe7927fc700 1 -- 172.21.15.86:0/461316531 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe78c014420 con 0x7fe79c06b960 2024-02-19T22:02:59.346 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:59.344+0000 7fe7a4006700 1 -- 172.21.15.86:0/461316531 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe79c06a800 con 0x7fe79c06b960 2024-02-19T22:02:59.347 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:59.345+0000 7fe7927fc700 1 -- 172.21.15.86:0/461316531 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe78c025060 con 0x7fe79c06b960 2024-02-19T22:02:59.347 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:59.345+0000 7fe7a4006700 1 -- 172.21.15.86:0/461316531 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe79c045ad0 con 0x7fe79c06b960 2024-02-19T22:02:59.347 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:59.345+0000 7fe7927fc700 1 --2- 172.21.15.86:0/461316531 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fe78802ff60 0x7fe788032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:02:59.347 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:59.345+0000 7fe7927fc700 1 -- 172.21.15.86:0/461316531 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe78c040e30 con 0x7fe79c06b960 2024-02-19T22:02:59.347 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:59.345+0000 7fe7a15a1700 1 --2- 172.21.15.86:0/461316531 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fe78802ff60 0x7fe788032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:02:59.347 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:59.346+0000 7fe7a15a1700 1 --2- 172.21.15.86:0/461316531 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fe78802ff60 0x7fe788032420 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7fe7980096a0 tx=0x7fe798006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:02:59.349 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:59.350+0000 7fe7927fc700 1 -- 172.21.15.86:0/461316531 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe78c01cb60 con 0x7fe79c06b960 2024-02-19T22:02:59.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:59.555+0000 7fe7a4006700 1 -- 172.21.15.86:0/461316531 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe79c10cae0 con 0x7fe79c06b960 2024-02-19T22:02:59.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:59.557+0000 7fe7927fc700 1 -- 172.21.15.86:0/461316531 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe78c02d2d0 con 0x7fe79c06b960 2024-02-19T22:02:59.556 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T22:02:59.556 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:02:59.557 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:59.557+0000 7fe7a4006700 1 -- 172.21.15.86:0/461316531 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fe78802ff60 msgr2=0x7fe788032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:02:59.557 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:59.557+0000 7fe7a4006700 1 --2- 172.21.15.86:0/461316531 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fe78802ff60 0x7fe788032420 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7fe7980096a0 tx=0x7fe798006b40).stop 2024-02-19T22:02:59.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:59.557+0000 7fe7a4006700 1 -- 172.21.15.86:0/461316531 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe79c06b960 msgr2=0x7fe79c069e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:02:59.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:59.557+0000 7fe7a4006700 1 --2- 172.21.15.86:0/461316531 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe79c06b960 0x7fe79c069e20 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0x7fe78c009f80 tx=0x7fe78c009b30).stop 2024-02-19T22:02:59.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:59.557+0000 7fe7a4006700 1 -- 172.21.15.86:0/461316531 shutdown_connections 2024-02-19T22:02:59.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:59.557+0000 7fe7a4006700 1 --2- 172.21.15.86:0/461316531 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fe78802ff60 0x7fe788032420 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:02:59.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:59.558+0000 7fe7a4006700 1 --2- 172.21.15.86:0/461316531 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe79c06b960 0x7fe79c069e20 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:02:59.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:59.558+0000 7fe7a4006700 1 -- 172.21.15.86:0/461316531 >> 172.21.15.86:0/461316531 conn(0x7fe79c0f1c70 msgr2=0x7fe79c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:02:59.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:59.558+0000 7fe7a4006700 1 -- 172.21.15.86:0/461316531 shutdown_connections 2024-02-19T22:02:59.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:02:59.558+0000 7fe7a4006700 1 -- 172.21.15.86:0/461316531 wait complete. 2024-02-19T22:02:59.558 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T22:03:00.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:03:00 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:02:59.495267+0000 mgr.smithi053.fyhcdi (mgr.14162) 198 : cluster [DBG] pgmap v186: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:03:00.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:03:00 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:02:59.557044+0000 mon.smithi053 (mon.0) 230 : audit [DBG] from='client.? 172.21.15.86:0/461316531' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:03:01.312 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:03:01.313 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:03:03.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:03:02 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:03:01.495584+0000 mgr.smithi053.fyhcdi (mgr.14162) 199 : cluster [DBG] pgmap v187: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:03:03.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:03.244+0000 7f1655493700 1 -- 172.21.15.86:0/915841119 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f16500f5410 msgr2=0x7f16500f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:03:03.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:03.244+0000 7f1655493700 1 --2- 172.21.15.86:0/915841119 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f16500f5410 0x7f16500f5830 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 rx=0x7f1640009fb0 tx=0x7f1640009b30).stop 2024-02-19T22:03:03.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:03.244+0000 7f1655493700 1 -- 172.21.15.86:0/915841119 shutdown_connections 2024-02-19T22:03:03.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:03.244+0000 7f1655493700 1 --2- 172.21.15.86:0/915841119 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f16500f5410 0x7f16500f5830 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:03:03.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:03.244+0000 7f1655493700 1 -- 172.21.15.86:0/915841119 >> 172.21.15.86:0/915841119 conn(0x7f16500f0b70 msgr2=0x7f16500f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:03:03.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:03.244+0000 7f1655493700 1 -- 172.21.15.86:0/915841119 shutdown_connections 2024-02-19T22:03:03.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:03.244+0000 7f1655493700 1 -- 172.21.15.86:0/915841119 wait complete. 2024-02-19T22:03:03.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:03.245+0000 7f1655493700 1 Processor -- start 2024-02-19T22:03:03.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:03.245+0000 7f1655493700 1 -- start start 2024-02-19T22:03:03.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:03.245+0000 7f1655493700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f16500f5410 0x7f1650109520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:03:03.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:03.245+0000 7f1655493700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1650060240 con 0x7f16500f5410 2024-02-19T22:03:03.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:03.245+0000 7f164effd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f16500f5410 0x7f1650109520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:03:03.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:03.246+0000 7f164effd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f16500f5410 0x7f1650109520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:44706/0 (socket says 172.21.15.86:44706) 2024-02-19T22:03:03.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:03.246+0000 7f164effd700 1 -- 172.21.15.86:0/2626330116 learned_addr learned my addr 172.21.15.86:0/2626330116 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T22:03:03.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:03.246+0000 7f164effd700 1 -- 172.21.15.86:0/2626330116 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1640005040 con 0x7f16500f5410 2024-02-19T22:03:03.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:03.246+0000 7f164effd700 1 --2- 172.21.15.86:0/2626330116 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f16500f5410 0x7f1650109520 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7f1640005010 tx=0x7f164000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:03:03.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:03.246+0000 7f16377fe700 1 -- 172.21.15.86:0/2626330116 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1640009320 con 0x7f16500f5410 2024-02-19T22:03:03.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:03.246+0000 7f1655493700 1 -- 172.21.15.86:0/2626330116 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1650060440 con 0x7f16500f5410 2024-02-19T22:03:03.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:03.246+0000 7f16377fe700 1 -- 172.21.15.86:0/2626330116 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1640013380 con 0x7f16500f5410 2024-02-19T22:03:03.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:03.246+0000 7f16377fe700 1 -- 172.21.15.86:0/2626330116 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1640014e50 con 0x7f16500f5410 2024-02-19T22:03:03.248 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:03.246+0000 7f1655493700 1 -- 172.21.15.86:0/2626330116 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1650109bd0 con 0x7f16500f5410 2024-02-19T22:03:03.248 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:03.247+0000 7f16377fe700 1 -- 172.21.15.86:0/2626330116 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f16400263a0 con 0x7f16500f5410 2024-02-19T22:03:03.249 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:03.247+0000 7f1655493700 1 -- 172.21.15.86:0/2626330116 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1650045ad0 con 0x7f16500f5410 2024-02-19T22:03:03.249 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:03.247+0000 7f16377fe700 1 --2- 172.21.15.86:0/2626330116 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f163802ff10 0x7f16380323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:03:03.249 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:03.247+0000 7f16377fe700 1 -- 172.21.15.86:0/2626330116 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1640041140 con 0x7f16500f5410 2024-02-19T22:03:03.249 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:03.247+0000 7f164e7fc700 1 --2- 172.21.15.86:0/2626330116 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f163802ff10 0x7f16380323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:03:03.249 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:03.248+0000 7f164e7fc700 1 --2- 172.21.15.86:0/2626330116 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f163802ff10 0x7f16380323d0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f16440096a0 tx=0x7f1644006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:03:03.251 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:03.253+0000 7f16377fe700 1 -- 172.21.15.86:0/2626330116 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f164001ccb0 con 0x7f16500f5410 2024-02-19T22:03:03.457 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:03.457+0000 7f1655493700 1 -- 172.21.15.86:0/2626330116 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f165010a3f0 con 0x7f16500f5410 2024-02-19T22:03:03.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:03.458+0000 7f16377fe700 1 -- 172.21.15.86:0/2626330116 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f164001cad0 con 0x7f16500f5410 2024-02-19T22:03:03.458 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T22:03:03.458 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:03:03.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:03.460+0000 7f1655493700 1 -- 172.21.15.86:0/2626330116 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f163802ff10 msgr2=0x7f16380323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:03:03.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:03.460+0000 7f1655493700 1 --2- 172.21.15.86:0/2626330116 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f163802ff10 0x7f16380323d0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f16440096a0 tx=0x7f1644006b40).stop 2024-02-19T22:03:03.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:03.460+0000 7f1655493700 1 -- 172.21.15.86:0/2626330116 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f16500f5410 msgr2=0x7f1650109520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:03:03.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:03.460+0000 7f1655493700 1 --2- 172.21.15.86:0/2626330116 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f16500f5410 0x7f1650109520 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7f1640005010 tx=0x7f164000b040).stop 2024-02-19T22:03:03.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:03.460+0000 7f1655493700 1 -- 172.21.15.86:0/2626330116 shutdown_connections 2024-02-19T22:03:03.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:03.460+0000 7f1655493700 1 --2- 172.21.15.86:0/2626330116 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f163802ff10 0x7f16380323d0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:03:03.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:03.460+0000 7f1655493700 1 --2- 172.21.15.86:0/2626330116 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f16500f5410 0x7f1650109520 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:03:03.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:03.460+0000 7f1655493700 1 -- 172.21.15.86:0/2626330116 >> 172.21.15.86:0/2626330116 conn(0x7f16500f0b70 msgr2=0x7f16500f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:03:03.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:03.460+0000 7f1655493700 1 -- 172.21.15.86:0/2626330116 shutdown_connections 2024-02-19T22:03:03.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:03.460+0000 7f1655493700 1 -- 172.21.15.86:0/2626330116 wait complete. 2024-02-19T22:03:03.461 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T22:03:04.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:03:03 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:03:03.459520+0000 mon.smithi053 (mon.0) 231 : audit [DBG] from='client.? 172.21.15.86:0/2626330116' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:03:05.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:03:04 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:03:03.495852+0000 mgr.smithi053.fyhcdi (mgr.14162) 200 : cluster [DBG] pgmap v188: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:03:05.237 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:03:05.237 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:03:07.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:03:06 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:03:05.496138+0000 mgr.smithi053.fyhcdi (mgr.14162) 201 : cluster [DBG] pgmap v189: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:03:07.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:07.145+0000 7f9cab564700 1 -- 172.21.15.86:0/1008496561 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9ca40601e0 msgr2=0x7f9ca40f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:03:07.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:07.145+0000 7f9cab564700 1 --2- 172.21.15.86:0/1008496561 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9ca40601e0 0x7f9ca40f8f20 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 rx=0x7f9ca0009fb0 tx=0x7f9ca0009b30).stop 2024-02-19T22:03:07.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:07.145+0000 7f9cab564700 1 -- 172.21.15.86:0/1008496561 shutdown_connections 2024-02-19T22:03:07.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:07.145+0000 7f9cab564700 1 --2- 172.21.15.86:0/1008496561 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9ca40601e0 0x7f9ca40f8f20 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:03:07.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:07.145+0000 7f9cab564700 1 -- 172.21.15.86:0/1008496561 >> 172.21.15.86:0/1008496561 conn(0x7f9ca40f09a0 msgr2=0x7f9ca40f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:03:07.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:07.145+0000 7f9cab564700 1 -- 172.21.15.86:0/1008496561 shutdown_connections 2024-02-19T22:03:07.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:07.145+0000 7f9cab564700 1 -- 172.21.15.86:0/1008496561 wait complete. 2024-02-19T22:03:07.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:07.146+0000 7f9cab564700 1 Processor -- start 2024-02-19T22:03:07.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:07.146+0000 7f9cab564700 1 -- start start 2024-02-19T22:03:07.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:07.146+0000 7f9cab564700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9ca40601e0 0x7f9ca40f69a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:03:07.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:07.146+0000 7f9cab564700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ca40f4e20 con 0x7f9ca40601e0 2024-02-19T22:03:07.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:07.146+0000 7f9caa562700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9ca40601e0 0x7f9ca40f69a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:03:07.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:07.146+0000 7f9caa562700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9ca40601e0 0x7f9ca40f69a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:44730/0 (socket says 172.21.15.86:44730) 2024-02-19T22:03:07.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:07.146+0000 7f9caa562700 1 -- 172.21.15.86:0/3603711054 learned_addr learned my addr 172.21.15.86:0/3603711054 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T22:03:07.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:07.146+0000 7f9caa562700 1 -- 172.21.15.86:0/3603711054 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ca0005040 con 0x7f9ca40601e0 2024-02-19T22:03:07.147 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:07.146+0000 7f9caa562700 1 --2- 172.21.15.86:0/3603711054 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9ca40601e0 0x7f9ca40f69a0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7f9ca0005010 tx=0x7f9ca000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:03:07.147 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:07.147+0000 7f9c9affd700 1 -- 172.21.15.86:0/3603711054 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9ca0009320 con 0x7f9ca40601e0 2024-02-19T22:03:07.147 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:07.147+0000 7f9cab564700 1 -- 172.21.15.86:0/3603711054 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ca40f4fc0 con 0x7f9ca40601e0 2024-02-19T22:03:07.147 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:07.147+0000 7f9c9affd700 1 -- 172.21.15.86:0/3603711054 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9ca0013380 con 0x7f9ca40601e0 2024-02-19T22:03:07.147 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:07.147+0000 7f9c9affd700 1 -- 172.21.15.86:0/3603711054 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9ca0014e50 con 0x7f9ca40601e0 2024-02-19T22:03:07.147 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:07.147+0000 7f9cab564700 1 -- 172.21.15.86:0/3603711054 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ca40f5450 con 0x7f9ca40601e0 2024-02-19T22:03:07.148 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:07.147+0000 7f9cab564700 1 -- 172.21.15.86:0/3603711054 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ca4059100 con 0x7f9ca40601e0 2024-02-19T22:03:07.148 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:07.148+0000 7f9c9affd700 1 -- 172.21.15.86:0/3603711054 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9ca00095f0 con 0x7f9ca40601e0 2024-02-19T22:03:07.148 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:07.148+0000 7f9c9affd700 1 --2- 172.21.15.86:0/3603711054 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f9c9002ff10 0x7f9c900323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:03:07.148 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:07.148+0000 7f9c9affd700 1 -- 172.21.15.86:0/3603711054 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9ca0040e90 con 0x7f9ca40601e0 2024-02-19T22:03:07.148 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:07.148+0000 7f9ca9d61700 1 --2- 172.21.15.86:0/3603711054 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f9c9002ff10 0x7f9c900323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:03:07.148 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:07.148+0000 7f9ca9d61700 1 --2- 172.21.15.86:0/3603711054 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f9c9002ff10 0x7f9c900323d0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f9c940096a0 tx=0x7f9c94006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:03:07.151 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:07.152+0000 7f9c9affd700 1 -- 172.21.15.86:0/3603711054 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9ca0014420 con 0x7f9ca40601e0 2024-02-19T22:03:07.354 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:07.355+0000 7f9cab564700 1 -- 172.21.15.86:0/3603711054 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9ca4045ad0 con 0x7f9ca40601e0 2024-02-19T22:03:07.354 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:07.355+0000 7f9c9affd700 1 -- 172.21.15.86:0/3603711054 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9ca0010230 con 0x7f9ca40601e0 2024-02-19T22:03:07.354 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T22:03:07.354 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:03:07.356 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:07.357+0000 7f9cab564700 1 -- 172.21.15.86:0/3603711054 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f9c9002ff10 msgr2=0x7f9c900323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:03:07.356 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:07.357+0000 7f9cab564700 1 --2- 172.21.15.86:0/3603711054 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f9c9002ff10 0x7f9c900323d0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f9c940096a0 tx=0x7f9c94006b40).stop 2024-02-19T22:03:07.356 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:07.357+0000 7f9cab564700 1 -- 172.21.15.86:0/3603711054 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9ca40601e0 msgr2=0x7f9ca40f69a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:03:07.356 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:07.357+0000 7f9cab564700 1 --2- 172.21.15.86:0/3603711054 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9ca40601e0 0x7f9ca40f69a0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7f9ca0005010 tx=0x7f9ca000b040).stop 2024-02-19T22:03:07.356 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:07.357+0000 7f9cab564700 1 -- 172.21.15.86:0/3603711054 shutdown_connections 2024-02-19T22:03:07.356 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:07.357+0000 7f9cab564700 1 --2- 172.21.15.86:0/3603711054 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f9c9002ff10 0x7f9c900323d0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:03:07.356 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:07.357+0000 7f9cab564700 1 --2- 172.21.15.86:0/3603711054 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9ca40601e0 0x7f9ca40f69a0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:03:07.357 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:07.357+0000 7f9cab564700 1 -- 172.21.15.86:0/3603711054 >> 172.21.15.86:0/3603711054 conn(0x7f9ca40f09a0 msgr2=0x7f9ca40f13d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:03:07.357 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:07.357+0000 7f9cab564700 1 -- 172.21.15.86:0/3603711054 shutdown_connections 2024-02-19T22:03:07.357 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:07.357+0000 7f9cab564700 1 -- 172.21.15.86:0/3603711054 wait complete. 2024-02-19T22:03:07.357 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T22:03:08.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:03:07 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:03:07.355809+0000 mon.smithi053 (mon.0) 232 : audit [DBG] from='client.? 172.21.15.86:0/3603711054' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:03:09.003 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:03:09.003 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:03:09.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:03:08 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:03:07.496476+0000 mgr.smithi053.fyhcdi (mgr.14162) 202 : cluster [DBG] pgmap v190: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:03:10.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:03:10 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:03:09.496854+0000 mgr.smithi053.fyhcdi (mgr.14162) 203 : cluster [DBG] pgmap v191: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:03:10.932 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:10.932+0000 7f708f2ae700 1 -- 172.21.15.86:0/846476386 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f70880f6510 msgr2=0x7f70880f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:03:10.932 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:10.932+0000 7f708f2ae700 1 --2- 172.21.15.86:0/846476386 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f70880f6510 0x7f70880f6930 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 rx=0x7f7084009fb0 tx=0x7f7084009b30).stop 2024-02-19T22:03:10.932 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:10.932+0000 7f708f2ae700 1 -- 172.21.15.86:0/846476386 shutdown_connections 2024-02-19T22:03:10.932 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:10.932+0000 7f708f2ae700 1 --2- 172.21.15.86:0/846476386 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f70880f6510 0x7f70880f6930 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:03:10.932 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:10.932+0000 7f708f2ae700 1 -- 172.21.15.86:0/846476386 >> 172.21.15.86:0/846476386 conn(0x7f70880f1c70 msgr2=0x7f70880f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:03:10.933 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:10.932+0000 7f708f2ae700 1 -- 172.21.15.86:0/846476386 shutdown_connections 2024-02-19T22:03:10.933 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:10.932+0000 7f708f2ae700 1 -- 172.21.15.86:0/846476386 wait complete. 2024-02-19T22:03:10.933 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:10.932+0000 7f708f2ae700 1 Processor -- start 2024-02-19T22:03:10.933 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:10.932+0000 7f708f2ae700 1 -- start start 2024-02-19T22:03:10.933 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:10.933+0000 7f708f2ae700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f70880f6510 0x7f70881067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:03:10.933 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:10.933+0000 7f708f2ae700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7088106cd0 con 0x7f70880f6510 2024-02-19T22:03:10.933 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:10.933+0000 7f708d04a700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f70880f6510 0x7f70881067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:03:10.933 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:10.933+0000 7f708d04a700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f70880f6510 0x7f70881067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:46340/0 (socket says 172.21.15.86:46340) 2024-02-19T22:03:10.933 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:10.933+0000 7f708d04a700 1 -- 172.21.15.86:0/2233248744 learned_addr learned my addr 172.21.15.86:0/2233248744 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T22:03:10.933 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:10.933+0000 7f708d04a700 1 -- 172.21.15.86:0/2233248744 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7084005040 con 0x7f70880f6510 2024-02-19T22:03:10.934 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:10.933+0000 7f708d04a700 1 --2- 172.21.15.86:0/2233248744 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f70880f6510 0x7f70881067e0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7f7084005010 tx=0x7f708400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:03:10.934 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:10.933+0000 7f707dffb700 1 -- 172.21.15.86:0/2233248744 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7084009320 con 0x7f70880f6510 2024-02-19T22:03:10.934 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:10.933+0000 7f707dffb700 1 -- 172.21.15.86:0/2233248744 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7084013380 con 0x7f70880f6510 2024-02-19T22:03:10.934 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:10.933+0000 7f708f2ae700 1 -- 172.21.15.86:0/2233248744 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7088106ed0 con 0x7f70880f6510 2024-02-19T22:03:10.934 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:10.933+0000 7f707dffb700 1 -- 172.21.15.86:0/2233248744 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7084014e50 con 0x7f70880f6510 2024-02-19T22:03:10.934 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:10.934+0000 7f708f2ae700 1 -- 172.21.15.86:0/2233248744 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7088107360 con 0x7f70880f6510 2024-02-19T22:03:10.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:10.934+0000 7f707dffb700 1 -- 172.21.15.86:0/2233248744 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f70840095f0 con 0x7f70880f6510 2024-02-19T22:03:10.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:10.934+0000 7f708f2ae700 1 -- 172.21.15.86:0/2233248744 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7088045ad0 con 0x7f70880f6510 2024-02-19T22:03:10.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:10.935+0000 7f707dffb700 1 --2- 172.21.15.86:0/2233248744 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f707402ff10 0x7f70740323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:03:10.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:10.935+0000 7f707dffb700 1 -- 172.21.15.86:0/2233248744 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7084040f10 con 0x7f70880f6510 2024-02-19T22:03:10.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:10.935+0000 7f708c849700 1 --2- 172.21.15.86:0/2233248744 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f707402ff10 0x7f70740323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:03:10.936 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:10.935+0000 7f708c849700 1 --2- 172.21.15.86:0/2233248744 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f707402ff10 0x7f70740323d0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7f70780096a0 tx=0x7f7078006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:03:10.938 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:10.940+0000 7f707dffb700 1 -- 172.21.15.86:0/2233248744 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7084014420 con 0x7f70880f6510 2024-02-19T22:03:11.147 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:11.148+0000 7f708f2ae700 1 -- 172.21.15.86:0/2233248744 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7088108210 con 0x7f70880f6510 2024-02-19T22:03:11.147 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:11.149+0000 7f707dffb700 1 -- 172.21.15.86:0/2233248744 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7084010230 con 0x7f70880f6510 2024-02-19T22:03:11.148 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T22:03:11.148 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:03:11.150 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:11.150+0000 7f708f2ae700 1 -- 172.21.15.86:0/2233248744 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f707402ff10 msgr2=0x7f70740323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:03:11.150 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:11.150+0000 7f708f2ae700 1 --2- 172.21.15.86:0/2233248744 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f707402ff10 0x7f70740323d0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7f70780096a0 tx=0x7f7078006b40).stop 2024-02-19T22:03:11.150 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:11.150+0000 7f708f2ae700 1 -- 172.21.15.86:0/2233248744 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f70880f6510 msgr2=0x7f70881067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:03:11.150 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:11.150+0000 7f708f2ae700 1 --2- 172.21.15.86:0/2233248744 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f70880f6510 0x7f70881067e0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7f7084005010 tx=0x7f708400b040).stop 2024-02-19T22:03:11.150 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:11.150+0000 7f708f2ae700 1 -- 172.21.15.86:0/2233248744 shutdown_connections 2024-02-19T22:03:11.150 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:11.150+0000 7f708f2ae700 1 --2- 172.21.15.86:0/2233248744 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f707402ff10 0x7f70740323d0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:03:11.150 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:11.150+0000 7f708f2ae700 1 --2- 172.21.15.86:0/2233248744 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f70880f6510 0x7f70881067e0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:03:11.150 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:11.150+0000 7f708f2ae700 1 -- 172.21.15.86:0/2233248744 >> 172.21.15.86:0/2233248744 conn(0x7f70880f1c70 msgr2=0x7f70880f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:03:11.151 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:11.150+0000 7f708f2ae700 1 -- 172.21.15.86:0/2233248744 shutdown_connections 2024-02-19T22:03:11.151 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:11.151+0000 7f708f2ae700 1 -- 172.21.15.86:0/2233248744 wait complete. 2024-02-19T22:03:11.151 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T22:03:11.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:03:11 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:03:11.149195+0000 mon.smithi053 (mon.0) 233 : audit [DBG] from='client.? 172.21.15.86:0/2233248744' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:03:12.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:03:12 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:03:11.497133+0000 mgr.smithi053.fyhcdi (mgr.14162) 204 : cluster [DBG] pgmap v192: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:03:12.803 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:03:12.804 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:03:14.785 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:14.783+0000 7f1d1f4b1700 1 -- 172.21.15.86:0/592032329 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1d180f5410 msgr2=0x7f1d180f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:03:14.785 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:14.783+0000 7f1d1f4b1700 1 --2- 172.21.15.86:0/592032329 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1d180f5410 0x7f1d180f5830 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 rx=0x7f1d08009fb0 tx=0x7f1d08009b30).stop 2024-02-19T22:03:14.785 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:14.783+0000 7f1d1f4b1700 1 -- 172.21.15.86:0/592032329 shutdown_connections 2024-02-19T22:03:14.785 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:14.783+0000 7f1d1f4b1700 1 --2- 172.21.15.86:0/592032329 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1d180f5410 0x7f1d180f5830 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:03:14.785 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:14.783+0000 7f1d1f4b1700 1 -- 172.21.15.86:0/592032329 >> 172.21.15.86:0/592032329 conn(0x7f1d180f0b70 msgr2=0x7f1d180f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:03:14.785 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:14.783+0000 7f1d1f4b1700 1 -- 172.21.15.86:0/592032329 shutdown_connections 2024-02-19T22:03:14.785 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:14.783+0000 7f1d1f4b1700 1 -- 172.21.15.86:0/592032329 wait complete. 2024-02-19T22:03:14.785 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:14.784+0000 7f1d1f4b1700 1 Processor -- start 2024-02-19T22:03:14.785 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:14.784+0000 7f1d1f4b1700 1 -- start start 2024-02-19T22:03:14.785 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:14.784+0000 7f1d1f4b1700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1d180f5410 0x7f1d181089e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:03:14.785 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:14.784+0000 7f1d1f4b1700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d18108ed0 con 0x7f1d180f5410 2024-02-19T22:03:14.786 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:14.784+0000 7f1d1d24d700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1d180f5410 0x7f1d181089e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:03:14.786 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:14.784+0000 7f1d1d24d700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1d180f5410 0x7f1d181089e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:46366/0 (socket says 172.21.15.86:46366) 2024-02-19T22:03:14.786 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:14.784+0000 7f1d1d24d700 1 -- 172.21.15.86:0/269394108 learned_addr learned my addr 172.21.15.86:0/269394108 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T22:03:14.786 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:14.784+0000 7f1d1d24d700 1 -- 172.21.15.86:0/269394108 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d08005040 con 0x7f1d180f5410 2024-02-19T22:03:14.786 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:14.785+0000 7f1d1d24d700 1 --2- 172.21.15.86:0/269394108 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1d180f5410 0x7f1d181089e0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0x7f1d08005010 tx=0x7f1d0800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:03:14.786 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:14.785+0000 7f1d0dffb700 1 -- 172.21.15.86:0/269394108 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1d08009320 con 0x7f1d180f5410 2024-02-19T22:03:14.786 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:14.785+0000 7f1d0dffb700 1 -- 172.21.15.86:0/269394108 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1d08013380 con 0x7f1d180f5410 2024-02-19T22:03:14.786 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:14.785+0000 7f1d1f4b1700 1 -- 172.21.15.86:0/269394108 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d181090d0 con 0x7f1d180f5410 2024-02-19T22:03:14.786 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:14.785+0000 7f1d0dffb700 1 -- 172.21.15.86:0/269394108 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1d08014e50 con 0x7f1d180f5410 2024-02-19T22:03:14.786 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:14.785+0000 7f1d1f4b1700 1 -- 172.21.15.86:0/269394108 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d18109560 con 0x7f1d180f5410 2024-02-19T22:03:14.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:14.786+0000 7f1d0dffb700 1 -- 172.21.15.86:0/269394108 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1d080095f0 con 0x7f1d180f5410 2024-02-19T22:03:14.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:14.786+0000 7f1d1f4b1700 1 -- 172.21.15.86:0/269394108 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d18045ad0 con 0x7f1d180f5410 2024-02-19T22:03:14.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:14.786+0000 7f1d0dffb700 1 --2- 172.21.15.86:0/269394108 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f1d0402ff10 0x7f1d040323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:03:14.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:14.786+0000 7f1d0dffb700 1 -- 172.21.15.86:0/269394108 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1d08040e70 con 0x7f1d180f5410 2024-02-19T22:03:14.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:14.786+0000 7f1d1ca4c700 1 --2- 172.21.15.86:0/269394108 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f1d0402ff10 0x7f1d040323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:03:14.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:14.786+0000 7f1d1ca4c700 1 --2- 172.21.15.86:0/269394108 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f1d0402ff10 0x7f1d040323d0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f1d140096a0 tx=0x7f1d14006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:03:14.790 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:14.791+0000 7f1d0dffb700 1 -- 172.21.15.86:0/269394108 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1d08014420 con 0x7f1d180f5410 2024-02-19T22:03:14.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:14.997+0000 7f1d1f4b1700 1 -- 172.21.15.86:0/269394108 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1d1810a3b0 con 0x7f1d180f5410 2024-02-19T22:03:14.998 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:14.998+0000 7f1d0dffb700 1 -- 172.21.15.86:0/269394108 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1d08010230 con 0x7f1d180f5410 2024-02-19T22:03:14.998 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T22:03:14.998 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:03:15.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:14.999+0000 7f1d1f4b1700 1 -- 172.21.15.86:0/269394108 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f1d0402ff10 msgr2=0x7f1d040323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:03:15.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:14.999+0000 7f1d1f4b1700 1 --2- 172.21.15.86:0/269394108 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f1d0402ff10 0x7f1d040323d0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f1d140096a0 tx=0x7f1d14006b40).stop 2024-02-19T22:03:15.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:14.999+0000 7f1d1f4b1700 1 -- 172.21.15.86:0/269394108 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1d180f5410 msgr2=0x7f1d181089e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:03:15.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:14.999+0000 7f1d1f4b1700 1 --2- 172.21.15.86:0/269394108 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1d180f5410 0x7f1d181089e0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0x7f1d08005010 tx=0x7f1d0800b040).stop 2024-02-19T22:03:15.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:15.000+0000 7f1d1f4b1700 1 -- 172.21.15.86:0/269394108 shutdown_connections 2024-02-19T22:03:15.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:15.000+0000 7f1d1f4b1700 1 --2- 172.21.15.86:0/269394108 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f1d0402ff10 0x7f1d040323d0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:03:15.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:15.000+0000 7f1d1f4b1700 1 --2- 172.21.15.86:0/269394108 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1d180f5410 0x7f1d181089e0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:03:15.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:15.000+0000 7f1d1f4b1700 1 -- 172.21.15.86:0/269394108 >> 172.21.15.86:0/269394108 conn(0x7f1d180f0b70 msgr2=0x7f1d1805fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:03:15.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:15.000+0000 7f1d1f4b1700 1 -- 172.21.15.86:0/269394108 shutdown_connections 2024-02-19T22:03:15.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:15.000+0000 7f1d1f4b1700 1 -- 172.21.15.86:0/269394108 wait complete. 2024-02-19T22:03:15.001 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T22:03:15.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:03:14 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:03:13.497444+0000 mgr.smithi053.fyhcdi (mgr.14162) 205 : cluster [DBG] pgmap v193: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:03:16.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:03:15 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:03:14.999210+0000 mon.smithi053 (mon.0) 234 : audit [DBG] from='client.? 172.21.15.86:0/269394108' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:03:16.660 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:03:16.661 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:03:17.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:03:16 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:03:15.497809+0000 mgr.smithi053.fyhcdi (mgr.14162) 206 : cluster [DBG] pgmap v194: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:03:18.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:18.591+0000 7fcf1fbda700 1 -- 172.21.15.86:0/1843931583 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcf180f6510 msgr2=0x7fcf180f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:03:18.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:18.591+0000 7fcf1fbda700 1 --2- 172.21.15.86:0/1843931583 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcf180f6510 0x7fcf180f6930 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 rx=0x7fcf08009fb0 tx=0x7fcf08009b30).stop 2024-02-19T22:03:18.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:18.591+0000 7fcf1fbda700 1 -- 172.21.15.86:0/1843931583 shutdown_connections 2024-02-19T22:03:18.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:18.591+0000 7fcf1fbda700 1 --2- 172.21.15.86:0/1843931583 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcf180f6510 0x7fcf180f6930 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:03:18.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:18.591+0000 7fcf1fbda700 1 -- 172.21.15.86:0/1843931583 >> 172.21.15.86:0/1843931583 conn(0x7fcf180f1c70 msgr2=0x7fcf180f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:03:18.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:18.591+0000 7fcf1fbda700 1 -- 172.21.15.86:0/1843931583 shutdown_connections 2024-02-19T22:03:18.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:18.591+0000 7fcf1fbda700 1 -- 172.21.15.86:0/1843931583 wait complete. 2024-02-19T22:03:18.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:18.592+0000 7fcf1fbda700 1 Processor -- start 2024-02-19T22:03:18.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:18.592+0000 7fcf1fbda700 1 -- start start 2024-02-19T22:03:18.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:18.592+0000 7fcf1fbda700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcf180f6510 0x7fcf1806b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:03:18.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:18.592+0000 7fcf1fbda700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf18069e20 con 0x7fcf180f6510 2024-02-19T22:03:18.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:18.592+0000 7fcf1d976700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcf180f6510 0x7fcf1806b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:03:18.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:18.592+0000 7fcf1d976700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcf180f6510 0x7fcf1806b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:46384/0 (socket says 172.21.15.86:46384) 2024-02-19T22:03:18.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:18.592+0000 7fcf1d976700 1 -- 172.21.15.86:0/3316214744 learned_addr learned my addr 172.21.15.86:0/3316214744 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T22:03:18.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:18.592+0000 7fcf1d976700 1 -- 172.21.15.86:0/3316214744 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf08005040 con 0x7fcf180f6510 2024-02-19T22:03:18.595 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:18.592+0000 7fcf1d976700 1 --2- 172.21.15.86:0/3316214744 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcf180f6510 0x7fcf1806b9a0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7fcf08005010 tx=0x7fcf0800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:03:18.595 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:18.593+0000 7fcf0e7fc700 1 -- 172.21.15.86:0/3316214744 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcf08009320 con 0x7fcf180f6510 2024-02-19T22:03:18.595 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:18.593+0000 7fcf1fbda700 1 -- 172.21.15.86:0/3316214744 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf18069fc0 con 0x7fcf180f6510 2024-02-19T22:03:18.595 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:18.593+0000 7fcf0e7fc700 1 -- 172.21.15.86:0/3316214744 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcf08013380 con 0x7fcf180f6510 2024-02-19T22:03:18.595 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:18.593+0000 7fcf0e7fc700 1 -- 172.21.15.86:0/3316214744 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcf08014e50 con 0x7fcf180f6510 2024-02-19T22:03:18.595 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:18.593+0000 7fcf1fbda700 1 -- 172.21.15.86:0/3316214744 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf1806a450 con 0x7fcf180f6510 2024-02-19T22:03:18.595 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:18.593+0000 7fcf0e7fc700 1 -- 172.21.15.86:0/3316214744 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fcf08014420 con 0x7fcf180f6510 2024-02-19T22:03:18.595 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:18.594+0000 7fcf0e7fc700 1 --2- 172.21.15.86:0/3316214744 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fcf0402ff10 0x7fcf040323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:03:18.595 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:18.594+0000 7fcf1fbda700 1 -- 172.21.15.86:0/3316214744 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf18059100 con 0x7fcf180f6510 2024-02-19T22:03:18.595 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:18.594+0000 7fcf0e7fc700 1 -- 172.21.15.86:0/3316214744 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fcf08040eb0 con 0x7fcf180f6510 2024-02-19T22:03:18.595 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:18.594+0000 7fcf1d175700 1 --2- 172.21.15.86:0/3316214744 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fcf0402ff10 0x7fcf040323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:03:18.596 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:18.594+0000 7fcf1d175700 1 --2- 172.21.15.86:0/3316214744 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fcf0402ff10 0x7fcf040323d0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7fcf140096a0 tx=0x7fcf14006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:03:18.598 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:18.598+0000 7fcf0e7fc700 1 -- 172.21.15.86:0/3316214744 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcf080672b0 con 0x7fcf180f6510 2024-02-19T22:03:18.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:18.801+0000 7fcf1fbda700 1 -- 172.21.15.86:0/3316214744 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcf180fc0a0 con 0x7fcf180f6510 2024-02-19T22:03:18.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:18.802+0000 7fcf0e7fc700 1 -- 172.21.15.86:0/3316214744 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fcf080095f0 con 0x7fcf180f6510 2024-02-19T22:03:18.803 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T22:03:18.803 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:03:18.805 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:18.804+0000 7fcf1fbda700 1 -- 172.21.15.86:0/3316214744 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fcf0402ff10 msgr2=0x7fcf040323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:03:18.805 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:18.804+0000 7fcf1fbda700 1 --2- 172.21.15.86:0/3316214744 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fcf0402ff10 0x7fcf040323d0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7fcf140096a0 tx=0x7fcf14006b40).stop 2024-02-19T22:03:18.805 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:18.804+0000 7fcf1fbda700 1 -- 172.21.15.86:0/3316214744 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcf180f6510 msgr2=0x7fcf1806b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:03:18.805 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:18.804+0000 7fcf1fbda700 1 --2- 172.21.15.86:0/3316214744 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcf180f6510 0x7fcf1806b9a0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7fcf08005010 tx=0x7fcf0800b040).stop 2024-02-19T22:03:18.806 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:18.804+0000 7fcf1fbda700 1 -- 172.21.15.86:0/3316214744 shutdown_connections 2024-02-19T22:03:18.806 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:18.804+0000 7fcf1fbda700 1 --2- 172.21.15.86:0/3316214744 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fcf0402ff10 0x7fcf040323d0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:03:18.806 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:18.804+0000 7fcf1fbda700 1 --2- 172.21.15.86:0/3316214744 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcf180f6510 0x7fcf1806b9a0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:03:18.806 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:18.804+0000 7fcf1fbda700 1 -- 172.21.15.86:0/3316214744 >> 172.21.15.86:0/3316214744 conn(0x7fcf180f1c70 msgr2=0x7fcf180f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:03:18.806 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:18.804+0000 7fcf1fbda700 1 -- 172.21.15.86:0/3316214744 shutdown_connections 2024-02-19T22:03:18.806 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:18.804+0000 7fcf1fbda700 1 -- 172.21.15.86:0/3316214744 wait complete. 2024-02-19T22:03:18.807 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T22:03:19.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:03:18 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:03:17.498218+0000 mgr.smithi053.fyhcdi (mgr.14162) 207 : cluster [DBG] pgmap v195: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:03:20.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:03:19 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:03:18.803485+0000 mon.smithi053 (mon.0) 235 : audit [DBG] from='client.? 172.21.15.86:0/3316214744' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:03:20.588 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:03:20.588 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:03:21.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:03:20 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:03:19.498474+0000 mgr.smithi053.fyhcdi (mgr.14162) 208 : cluster [DBG] pgmap v196: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:03:22.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:22.513+0000 7f18168fb700 1 -- 172.21.15.86:0/1002690432 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f18100f62a0 msgr2=0x7f18100f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:03:22.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:22.513+0000 7f18168fb700 1 --2- 172.21.15.86:0/1002690432 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f18100f62a0 0x7f18100f66c0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 rx=0x7f1800009fb0 tx=0x7f1800009b30).stop 2024-02-19T22:03:22.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:22.513+0000 7f18168fb700 1 -- 172.21.15.86:0/1002690432 shutdown_connections 2024-02-19T22:03:22.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:22.513+0000 7f18168fb700 1 --2- 172.21.15.86:0/1002690432 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f18100f62a0 0x7f18100f66c0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:03:22.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:22.513+0000 7f18168fb700 1 -- 172.21.15.86:0/1002690432 >> 172.21.15.86:0/1002690432 conn(0x7f18100f1aa0 msgr2=0x7f18100f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:03:22.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:22.513+0000 7f18168fb700 1 -- 172.21.15.86:0/1002690432 shutdown_connections 2024-02-19T22:03:22.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:22.513+0000 7f18168fb700 1 -- 172.21.15.86:0/1002690432 wait complete. 2024-02-19T22:03:22.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:22.513+0000 7f18168fb700 1 Processor -- start 2024-02-19T22:03:22.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:22.513+0000 7f18168fb700 1 -- start start 2024-02-19T22:03:22.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:22.513+0000 7f18168fb700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f18100f62a0 0x7f181006b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:03:22.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:22.513+0000 7f18168fb700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1810069e20 con 0x7f18100f62a0 2024-02-19T22:03:22.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:22.514+0000 7f18158f9700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f18100f62a0 0x7f181006b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:03:22.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:22.514+0000 7f18158f9700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f18100f62a0 0x7f181006b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:43472/0 (socket says 172.21.15.86:43472) 2024-02-19T22:03:22.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:22.514+0000 7f18158f9700 1 -- 172.21.15.86:0/2432646192 learned_addr learned my addr 172.21.15.86:0/2432646192 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T22:03:22.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:22.514+0000 7f18158f9700 1 -- 172.21.15.86:0/2432646192 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1800005040 con 0x7f18100f62a0 2024-02-19T22:03:22.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:22.514+0000 7f18158f9700 1 --2- 172.21.15.86:0/2432646192 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f18100f62a0 0x7f181006b9a0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0x7f18000089f0 tx=0x7f180000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:03:22.516 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:22.514+0000 7f18067fc700 1 -- 172.21.15.86:0/2432646192 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1800009320 con 0x7f18100f62a0 2024-02-19T22:03:22.516 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:22.514+0000 7f18067fc700 1 -- 172.21.15.86:0/2432646192 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1800013380 con 0x7f18100f62a0 2024-02-19T22:03:22.516 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:22.514+0000 7f18168fb700 1 -- 172.21.15.86:0/2432646192 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1810069fc0 con 0x7f18100f62a0 2024-02-19T22:03:22.516 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:22.514+0000 7f18067fc700 1 -- 172.21.15.86:0/2432646192 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1800014e20 con 0x7f18100f62a0 2024-02-19T22:03:22.516 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:22.514+0000 7f18168fb700 1 -- 172.21.15.86:0/2432646192 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f181006a450 con 0x7f18100f62a0 2024-02-19T22:03:22.517 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:22.515+0000 7f18067fc700 1 -- 172.21.15.86:0/2432646192 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1800014420 con 0x7f18100f62a0 2024-02-19T22:03:22.517 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:22.515+0000 7f18168fb700 1 -- 172.21.15.86:0/2432646192 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1810059100 con 0x7f18100f62a0 2024-02-19T22:03:22.517 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:22.515+0000 7f18067fc700 1 --2- 172.21.15.86:0/2432646192 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f17fc02ff10 0x7f17fc0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:03:22.517 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:22.515+0000 7f18067fc700 1 -- 172.21.15.86:0/2432646192 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1800040f20 con 0x7f18100f62a0 2024-02-19T22:03:22.517 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:22.515+0000 7f18150f8700 1 --2- 172.21.15.86:0/2432646192 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f17fc02ff10 0x7f17fc0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:03:22.517 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:22.516+0000 7f18150f8700 1 --2- 172.21.15.86:0/2432646192 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f17fc02ff10 0x7f17fc0323d0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f180c0096a0 tx=0x7f180c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:03:22.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:22.520+0000 7f18067fc700 1 -- 172.21.15.86:0/2432646192 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f180001d980 con 0x7f18100f62a0 2024-02-19T22:03:22.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:22.723+0000 7f18168fb700 1 -- 172.21.15.86:0/2432646192 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f18100fbe30 con 0x7f18100f62a0 2024-02-19T22:03:22.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:22.723+0000 7f18067fc700 1 -- 172.21.15.86:0/2432646192 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1800010230 con 0x7f18100f62a0 2024-02-19T22:03:22.723 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T22:03:22.723 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:03:22.725 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:22.725+0000 7f18168fb700 1 -- 172.21.15.86:0/2432646192 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f17fc02ff10 msgr2=0x7f17fc0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:03:22.725 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:22.725+0000 7f18168fb700 1 --2- 172.21.15.86:0/2432646192 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f17fc02ff10 0x7f17fc0323d0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f180c0096a0 tx=0x7f180c006b40).stop 2024-02-19T22:03:22.725 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:22.725+0000 7f18168fb700 1 -- 172.21.15.86:0/2432646192 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f18100f62a0 msgr2=0x7f181006b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:03:22.725 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:22.725+0000 7f18168fb700 1 --2- 172.21.15.86:0/2432646192 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f18100f62a0 0x7f181006b9a0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0x7f18000089f0 tx=0x7f180000b040).stop 2024-02-19T22:03:22.725 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:22.725+0000 7f18168fb700 1 -- 172.21.15.86:0/2432646192 shutdown_connections 2024-02-19T22:03:22.725 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:22.725+0000 7f18168fb700 1 --2- 172.21.15.86:0/2432646192 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f17fc02ff10 0x7f17fc0323d0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:03:22.725 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:22.725+0000 7f18168fb700 1 --2- 172.21.15.86:0/2432646192 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f18100f62a0 0x7f181006b9a0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:03:22.725 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:22.725+0000 7f18168fb700 1 -- 172.21.15.86:0/2432646192 >> 172.21.15.86:0/2432646192 conn(0x7f18100f1aa0 msgr2=0x7f18100f2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:03:22.725 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:22.725+0000 7f18168fb700 1 -- 172.21.15.86:0/2432646192 shutdown_connections 2024-02-19T22:03:22.725 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:22.725+0000 7f18168fb700 1 -- 172.21.15.86:0/2432646192 wait complete. 2024-02-19T22:03:22.725 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T22:03:22.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:03:22 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:03:21.498765+0000 mgr.smithi053.fyhcdi (mgr.14162) 209 : cluster [DBG] pgmap v197: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:03:24.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:03:23 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:03:22.724621+0000 mon.smithi053 (mon.0) 236 : audit [DBG] from='client.? 172.21.15.86:0/2432646192' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:03:24.445 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:03:24.445 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:03:25.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:03:24 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:03:23.499116+0000 mgr.smithi053.fyhcdi (mgr.14162) 210 : cluster [DBG] pgmap v198: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:03:26.557 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:26.462+0000 7fd0d32a5700 1 -- 172.21.15.86:0/2991650496 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd0cc0f5410 msgr2=0x7fd0cc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:03:26.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:26.463+0000 7fd0d32a5700 1 --2- 172.21.15.86:0/2991650496 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd0cc0f5410 0x7fd0cc0f5830 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 rx=0x7fd0bc009fb0 tx=0x7fd0bc009b30).stop 2024-02-19T22:03:26.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:26.463+0000 7fd0d32a5700 1 -- 172.21.15.86:0/2991650496 shutdown_connections 2024-02-19T22:03:26.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:26.463+0000 7fd0d32a5700 1 --2- 172.21.15.86:0/2991650496 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd0cc0f5410 0x7fd0cc0f5830 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:03:26.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:26.463+0000 7fd0d32a5700 1 -- 172.21.15.86:0/2991650496 >> 172.21.15.86:0/2991650496 conn(0x7fd0cc0f0b70 msgr2=0x7fd0cc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:03:26.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:26.463+0000 7fd0d32a5700 1 -- 172.21.15.86:0/2991650496 shutdown_connections 2024-02-19T22:03:26.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:26.463+0000 7fd0d32a5700 1 -- 172.21.15.86:0/2991650496 wait complete. 2024-02-19T22:03:26.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:26.463+0000 7fd0d32a5700 1 Processor -- start 2024-02-19T22:03:26.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:26.463+0000 7fd0d32a5700 1 -- start start 2024-02-19T22:03:26.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:26.463+0000 7fd0d32a5700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd0cc0f5410 0x7fd0cc1056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:03:26.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:26.463+0000 7fd0d32a5700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0cc105bd0 con 0x7fd0cc0f5410 2024-02-19T22:03:26.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:26.464+0000 7fd0d1041700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd0cc0f5410 0x7fd0cc1056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:03:26.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:26.464+0000 7fd0d1041700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd0cc0f5410 0x7fd0cc1056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:43500/0 (socket says 172.21.15.86:43500) 2024-02-19T22:03:26.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:26.464+0000 7fd0d1041700 1 -- 172.21.15.86:0/888629418 learned_addr learned my addr 172.21.15.86:0/888629418 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T22:03:26.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:26.464+0000 7fd0d1041700 1 -- 172.21.15.86:0/888629418 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0bc005040 con 0x7fd0cc0f5410 2024-02-19T22:03:26.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:26.464+0000 7fd0d1041700 1 --2- 172.21.15.86:0/888629418 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd0cc0f5410 0x7fd0cc1056e0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0x7fd0bc005010 tx=0x7fd0bc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:03:26.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:26.464+0000 7fd0c1ffb700 1 -- 172.21.15.86:0/888629418 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd0bc009320 con 0x7fd0cc0f5410 2024-02-19T22:03:26.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:26.464+0000 7fd0d32a5700 1 -- 172.21.15.86:0/888629418 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0cc105dd0 con 0x7fd0cc0f5410 2024-02-19T22:03:26.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:26.464+0000 7fd0c1ffb700 1 -- 172.21.15.86:0/888629418 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd0bc013380 con 0x7fd0cc0f5410 2024-02-19T22:03:26.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:26.464+0000 7fd0c1ffb700 1 -- 172.21.15.86:0/888629418 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd0bc014e50 con 0x7fd0cc0f5410 2024-02-19T22:03:26.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:26.464+0000 7fd0d32a5700 1 -- 172.21.15.86:0/888629418 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0cc106260 con 0x7fd0cc0f5410 2024-02-19T22:03:26.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:26.465+0000 7fd0c1ffb700 1 -- 172.21.15.86:0/888629418 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd0bc0095f0 con 0x7fd0cc0f5410 2024-02-19T22:03:26.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:26.465+0000 7fd0d32a5700 1 -- 172.21.15.86:0/888629418 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0cc045ad0 con 0x7fd0cc0f5410 2024-02-19T22:03:26.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:26.465+0000 7fd0c1ffb700 1 --2- 172.21.15.86:0/888629418 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fd0b802ff10 0x7fd0b80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:03:26.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:26.465+0000 7fd0c1ffb700 1 -- 172.21.15.86:0/888629418 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd0bc040f80 con 0x7fd0cc0f5410 2024-02-19T22:03:26.560 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:26.466+0000 7fd0d0840700 1 --2- 172.21.15.86:0/888629418 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fd0b802ff10 0x7fd0b80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:03:26.560 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:26.466+0000 7fd0d0840700 1 --2- 172.21.15.86:0/888629418 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fd0b802ff10 0x7fd0b80323d0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7fd0c80096a0 tx=0x7fd0c8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:03:26.560 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:26.470+0000 7fd0c1ffb700 1 -- 172.21.15.86:0/888629418 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd0bc014420 con 0x7fd0cc0f5410 2024-02-19T22:03:26.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:26.677+0000 7fd0d32a5700 1 -- 172.21.15.86:0/888629418 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd0cc107110 con 0x7fd0cc0f5410 2024-02-19T22:03:26.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:26.678+0000 7fd0c1ffb700 1 -- 172.21.15.86:0/888629418 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd0bc010230 con 0x7fd0cc0f5410 2024-02-19T22:03:26.678 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T22:03:26.678 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:03:26.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:26.680+0000 7fd0d32a5700 1 -- 172.21.15.86:0/888629418 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fd0b802ff10 msgr2=0x7fd0b80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:03:26.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:26.680+0000 7fd0d32a5700 1 --2- 172.21.15.86:0/888629418 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fd0b802ff10 0x7fd0b80323d0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7fd0c80096a0 tx=0x7fd0c8006b40).stop 2024-02-19T22:03:26.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:26.680+0000 7fd0d32a5700 1 -- 172.21.15.86:0/888629418 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd0cc0f5410 msgr2=0x7fd0cc1056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:03:26.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:26.680+0000 7fd0d32a5700 1 --2- 172.21.15.86:0/888629418 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd0cc0f5410 0x7fd0cc1056e0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0x7fd0bc005010 tx=0x7fd0bc00b040).stop 2024-02-19T22:03:26.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:26.680+0000 7fd0d32a5700 1 -- 172.21.15.86:0/888629418 shutdown_connections 2024-02-19T22:03:26.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:26.680+0000 7fd0d32a5700 1 --2- 172.21.15.86:0/888629418 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fd0b802ff10 0x7fd0b80323d0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:03:26.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:26.680+0000 7fd0d32a5700 1 --2- 172.21.15.86:0/888629418 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd0cc0f5410 0x7fd0cc1056e0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:03:26.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:26.680+0000 7fd0d32a5700 1 -- 172.21.15.86:0/888629418 >> 172.21.15.86:0/888629418 conn(0x7fd0cc0f0b70 msgr2=0x7fd0cc0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:03:26.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:26.680+0000 7fd0d32a5700 1 -- 172.21.15.86:0/888629418 shutdown_connections 2024-02-19T22:03:26.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:26.680+0000 7fd0d32a5700 1 -- 172.21.15.86:0/888629418 wait complete. 2024-02-19T22:03:26.681 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T22:03:27.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:03:26 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:03:25.499462+0000 mgr.smithi053.fyhcdi (mgr.14162) 211 : cluster [DBG] pgmap v199: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:03:28.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:03:27 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:03:26.679263+0000 mon.smithi053 (mon.0) 237 : audit [DBG] from='client.? 172.21.15.86:0/888629418' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:03:28.481 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:03:28.481 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:03:29.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:03:28 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:03:27.499785+0000 mgr.smithi053.fyhcdi (mgr.14162) 212 : cluster [DBG] pgmap v200: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:03:30.471 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:30.470+0000 7f104ebbb700 1 -- 172.21.15.86:0/4025357460 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f10480f5410 msgr2=0x7f10480f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:03:30.472 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:30.470+0000 7f104ebbb700 1 --2- 172.21.15.86:0/4025357460 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f10480f5410 0x7f10480f5830 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 rx=0x7f103c009fb0 tx=0x7f103c009b30).stop 2024-02-19T22:03:30.472 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:30.470+0000 7f104ebbb700 1 -- 172.21.15.86:0/4025357460 shutdown_connections 2024-02-19T22:03:30.472 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:30.470+0000 7f104ebbb700 1 --2- 172.21.15.86:0/4025357460 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f10480f5410 0x7f10480f5830 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:03:30.472 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:30.470+0000 7f104ebbb700 1 -- 172.21.15.86:0/4025357460 >> 172.21.15.86:0/4025357460 conn(0x7f10480f0b70 msgr2=0x7f10480f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:03:30.472 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:30.470+0000 7f104ebbb700 1 -- 172.21.15.86:0/4025357460 shutdown_connections 2024-02-19T22:03:30.472 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:30.470+0000 7f104ebbb700 1 -- 172.21.15.86:0/4025357460 wait complete. 2024-02-19T22:03:30.472 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:30.470+0000 7f104ebbb700 1 Processor -- start 2024-02-19T22:03:30.473 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:30.470+0000 7f104ebbb700 1 -- start start 2024-02-19T22:03:30.473 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:30.471+0000 7f104ebbb700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f10480f5410 0x7f10481094e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:03:30.473 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:30.471+0000 7f104ebbb700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1048060240 con 0x7f10480f5410 2024-02-19T22:03:30.473 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:30.471+0000 7f104c957700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f10480f5410 0x7f10481094e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:03:30.473 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:30.471+0000 7f104c957700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f10480f5410 0x7f10481094e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:35480/0 (socket says 172.21.15.86:35480) 2024-02-19T22:03:30.473 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:30.471+0000 7f104c957700 1 -- 172.21.15.86:0/2580893175 learned_addr learned my addr 172.21.15.86:0/2580893175 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T22:03:30.473 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:30.471+0000 7f104c957700 1 -- 172.21.15.86:0/2580893175 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f103c005040 con 0x7f10480f5410 2024-02-19T22:03:30.473 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:30.471+0000 7f104c957700 1 --2- 172.21.15.86:0/2580893175 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f10480f5410 0x7f10481094e0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7f103c005010 tx=0x7f103c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:03:30.473 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:30.471+0000 7f10457fa700 1 -- 172.21.15.86:0/2580893175 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f103c009320 con 0x7f10480f5410 2024-02-19T22:03:30.473 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:30.472+0000 7f104ebbb700 1 -- 172.21.15.86:0/2580893175 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1048060440 con 0x7f10480f5410 2024-02-19T22:03:30.473 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:30.472+0000 7f10457fa700 1 -- 172.21.15.86:0/2580893175 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f103c013380 con 0x7f10480f5410 2024-02-19T22:03:30.474 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:30.472+0000 7f10457fa700 1 -- 172.21.15.86:0/2580893175 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f103c014e50 con 0x7f10480f5410 2024-02-19T22:03:30.474 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:30.472+0000 7f104ebbb700 1 -- 172.21.15.86:0/2580893175 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1048109b90 con 0x7f10480f5410 2024-02-19T22:03:30.475 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:30.472+0000 7f10457fa700 1 -- 172.21.15.86:0/2580893175 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f103c0095f0 con 0x7f10480f5410 2024-02-19T22:03:30.475 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:30.472+0000 7f10457fa700 1 --2- 172.21.15.86:0/2580893175 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f103802ff10 0x7f10380323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:03:30.475 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:30.472+0000 7f10457fa700 1 -- 172.21.15.86:0/2580893175 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f103c040e90 con 0x7f10480f5410 2024-02-19T22:03:30.475 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:30.472+0000 7f104ebbb700 1 -- 172.21.15.86:0/2580893175 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1048045ad0 con 0x7f10480f5410 2024-02-19T22:03:30.475 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:30.473+0000 7f1047fff700 1 --2- 172.21.15.86:0/2580893175 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f103802ff10 0x7f10380323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:03:30.475 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:30.473+0000 7f1047fff700 1 --2- 172.21.15.86:0/2580893175 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f103802ff10 0x7f10380323d0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f10340096a0 tx=0x7f1034006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:03:30.477 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:30.478+0000 7f10457fa700 1 -- 172.21.15.86:0/2580893175 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f103c014420 con 0x7f10480f5410 2024-02-19T22:03:30.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:30.682+0000 7f104ebbb700 1 -- 172.21.15.86:0/2580893175 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f104810a2f0 con 0x7f10480f5410 2024-02-19T22:03:30.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:30.682+0000 7f10457fa700 1 -- 172.21.15.86:0/2580893175 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f103c010230 con 0x7f10480f5410 2024-02-19T22:03:30.682 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T22:03:30.682 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:03:30.684 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:30.684+0000 7f104ebbb700 1 -- 172.21.15.86:0/2580893175 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f103802ff10 msgr2=0x7f10380323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:03:30.684 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:30.684+0000 7f104ebbb700 1 --2- 172.21.15.86:0/2580893175 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f103802ff10 0x7f10380323d0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f10340096a0 tx=0x7f1034006b40).stop 2024-02-19T22:03:30.684 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:30.684+0000 7f104ebbb700 1 -- 172.21.15.86:0/2580893175 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f10480f5410 msgr2=0x7f10481094e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:03:30.684 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:30.684+0000 7f104ebbb700 1 --2- 172.21.15.86:0/2580893175 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f10480f5410 0x7f10481094e0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7f103c005010 tx=0x7f103c00b040).stop 2024-02-19T22:03:30.684 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:30.684+0000 7f104ebbb700 1 -- 172.21.15.86:0/2580893175 shutdown_connections 2024-02-19T22:03:30.685 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:30.684+0000 7f104ebbb700 1 --2- 172.21.15.86:0/2580893175 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f103802ff10 0x7f10380323d0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:03:30.685 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:30.684+0000 7f104ebbb700 1 --2- 172.21.15.86:0/2580893175 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f10480f5410 0x7f10481094e0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:03:30.685 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:30.684+0000 7f104ebbb700 1 -- 172.21.15.86:0/2580893175 >> 172.21.15.86:0/2580893175 conn(0x7f10480f0b70 msgr2=0x7f10480f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:03:30.685 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:30.684+0000 7f104ebbb700 1 -- 172.21.15.86:0/2580893175 shutdown_connections 2024-02-19T22:03:30.685 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:30.684+0000 7f104ebbb700 1 -- 172.21.15.86:0/2580893175 wait complete. 2024-02-19T22:03:30.685 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T22:03:30.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:03:30 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:03:29.500063+0000 mgr.smithi053.fyhcdi (mgr.14162) 213 : cluster [DBG] pgmap v201: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:03:31.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:03:31 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:03:30.683649+0000 mon.smithi053 (mon.0) 238 : audit [DBG] from='client.? 172.21.15.86:0/2580893175' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:03:32.369 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:03:32.370 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:03:32.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:03:32 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:03:31.500289+0000 mgr.smithi053.fyhcdi (mgr.14162) 214 : cluster [DBG] pgmap v202: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:03:34.298 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:34.290+0000 7f0a8a388700 1 -- 172.21.15.86:0/1168189401 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0a840f62a0 msgr2=0x7f0a840f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:03:34.299 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:34.290+0000 7f0a8a388700 1 --2- 172.21.15.86:0/1168189401 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0a840f62a0 0x7f0a840f66c0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 rx=0x7f0a80009fb0 tx=0x7f0a80009b30).stop 2024-02-19T22:03:34.299 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:34.290+0000 7f0a8a388700 1 -- 172.21.15.86:0/1168189401 shutdown_connections 2024-02-19T22:03:34.299 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:34.290+0000 7f0a8a388700 1 --2- 172.21.15.86:0/1168189401 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0a840f62a0 0x7f0a840f66c0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:03:34.299 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:34.290+0000 7f0a8a388700 1 -- 172.21.15.86:0/1168189401 >> 172.21.15.86:0/1168189401 conn(0x7f0a840f1aa0 msgr2=0x7f0a840f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:03:34.299 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:34.290+0000 7f0a8a388700 1 -- 172.21.15.86:0/1168189401 shutdown_connections 2024-02-19T22:03:34.299 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:34.290+0000 7f0a8a388700 1 -- 172.21.15.86:0/1168189401 wait complete. 2024-02-19T22:03:34.299 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:34.291+0000 7f0a8a388700 1 Processor -- start 2024-02-19T22:03:34.299 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:34.291+0000 7f0a8a388700 1 -- start start 2024-02-19T22:03:34.299 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:34.291+0000 7f0a8a388700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0a840f62a0 0x7f0a8406b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:03:34.299 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:34.291+0000 7f0a8a388700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a84069e20 con 0x7f0a840f62a0 2024-02-19T22:03:34.299 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:34.291+0000 7f0a89386700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0a840f62a0 0x7f0a8406b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:03:34.300 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:34.291+0000 7f0a89386700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0a840f62a0 0x7f0a8406b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:35500/0 (socket says 172.21.15.86:35500) 2024-02-19T22:03:34.300 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:34.291+0000 7f0a89386700 1 -- 172.21.15.86:0/416173613 learned_addr learned my addr 172.21.15.86:0/416173613 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T22:03:34.300 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:34.291+0000 7f0a89386700 1 -- 172.21.15.86:0/416173613 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a80005040 con 0x7f0a840f62a0 2024-02-19T22:03:34.300 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:34.292+0000 7f0a89386700 1 --2- 172.21.15.86:0/416173613 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0a840f62a0 0x7f0a8406b9a0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7f0a80009830 tx=0x7f0a8000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:03:34.300 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:34.292+0000 7f0a79ffb700 1 -- 172.21.15.86:0/416173613 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0a80009320 con 0x7f0a840f62a0 2024-02-19T22:03:34.300 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:34.292+0000 7f0a8a388700 1 -- 172.21.15.86:0/416173613 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a84069fc0 con 0x7f0a840f62a0 2024-02-19T22:03:34.300 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:34.292+0000 7f0a79ffb700 1 -- 172.21.15.86:0/416173613 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0a80013380 con 0x7f0a840f62a0 2024-02-19T22:03:34.300 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:34.292+0000 7f0a79ffb700 1 -- 172.21.15.86:0/416173613 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0a80014420 con 0x7f0a840f62a0 2024-02-19T22:03:34.300 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:34.292+0000 7f0a8a388700 1 -- 172.21.15.86:0/416173613 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a8406a450 con 0x7f0a840f62a0 2024-02-19T22:03:34.300 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:34.293+0000 7f0a79ffb700 1 -- 172.21.15.86:0/416173613 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0a8001e420 con 0x7f0a840f62a0 2024-02-19T22:03:34.300 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:34.293+0000 7f0a8a388700 1 -- 172.21.15.86:0/416173613 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a84059100 con 0x7f0a840f62a0 2024-02-19T22:03:34.300 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:34.293+0000 7f0a79ffb700 1 --2- 172.21.15.86:0/416173613 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f0a7002ff60 0x7f0a70032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:03:34.300 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:34.293+0000 7f0a79ffb700 1 -- 172.21.15.86:0/416173613 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0a80040e20 con 0x7f0a840f62a0 2024-02-19T22:03:34.301 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:34.293+0000 7f0a88b85700 1 --2- 172.21.15.86:0/416173613 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f0a7002ff60 0x7f0a70032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:03:34.301 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:34.293+0000 7f0a88b85700 1 --2- 172.21.15.86:0/416173613 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f0a7002ff60 0x7f0a70032420 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f0a740096a0 tx=0x7f0a74006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:03:34.301 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:34.298+0000 7f0a79ffb700 1 -- 172.21.15.86:0/416173613 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0a80014aa0 con 0x7f0a840f62a0 2024-02-19T22:03:34.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:34.502+0000 7f0a8a388700 1 -- 172.21.15.86:0/416173613 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0a840fbf80 con 0x7f0a840f62a0 2024-02-19T22:03:34.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:34.503+0000 7f0a79ffb700 1 -- 172.21.15.86:0/416173613 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0a8002c2d0 con 0x7f0a840f62a0 2024-02-19T22:03:34.502 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T22:03:34.503 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:03:34.504 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:34.504+0000 7f0a8a388700 1 -- 172.21.15.86:0/416173613 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f0a7002ff60 msgr2=0x7f0a70032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:03:34.504 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:34.504+0000 7f0a8a388700 1 --2- 172.21.15.86:0/416173613 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f0a7002ff60 0x7f0a70032420 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f0a740096a0 tx=0x7f0a74006b40).stop 2024-02-19T22:03:34.504 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:34.504+0000 7f0a8a388700 1 -- 172.21.15.86:0/416173613 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0a840f62a0 msgr2=0x7f0a8406b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:03:34.505 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:34.504+0000 7f0a8a388700 1 --2- 172.21.15.86:0/416173613 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0a840f62a0 0x7f0a8406b9a0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7f0a80009830 tx=0x7f0a8000b040).stop 2024-02-19T22:03:34.505 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:34.504+0000 7f0a8a388700 1 -- 172.21.15.86:0/416173613 shutdown_connections 2024-02-19T22:03:34.505 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:34.504+0000 7f0a8a388700 1 --2- 172.21.15.86:0/416173613 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f0a7002ff60 0x7f0a70032420 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:03:34.505 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:34.504+0000 7f0a8a388700 1 --2- 172.21.15.86:0/416173613 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0a840f62a0 0x7f0a8406b9a0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:03:34.505 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:34.504+0000 7f0a8a388700 1 -- 172.21.15.86:0/416173613 >> 172.21.15.86:0/416173613 conn(0x7f0a840f1aa0 msgr2=0x7f0a840f2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:03:34.505 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:34.504+0000 7f0a8a388700 1 -- 172.21.15.86:0/416173613 shutdown_connections 2024-02-19T22:03:34.505 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:34.504+0000 7f0a8a388700 1 -- 172.21.15.86:0/416173613 wait complete. 2024-02-19T22:03:34.505 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T22:03:34.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:03:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:03:33.500586+0000 mgr.smithi053.fyhcdi (mgr.14162) 215 : cluster [DBG] pgmap v203: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:03:35.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:03:35 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:03:34.503854+0000 mon.smithi053 (mon.0) 239 : audit [DBG] from='client.? 172.21.15.86:0/416173613' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:03:36.114 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:03:36.114 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:03:36.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:03:36 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:03:35.500926+0000 mgr.smithi053.fyhcdi (mgr.14162) 216 : cluster [DBG] pgmap v204: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:03:36.794 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:03:36 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:03:35.603230+0000 mon.smithi053 (mon.0) 240 : audit [INF] from='mgr.14162 172.21.15.53:0/2860358250' entity='mgr.smithi053.fyhcdi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi053.fyhcdi/mirror_snapshot_schedule"}]: dispatch 2024-02-19T22:03:36.794 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:03:36 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:03:35.606291+0000 mon.smithi053 (mon.0) 241 : audit [INF] from='mgr.14162 172.21.15.53:0/2860358250' entity='mgr.smithi053.fyhcdi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi053.fyhcdi/trash_purge_schedule"}]: dispatch 2024-02-19T22:03:38.021 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:38.021+0000 7ff1e16d1700 1 -- 172.21.15.86:0/2629507677 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff1dc0f59d0 msgr2=0x7ff1dc0f5df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:03:38.021 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:38.021+0000 7ff1e16d1700 1 --2- 172.21.15.86:0/2629507677 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff1dc0f59d0 0x7ff1dc0f5df0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 rx=0x7ff1d0009fb0 tx=0x7ff1d0009b30).stop 2024-02-19T22:03:38.021 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:38.021+0000 7ff1e16d1700 1 -- 172.21.15.86:0/2629507677 shutdown_connections 2024-02-19T22:03:38.021 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:38.021+0000 7ff1e16d1700 1 --2- 172.21.15.86:0/2629507677 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff1dc0f59d0 0x7ff1dc0f5df0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:03:38.021 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:38.021+0000 7ff1e16d1700 1 -- 172.21.15.86:0/2629507677 >> 172.21.15.86:0/2629507677 conn(0x7ff1dc0f02e0 msgr2=0x7ff1dc0f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:03:38.021 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:38.021+0000 7ff1e16d1700 1 -- 172.21.15.86:0/2629507677 shutdown_connections 2024-02-19T22:03:38.021 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:38.021+0000 7ff1e16d1700 1 -- 172.21.15.86:0/2629507677 wait complete. 2024-02-19T22:03:38.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:38.022+0000 7ff1e16d1700 1 Processor -- start 2024-02-19T22:03:38.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:38.022+0000 7ff1e16d1700 1 -- start start 2024-02-19T22:03:38.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:38.022+0000 7ff1e16d1700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff1dc0f59d0 0x7ff1dc1067a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:03:38.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:38.022+0000 7ff1e16d1700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1dc106c90 con 0x7ff1dc0f59d0 2024-02-19T22:03:38.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:38.022+0000 7ff1daffd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff1dc0f59d0 0x7ff1dc1067a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:03:38.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:38.022+0000 7ff1daffd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff1dc0f59d0 0x7ff1dc1067a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:35514/0 (socket says 172.21.15.86:35514) 2024-02-19T22:03:38.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:38.022+0000 7ff1daffd700 1 -- 172.21.15.86:0/4257789455 learned_addr learned my addr 172.21.15.86:0/4257789455 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T22:03:38.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:38.023+0000 7ff1daffd700 1 -- 172.21.15.86:0/4257789455 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1d0005040 con 0x7ff1dc0f59d0 2024-02-19T22:03:38.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:38.023+0000 7ff1daffd700 1 --2- 172.21.15.86:0/4257789455 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff1dc0f59d0 0x7ff1dc1067a0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0x7ff1d0005010 tx=0x7ff1d000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:03:38.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:38.023+0000 7ff1c37fe700 1 -- 172.21.15.86:0/4257789455 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff1d0009320 con 0x7ff1dc0f59d0 2024-02-19T22:03:38.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:38.023+0000 7ff1e16d1700 1 -- 172.21.15.86:0/4257789455 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1dc106e90 con 0x7ff1dc0f59d0 2024-02-19T22:03:38.024 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:38.023+0000 7ff1c37fe700 1 -- 172.21.15.86:0/4257789455 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff1d0013380 con 0x7ff1dc0f59d0 2024-02-19T22:03:38.024 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:38.023+0000 7ff1c37fe700 1 -- 172.21.15.86:0/4257789455 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff1d0014e50 con 0x7ff1dc0f59d0 2024-02-19T22:03:38.024 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:38.023+0000 7ff1e16d1700 1 -- 172.21.15.86:0/4257789455 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1dc107320 con 0x7ff1dc0f59d0 2024-02-19T22:03:38.025 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:38.024+0000 7ff1c37fe700 1 -- 172.21.15.86:0/4257789455 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff1d0014420 con 0x7ff1dc0f59d0 2024-02-19T22:03:38.025 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:38.024+0000 7ff1e16d1700 1 -- 172.21.15.86:0/4257789455 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1dc045ad0 con 0x7ff1dc0f59d0 2024-02-19T22:03:38.025 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:38.024+0000 7ff1c37fe700 1 --2- 172.21.15.86:0/4257789455 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7ff1c402ff10 0x7ff1c40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:03:38.025 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:38.024+0000 7ff1c37fe700 1 -- 172.21.15.86:0/4257789455 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff1d0040f70 con 0x7ff1dc0f59d0 2024-02-19T22:03:38.025 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:38.024+0000 7ff1da7fc700 1 --2- 172.21.15.86:0/4257789455 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7ff1c402ff10 0x7ff1c40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:03:38.025 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:38.025+0000 7ff1da7fc700 1 --2- 172.21.15.86:0/4257789455 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7ff1c402ff10 0x7ff1c40323d0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7ff1cc0096a0 tx=0x7ff1cc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:03:38.028 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:38.029+0000 7ff1c37fe700 1 -- 172.21.15.86:0/4257789455 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff1d00095f0 con 0x7ff1dc0f59d0 2024-02-19T22:03:38.232 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:38.233+0000 7ff1e16d1700 1 -- 172.21.15.86:0/4257789455 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff1dc1081d0 con 0x7ff1dc0f59d0 2024-02-19T22:03:38.233 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:38.233+0000 7ff1c37fe700 1 -- 172.21.15.86:0/4257789455 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff1d001db70 con 0x7ff1dc0f59d0 2024-02-19T22:03:38.233 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T22:03:38.233 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:03:38.234 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:38.235+0000 7ff1e16d1700 1 -- 172.21.15.86:0/4257789455 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7ff1c402ff10 msgr2=0x7ff1c40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:03:38.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:38.235+0000 7ff1e16d1700 1 --2- 172.21.15.86:0/4257789455 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7ff1c402ff10 0x7ff1c40323d0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7ff1cc0096a0 tx=0x7ff1cc006b40).stop 2024-02-19T22:03:38.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:38.235+0000 7ff1e16d1700 1 -- 172.21.15.86:0/4257789455 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff1dc0f59d0 msgr2=0x7ff1dc1067a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:03:38.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:38.235+0000 7ff1e16d1700 1 --2- 172.21.15.86:0/4257789455 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff1dc0f59d0 0x7ff1dc1067a0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0x7ff1d0005010 tx=0x7ff1d000b040).stop 2024-02-19T22:03:38.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:38.235+0000 7ff1e16d1700 1 -- 172.21.15.86:0/4257789455 shutdown_connections 2024-02-19T22:03:38.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:38.235+0000 7ff1e16d1700 1 --2- 172.21.15.86:0/4257789455 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7ff1c402ff10 0x7ff1c40323d0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:03:38.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:38.235+0000 7ff1e16d1700 1 --2- 172.21.15.86:0/4257789455 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff1dc0f59d0 0x7ff1dc1067a0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:03:38.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:38.235+0000 7ff1e16d1700 1 -- 172.21.15.86:0/4257789455 >> 172.21.15.86:0/4257789455 conn(0x7ff1dc0f02e0 msgr2=0x7ff1dc0f4f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:03:38.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:38.235+0000 7ff1e16d1700 1 -- 172.21.15.86:0/4257789455 shutdown_connections 2024-02-19T22:03:38.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:38.235+0000 7ff1e16d1700 1 -- 172.21.15.86:0/4257789455 wait complete. 2024-02-19T22:03:38.236 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T22:03:38.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:03:38 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:03:37.501282+0000 mgr.smithi053.fyhcdi (mgr.14162) 217 : cluster [DBG] pgmap v205: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:03:38.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:03:38 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:03:38.234598+0000 mon.smithi053 (mon.0) 242 : audit [DBG] from='client.? 172.21.15.86:0/4257789455' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:03:39.833 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:03:39.833 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:03:40.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:03:40 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:03:39.501602+0000 mgr.smithi053.fyhcdi (mgr.14162) 218 : cluster [DBG] pgmap v206: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:03:41.849 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:41.849+0000 7fc503318700 1 -- 172.21.15.86:0/2033233501 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc4fc0f62a0 msgr2=0x7fc4fc0f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:03:41.849 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:41.849+0000 7fc503318700 1 --2- 172.21.15.86:0/2033233501 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc4fc0f62a0 0x7fc4fc0f66c0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 rx=0x7fc4f8009fb0 tx=0x7fc4f8009b30).stop 2024-02-19T22:03:41.849 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:41.849+0000 7fc503318700 1 -- 172.21.15.86:0/2033233501 shutdown_connections 2024-02-19T22:03:41.849 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:41.849+0000 7fc503318700 1 --2- 172.21.15.86:0/2033233501 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc4fc0f62a0 0x7fc4fc0f66c0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:03:41.849 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:41.849+0000 7fc503318700 1 -- 172.21.15.86:0/2033233501 >> 172.21.15.86:0/2033233501 conn(0x7fc4fc0f1aa0 msgr2=0x7fc4fc0f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:03:41.849 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:41.849+0000 7fc503318700 1 -- 172.21.15.86:0/2033233501 shutdown_connections 2024-02-19T22:03:41.849 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:41.849+0000 7fc503318700 1 -- 172.21.15.86:0/2033233501 wait complete. 2024-02-19T22:03:41.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:41.850+0000 7fc503318700 1 Processor -- start 2024-02-19T22:03:41.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:41.850+0000 7fc503318700 1 -- start start 2024-02-19T22:03:41.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:41.850+0000 7fc503318700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc4fc0f62a0 0x7fc4fc1065d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:03:41.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:41.850+0000 7fc503318700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4fc106ac0 con 0x7fc4fc0f62a0 2024-02-19T22:03:41.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:41.850+0000 7fc502316700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc4fc0f62a0 0x7fc4fc1065d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:03:41.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:41.850+0000 7fc502316700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc4fc0f62a0 0x7fc4fc1065d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:34486/0 (socket says 172.21.15.86:34486) 2024-02-19T22:03:41.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:41.850+0000 7fc502316700 1 -- 172.21.15.86:0/3743796924 learned_addr learned my addr 172.21.15.86:0/3743796924 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T22:03:41.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:41.850+0000 7fc502316700 1 -- 172.21.15.86:0/3743796924 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4f8005040 con 0x7fc4fc0f62a0 2024-02-19T22:03:41.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:41.851+0000 7fc502316700 1 --2- 172.21.15.86:0/3743796924 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc4fc0f62a0 0x7fc4fc1065d0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0x7fc4f80089f0 tx=0x7fc4f800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:03:41.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:41.851+0000 7fc4f2ffd700 1 -- 172.21.15.86:0/3743796924 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc4f8009320 con 0x7fc4fc0f62a0 2024-02-19T22:03:41.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:41.851+0000 7fc503318700 1 -- 172.21.15.86:0/3743796924 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4fc106cc0 con 0x7fc4fc0f62a0 2024-02-19T22:03:41.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:41.851+0000 7fc503318700 1 -- 172.21.15.86:0/3743796924 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4fc107150 con 0x7fc4fc0f62a0 2024-02-19T22:03:41.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:41.851+0000 7fc4f2ffd700 1 -- 172.21.15.86:0/3743796924 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc4f8013380 con 0x7fc4fc0f62a0 2024-02-19T22:03:41.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:41.851+0000 7fc4f2ffd700 1 -- 172.21.15.86:0/3743796924 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc4f8014dc0 con 0x7fc4fc0f62a0 2024-02-19T22:03:41.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:41.852+0000 7fc503318700 1 -- 172.21.15.86:0/3743796924 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4e4004fa0 con 0x7fc4fc0f62a0 2024-02-19T22:03:41.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:41.852+0000 7fc4f2ffd700 1 -- 172.21.15.86:0/3743796924 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc4f8014420 con 0x7fc4fc0f62a0 2024-02-19T22:03:41.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:41.852+0000 7fc4f2ffd700 1 --2- 172.21.15.86:0/3743796924 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fc4e802ff60 0x7fc4e8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:03:41.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:41.852+0000 7fc4f2ffd700 1 -- 172.21.15.86:0/3743796924 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc4f8040eb0 con 0x7fc4fc0f62a0 2024-02-19T22:03:41.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:41.852+0000 7fc501b15700 1 --2- 172.21.15.86:0/3743796924 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fc4e802ff60 0x7fc4e8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:03:41.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:41.853+0000 7fc501b15700 1 --2- 172.21.15.86:0/3743796924 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fc4e802ff60 0x7fc4e8032420 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7fc4ec0096a0 tx=0x7fc4ec006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:03:41.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:41.857+0000 7fc4f2ffd700 1 -- 172.21.15.86:0/3743796924 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc4f8009680 con 0x7fc4fc0f62a0 2024-02-19T22:03:42.062 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:42.062+0000 7fc503318700 1 -- 172.21.15.86:0/3743796924 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc4e4005670 con 0x7fc4fc0f62a0 2024-02-19T22:03:42.062 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:42.062+0000 7fc4f2ffd700 1 -- 172.21.15.86:0/3743796924 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc4f801d9d0 con 0x7fc4fc0f62a0 2024-02-19T22:03:42.063 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T22:03:42.063 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:03:42.064 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:42.064+0000 7fc503318700 1 -- 172.21.15.86:0/3743796924 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fc4e802ff60 msgr2=0x7fc4e8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:03:42.065 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:42.064+0000 7fc503318700 1 --2- 172.21.15.86:0/3743796924 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fc4e802ff60 0x7fc4e8032420 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7fc4ec0096a0 tx=0x7fc4ec006b40).stop 2024-02-19T22:03:42.065 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:42.064+0000 7fc503318700 1 -- 172.21.15.86:0/3743796924 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc4fc0f62a0 msgr2=0x7fc4fc1065d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:03:42.065 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:42.064+0000 7fc503318700 1 --2- 172.21.15.86:0/3743796924 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc4fc0f62a0 0x7fc4fc1065d0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0x7fc4f80089f0 tx=0x7fc4f800b040).stop 2024-02-19T22:03:42.065 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:42.064+0000 7fc503318700 1 -- 172.21.15.86:0/3743796924 shutdown_connections 2024-02-19T22:03:42.065 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:42.064+0000 7fc503318700 1 --2- 172.21.15.86:0/3743796924 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fc4e802ff60 0x7fc4e8032420 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:03:42.065 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:42.064+0000 7fc503318700 1 --2- 172.21.15.86:0/3743796924 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc4fc0f62a0 0x7fc4fc1065d0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:03:42.066 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:42.064+0000 7fc503318700 1 -- 172.21.15.86:0/3743796924 >> 172.21.15.86:0/3743796924 conn(0x7fc4fc0f1aa0 msgr2=0x7fc4fc0f2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:03:42.066 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:42.064+0000 7fc503318700 1 -- 172.21.15.86:0/3743796924 shutdown_connections 2024-02-19T22:03:42.066 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:42.064+0000 7fc503318700 1 -- 172.21.15.86:0/3743796924 wait complete. 2024-02-19T22:03:42.066 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T22:03:43.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:03:42 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:03:41.501888+0000 mgr.smithi053.fyhcdi (mgr.14162) 219 : cluster [DBG] pgmap v207: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:03:43.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:03:42 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:03:42.063462+0000 mon.smithi053 (mon.0) 243 : audit [DBG] from='client.? 172.21.15.86:0/3743796924' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:03:43.672 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:03:43.673 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:03:45.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:03:44 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:03:43.502233+0000 mgr.smithi053.fyhcdi (mgr.14162) 220 : cluster [DBG] pgmap v208: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:03:45.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:45.549+0000 7f9b1f663700 1 -- 172.21.15.86:0/4026588983 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9b180f5410 msgr2=0x7f9b180f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:03:45.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:45.549+0000 7f9b1f663700 1 --2- 172.21.15.86:0/4026588983 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9b180f5410 0x7f9b180f5830 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 rx=0x7f9b14009fb0 tx=0x7f9b14009b30).stop 2024-02-19T22:03:45.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:45.549+0000 7f9b1f663700 1 -- 172.21.15.86:0/4026588983 shutdown_connections 2024-02-19T22:03:45.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:45.549+0000 7f9b1f663700 1 --2- 172.21.15.86:0/4026588983 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9b180f5410 0x7f9b180f5830 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:03:45.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:45.549+0000 7f9b1f663700 1 -- 172.21.15.86:0/4026588983 >> 172.21.15.86:0/4026588983 conn(0x7f9b180f0b70 msgr2=0x7f9b180f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:03:45.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:45.549+0000 7f9b1f663700 1 -- 172.21.15.86:0/4026588983 shutdown_connections 2024-02-19T22:03:45.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:45.549+0000 7f9b1f663700 1 -- 172.21.15.86:0/4026588983 wait complete. 2024-02-19T22:03:45.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:45.550+0000 7f9b1f663700 1 Processor -- start 2024-02-19T22:03:45.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:45.550+0000 7f9b1f663700 1 -- start start 2024-02-19T22:03:45.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:45.550+0000 7f9b1f663700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9b180f5410 0x7f9b181056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:03:45.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:45.550+0000 7f9b1f663700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b18105bd0 con 0x7f9b180f5410 2024-02-19T22:03:45.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:45.550+0000 7f9b1d3ff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9b180f5410 0x7f9b181056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:03:45.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:45.550+0000 7f9b1d3ff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9b180f5410 0x7f9b181056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:34512/0 (socket says 172.21.15.86:34512) 2024-02-19T22:03:45.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:45.550+0000 7f9b1d3ff700 1 -- 172.21.15.86:0/1673796657 learned_addr learned my addr 172.21.15.86:0/1673796657 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T22:03:45.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:45.550+0000 7f9b1d3ff700 1 -- 172.21.15.86:0/1673796657 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b14005040 con 0x7f9b180f5410 2024-02-19T22:03:45.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:45.551+0000 7f9b1d3ff700 1 --2- 172.21.15.86:0/1673796657 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9b180f5410 0x7f9b181056e0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7f9b140089f0 tx=0x7f9b1400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:03:45.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:45.551+0000 7f9b09ffb700 1 -- 172.21.15.86:0/1673796657 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9b14009320 con 0x7f9b180f5410 2024-02-19T22:03:45.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:45.551+0000 7f9b1f663700 1 -- 172.21.15.86:0/1673796657 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b18105dd0 con 0x7f9b180f5410 2024-02-19T22:03:45.560 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:45.551+0000 7f9b09ffb700 1 -- 172.21.15.86:0/1673796657 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9b14013380 con 0x7f9b180f5410 2024-02-19T22:03:45.560 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:45.551+0000 7f9b1f663700 1 -- 172.21.15.86:0/1673796657 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b18106260 con 0x7f9b180f5410 2024-02-19T22:03:45.560 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:45.551+0000 7f9b09ffb700 1 -- 172.21.15.86:0/1673796657 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9b1401ed60 con 0x7f9b180f5410 2024-02-19T22:03:45.560 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:45.552+0000 7f9b09ffb700 1 -- 172.21.15.86:0/1673796657 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9b1401e420 con 0x7f9b180f5410 2024-02-19T22:03:45.560 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:45.552+0000 7f9b1f663700 1 -- 172.21.15.86:0/1673796657 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b18045ad0 con 0x7f9b180f5410 2024-02-19T22:03:45.560 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:45.552+0000 7f9b09ffb700 1 --2- 172.21.15.86:0/1673796657 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f9b0402ff60 0x7f9b04032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:03:45.560 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:45.552+0000 7f9b09ffb700 1 -- 172.21.15.86:0/1673796657 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9b14041110 con 0x7f9b180f5410 2024-02-19T22:03:45.560 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:45.552+0000 7f9b1cbfe700 1 --2- 172.21.15.86:0/1673796657 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f9b0402ff60 0x7f9b04032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:03:45.560 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:45.552+0000 7f9b1cbfe700 1 --2- 172.21.15.86:0/1673796657 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f9b0402ff60 0x7f9b04032420 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f9b0c0096a0 tx=0x7f9b0c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:03:45.560 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:45.557+0000 7f9b09ffb700 1 -- 172.21.15.86:0/1673796657 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9b140095f0 con 0x7f9b180f5410 2024-02-19T22:03:45.760 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:45.760+0000 7f9b1f663700 1 -- 172.21.15.86:0/1673796657 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9b181070b0 con 0x7f9b180f5410 2024-02-19T22:03:45.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:45.761+0000 7f9b09ffb700 1 -- 172.21.15.86:0/1673796657 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9b1401cb10 con 0x7f9b180f5410 2024-02-19T22:03:45.761 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T22:03:45.761 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:03:45.763 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:45.763+0000 7f9b1f663700 1 -- 172.21.15.86:0/1673796657 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f9b0402ff60 msgr2=0x7f9b04032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:03:45.763 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:45.763+0000 7f9b1f663700 1 --2- 172.21.15.86:0/1673796657 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f9b0402ff60 0x7f9b04032420 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f9b0c0096a0 tx=0x7f9b0c006b40).stop 2024-02-19T22:03:45.763 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:45.763+0000 7f9b1f663700 1 -- 172.21.15.86:0/1673796657 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9b180f5410 msgr2=0x7f9b181056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:03:45.763 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:45.763+0000 7f9b1f663700 1 --2- 172.21.15.86:0/1673796657 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9b180f5410 0x7f9b181056e0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7f9b140089f0 tx=0x7f9b1400b040).stop 2024-02-19T22:03:45.763 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:45.763+0000 7f9b1f663700 1 -- 172.21.15.86:0/1673796657 shutdown_connections 2024-02-19T22:03:45.763 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:45.763+0000 7f9b1f663700 1 --2- 172.21.15.86:0/1673796657 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f9b0402ff60 0x7f9b04032420 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:03:45.763 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:45.763+0000 7f9b1f663700 1 --2- 172.21.15.86:0/1673796657 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9b180f5410 0x7f9b181056e0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:03:45.763 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:45.763+0000 7f9b1f663700 1 -- 172.21.15.86:0/1673796657 >> 172.21.15.86:0/1673796657 conn(0x7f9b180f0b70 msgr2=0x7f9b180f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:03:45.763 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:45.763+0000 7f9b1f663700 1 -- 172.21.15.86:0/1673796657 shutdown_connections 2024-02-19T22:03:45.763 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:45.763+0000 7f9b1f663700 1 -- 172.21.15.86:0/1673796657 wait complete. 2024-02-19T22:03:45.764 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T22:03:47.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:03:46 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:03:45.502619+0000 mgr.smithi053.fyhcdi (mgr.14162) 221 : cluster [DBG] pgmap v209: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:03:47.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:03:46 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:03:45.762460+0000 mon.smithi053 (mon.0) 244 : audit [DBG] from='client.? 172.21.15.86:0/1673796657' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:03:47.395 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:03:47.396 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:03:49.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:03:48 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:03:47.502939+0000 mgr.smithi053.fyhcdi (mgr.14162) 222 : cluster [DBG] pgmap v210: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:03:49.341 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:49.340+0000 7f3a70758700 1 -- 172.21.15.86:0/3485908171 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3a680f6510 msgr2=0x7f3a680f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:03:49.341 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:49.340+0000 7f3a70758700 1 --2- 172.21.15.86:0/3485908171 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3a680f6510 0x7f3a680f6930 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 rx=0x7f3a64009fb0 tx=0x7f3a64009b30).stop 2024-02-19T22:03:49.341 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:49.340+0000 7f3a70758700 1 -- 172.21.15.86:0/3485908171 shutdown_connections 2024-02-19T22:03:49.342 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:49.340+0000 7f3a70758700 1 --2- 172.21.15.86:0/3485908171 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3a680f6510 0x7f3a680f6930 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:03:49.342 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:49.340+0000 7f3a70758700 1 -- 172.21.15.86:0/3485908171 >> 172.21.15.86:0/3485908171 conn(0x7f3a680f1c70 msgr2=0x7f3a680f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:03:49.342 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:49.341+0000 7f3a70758700 1 -- 172.21.15.86:0/3485908171 shutdown_connections 2024-02-19T22:03:49.342 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:49.341+0000 7f3a70758700 1 -- 172.21.15.86:0/3485908171 wait complete. 2024-02-19T22:03:49.342 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:49.341+0000 7f3a70758700 1 Processor -- start 2024-02-19T22:03:49.342 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:49.341+0000 7f3a70758700 1 -- start start 2024-02-19T22:03:49.342 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:49.341+0000 7f3a70758700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3a680f6510 0x7f3a68106810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:03:49.342 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:49.341+0000 7f3a70758700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a68106d00 con 0x7f3a680f6510 2024-02-19T22:03:49.342 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:49.341+0000 7f3a6e4f4700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3a680f6510 0x7f3a68106810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:03:49.342 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:49.341+0000 7f3a6e4f4700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3a680f6510 0x7f3a68106810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:36774/0 (socket says 172.21.15.86:36774) 2024-02-19T22:03:49.342 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:49.341+0000 7f3a6e4f4700 1 -- 172.21.15.86:0/3302563282 learned_addr learned my addr 172.21.15.86:0/3302563282 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T22:03:49.342 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:49.342+0000 7f3a6e4f4700 1 -- 172.21.15.86:0/3302563282 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a64005040 con 0x7f3a680f6510 2024-02-19T22:03:49.342 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:49.342+0000 7f3a6e4f4700 1 --2- 172.21.15.86:0/3302563282 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3a680f6510 0x7f3a68106810 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7f3a64005010 tx=0x7f3a6400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:03:49.344 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:49.342+0000 7f3a5effd700 1 -- 172.21.15.86:0/3302563282 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3a64009320 con 0x7f3a680f6510 2024-02-19T22:03:49.344 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:49.342+0000 7f3a70758700 1 -- 172.21.15.86:0/3302563282 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a68106f00 con 0x7f3a680f6510 2024-02-19T22:03:49.344 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:49.342+0000 7f3a5effd700 1 -- 172.21.15.86:0/3302563282 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3a64013380 con 0x7f3a680f6510 2024-02-19T22:03:49.344 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:49.342+0000 7f3a5effd700 1 -- 172.21.15.86:0/3302563282 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3a64014e50 con 0x7f3a680f6510 2024-02-19T22:03:49.344 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:49.342+0000 7f3a70758700 1 -- 172.21.15.86:0/3302563282 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a68107390 con 0x7f3a680f6510 2024-02-19T22:03:49.344 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:49.343+0000 7f3a5effd700 1 -- 172.21.15.86:0/3302563282 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3a640263a0 con 0x7f3a680f6510 2024-02-19T22:03:49.344 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:49.343+0000 7f3a5effd700 1 --2- 172.21.15.86:0/3302563282 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f3a5402ff10 0x7f3a540323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:03:49.344 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:49.343+0000 7f3a5effd700 1 -- 172.21.15.86:0/3302563282 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3a64040ec0 con 0x7f3a680f6510 2024-02-19T22:03:49.344 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:49.343+0000 7f3a70758700 1 -- 172.21.15.86:0/3302563282 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a68045ad0 con 0x7f3a680f6510 2024-02-19T22:03:49.345 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:49.343+0000 7f3a6dcf3700 1 --2- 172.21.15.86:0/3302563282 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f3a5402ff10 0x7f3a540323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:03:49.345 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:49.344+0000 7f3a6dcf3700 1 --2- 172.21.15.86:0/3302563282 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f3a5402ff10 0x7f3a540323d0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f3a580096a0 tx=0x7f3a58006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:03:49.348 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:49.349+0000 7f3a5effd700 1 -- 172.21.15.86:0/3302563282 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3a64012920 con 0x7f3a680f6510 2024-02-19T22:03:49.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:49.553+0000 7f3a70758700 1 -- 172.21.15.86:0/3302563282 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3a680fc190 con 0x7f3a680f6510 2024-02-19T22:03:49.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:49.554+0000 7f3a5effd700 1 -- 172.21.15.86:0/3302563282 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3a6402b020 con 0x7f3a680f6510 2024-02-19T22:03:49.554 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T22:03:49.554 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:03:49.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:49.556+0000 7f3a70758700 1 -- 172.21.15.86:0/3302563282 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f3a5402ff10 msgr2=0x7f3a540323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:03:49.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:49.556+0000 7f3a70758700 1 --2- 172.21.15.86:0/3302563282 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f3a5402ff10 0x7f3a540323d0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f3a580096a0 tx=0x7f3a58006b40).stop 2024-02-19T22:03:49.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:49.556+0000 7f3a70758700 1 -- 172.21.15.86:0/3302563282 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3a680f6510 msgr2=0x7f3a68106810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:03:49.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:49.556+0000 7f3a70758700 1 --2- 172.21.15.86:0/3302563282 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3a680f6510 0x7f3a68106810 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7f3a64005010 tx=0x7f3a6400b040).stop 2024-02-19T22:03:49.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:49.556+0000 7f3a70758700 1 -- 172.21.15.86:0/3302563282 shutdown_connections 2024-02-19T22:03:49.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:49.556+0000 7f3a70758700 1 --2- 172.21.15.86:0/3302563282 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f3a5402ff10 0x7f3a540323d0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:03:49.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:49.556+0000 7f3a70758700 1 --2- 172.21.15.86:0/3302563282 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3a680f6510 0x7f3a68106810 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:03:49.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:49.556+0000 7f3a70758700 1 -- 172.21.15.86:0/3302563282 >> 172.21.15.86:0/3302563282 conn(0x7f3a680f1c70 msgr2=0x7f3a680f26f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:03:49.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:49.556+0000 7f3a70758700 1 -- 172.21.15.86:0/3302563282 shutdown_connections 2024-02-19T22:03:49.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:49.556+0000 7f3a70758700 1 -- 172.21.15.86:0/3302563282 wait complete. 2024-02-19T22:03:49.556 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T22:03:50.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:03:50 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:03:49.503182+0000 mgr.smithi053.fyhcdi (mgr.14162) 223 : cluster [DBG] pgmap v211: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:03:50.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:03:50 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:03:49.555436+0000 mon.smithi053 (mon.0) 245 : audit [DBG] from='client.? 172.21.15.86:0/3302563282' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:03:51.321 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:03:51.321 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:03:53.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:03:52 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:03:51.503528+0000 mgr.smithi053.fyhcdi (mgr.14162) 224 : cluster [DBG] pgmap v212: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:03:53.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:53.338+0000 7f24311ec700 1 -- 172.21.15.86:0/3900693160 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f242c060220 msgr2=0x7f242c0fb550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:03:53.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:53.338+0000 7f24311ec700 1 --2- 172.21.15.86:0/3900693160 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f242c060220 0x7f242c0fb550 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 rx=0x7f241c009fb0 tx=0x7f241c009b30).stop 2024-02-19T22:03:53.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:53.338+0000 7f24311ec700 1 -- 172.21.15.86:0/3900693160 shutdown_connections 2024-02-19T22:03:53.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:53.338+0000 7f24311ec700 1 --2- 172.21.15.86:0/3900693160 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f242c060220 0x7f242c0fb550 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:03:53.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:53.338+0000 7f24311ec700 1 -- 172.21.15.86:0/3900693160 >> 172.21.15.86:0/3900693160 conn(0x7f242c0f0b70 msgr2=0x7f242c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:03:53.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:53.338+0000 7f24311ec700 1 -- 172.21.15.86:0/3900693160 shutdown_connections 2024-02-19T22:03:53.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:53.338+0000 7f24311ec700 1 -- 172.21.15.86:0/3900693160 wait complete. 2024-02-19T22:03:53.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:53.338+0000 7f24311ec700 1 Processor -- start 2024-02-19T22:03:53.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:53.338+0000 7f24311ec700 1 -- start start 2024-02-19T22:03:53.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:53.339+0000 7f24311ec700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f242c060220 0x7f242c1089a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:03:53.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:53.339+0000 7f24311ec700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f242c108e90 con 0x7f242c060220 2024-02-19T22:03:53.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:53.339+0000 7f242ad9d700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f242c060220 0x7f242c1089a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:03:53.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:53.339+0000 7f242ad9d700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f242c060220 0x7f242c1089a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:36800/0 (socket says 172.21.15.86:36800) 2024-02-19T22:03:53.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:53.339+0000 7f242ad9d700 1 -- 172.21.15.86:0/4130343370 learned_addr learned my addr 172.21.15.86:0/4130343370 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T22:03:53.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:53.339+0000 7f242ad9d700 1 -- 172.21.15.86:0/4130343370 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f241c005040 con 0x7f242c060220 2024-02-19T22:03:53.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:53.339+0000 7f242ad9d700 1 --2- 172.21.15.86:0/4130343370 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f242c060220 0x7f242c1089a0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7f241c0089f0 tx=0x7f241c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:03:53.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:53.339+0000 7f24137fe700 1 -- 172.21.15.86:0/4130343370 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f241c009320 con 0x7f242c060220 2024-02-19T22:03:53.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:53.339+0000 7f24137fe700 1 -- 172.21.15.86:0/4130343370 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f241c013380 con 0x7f242c060220 2024-02-19T22:03:53.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:53.339+0000 7f24311ec700 1 -- 172.21.15.86:0/4130343370 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f242c109090 con 0x7f242c060220 2024-02-19T22:03:53.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:53.340+0000 7f24137fe700 1 -- 172.21.15.86:0/4130343370 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f241c014e20 con 0x7f242c060220 2024-02-19T22:03:53.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:53.340+0000 7f24311ec700 1 -- 172.21.15.86:0/4130343370 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f242c109520 con 0x7f242c060220 2024-02-19T22:03:53.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:53.340+0000 7f24137fe700 1 -- 172.21.15.86:0/4130343370 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f241c014420 con 0x7f242c060220 2024-02-19T22:03:53.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:53.340+0000 7f24311ec700 1 -- 172.21.15.86:0/4130343370 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f242c045ad0 con 0x7f242c060220 2024-02-19T22:03:53.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:53.341+0000 7f24137fe700 1 --2- 172.21.15.86:0/4130343370 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f241402ff60 0x7f2414032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:03:53.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:53.341+0000 7f24137fe700 1 -- 172.21.15.86:0/4130343370 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f241c040f00 con 0x7f242c060220 2024-02-19T22:03:53.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:53.341+0000 7f242a59c700 1 --2- 172.21.15.86:0/4130343370 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f241402ff60 0x7f2414032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:03:53.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:53.341+0000 7f242a59c700 1 --2- 172.21.15.86:0/4130343370 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f241402ff60 0x7f2414032420 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f24200096a0 tx=0x7f2420006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:03:53.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:53.345+0000 7f24137fe700 1 -- 172.21.15.86:0/4130343370 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f241c0095f0 con 0x7f242c060220 2024-02-19T22:03:53.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:53.553+0000 7f24311ec700 1 -- 172.21.15.86:0/4130343370 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f242c10a3d0 con 0x7f242c060220 2024-02-19T22:03:53.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:53.553+0000 7f24137fe700 1 -- 172.21.15.86:0/4130343370 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f241c01db70 con 0x7f242c060220 2024-02-19T22:03:53.552 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T22:03:53.552 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:03:53.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:53.555+0000 7f24311ec700 1 -- 172.21.15.86:0/4130343370 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f241402ff60 msgr2=0x7f2414032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:03:53.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:53.555+0000 7f24311ec700 1 --2- 172.21.15.86:0/4130343370 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f241402ff60 0x7f2414032420 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f24200096a0 tx=0x7f2420006b40).stop 2024-02-19T22:03:53.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:53.555+0000 7f24311ec700 1 -- 172.21.15.86:0/4130343370 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f242c060220 msgr2=0x7f242c1089a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:03:53.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:53.555+0000 7f24311ec700 1 --2- 172.21.15.86:0/4130343370 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f242c060220 0x7f242c1089a0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7f241c0089f0 tx=0x7f241c00b040).stop 2024-02-19T22:03:53.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:53.555+0000 7f24311ec700 1 -- 172.21.15.86:0/4130343370 shutdown_connections 2024-02-19T22:03:53.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:53.555+0000 7f24311ec700 1 --2- 172.21.15.86:0/4130343370 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f241402ff60 0x7f2414032420 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:03:53.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:53.555+0000 7f24311ec700 1 --2- 172.21.15.86:0/4130343370 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f242c060220 0x7f242c1089a0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:03:53.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:53.555+0000 7f24311ec700 1 -- 172.21.15.86:0/4130343370 >> 172.21.15.86:0/4130343370 conn(0x7f242c0f0b70 msgr2=0x7f242c0f1600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:03:53.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:53.555+0000 7f24311ec700 1 -- 172.21.15.86:0/4130343370 shutdown_connections 2024-02-19T22:03:53.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:53.555+0000 7f24311ec700 1 -- 172.21.15.86:0/4130343370 wait complete. 2024-02-19T22:03:53.555 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T22:03:54.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:03:53 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:03:53.553613+0000 mon.smithi053 (mon.0) 246 : audit [DBG] from='client.? 172.21.15.86:0/4130343370' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:03:55.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:03:54 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:03:53.503818+0000 mgr.smithi053.fyhcdi (mgr.14162) 225 : cluster [DBG] pgmap v213: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:03:55.120 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:03:55.120 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:03:57.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:03:56 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:03:55.504243+0000 mgr.smithi053.fyhcdi (mgr.14162) 226 : cluster [DBG] pgmap v214: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:03:57.326 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:57.327+0000 7f814316f700 1 -- 172.21.15.86:0/933089702 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f813c0f5410 msgr2=0x7f813c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:03:57.326 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:57.327+0000 7f814316f700 1 --2- 172.21.15.86:0/933089702 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f813c0f5410 0x7f813c0f5830 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 rx=0x7f8130009fb0 tx=0x7f8130009b30).stop 2024-02-19T22:03:57.326 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:57.327+0000 7f814316f700 1 -- 172.21.15.86:0/933089702 shutdown_connections 2024-02-19T22:03:57.326 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:57.327+0000 7f814316f700 1 --2- 172.21.15.86:0/933089702 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f813c0f5410 0x7f813c0f5830 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:03:57.327 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:57.327+0000 7f814316f700 1 -- 172.21.15.86:0/933089702 >> 172.21.15.86:0/933089702 conn(0x7f813c0f0b70 msgr2=0x7f813c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:03:57.327 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:57.327+0000 7f814316f700 1 -- 172.21.15.86:0/933089702 shutdown_connections 2024-02-19T22:03:57.327 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:57.328+0000 7f814316f700 1 -- 172.21.15.86:0/933089702 wait complete. 2024-02-19T22:03:57.328 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:57.328+0000 7f814316f700 1 Processor -- start 2024-02-19T22:03:57.328 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:57.328+0000 7f814316f700 1 -- start start 2024-02-19T22:03:57.328 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:57.328+0000 7f814316f700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f813c0f5410 0x7f813c1056c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:03:57.328 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:57.328+0000 7f814316f700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f813c105bb0 con 0x7f813c0f5410 2024-02-19T22:03:57.328 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:57.329+0000 7f8140f0b700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f813c0f5410 0x7f813c1056c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:03:57.328 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:57.329+0000 7f8140f0b700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f813c0f5410 0x7f813c1056c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:36822/0 (socket says 172.21.15.86:36822) 2024-02-19T22:03:57.328 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:57.329+0000 7f8140f0b700 1 -- 172.21.15.86:0/2348720806 learned_addr learned my addr 172.21.15.86:0/2348720806 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T22:03:57.329 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:57.329+0000 7f8140f0b700 1 -- 172.21.15.86:0/2348720806 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8130005040 con 0x7f813c0f5410 2024-02-19T22:03:57.329 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:57.329+0000 7f8140f0b700 1 --2- 172.21.15.86:0/2348720806 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f813c0f5410 0x7f813c1056c0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7f81300089f0 tx=0x7f813000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:03:57.329 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:57.329+0000 7f81397fa700 1 -- 172.21.15.86:0/2348720806 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8130009320 con 0x7f813c0f5410 2024-02-19T22:03:57.329 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:57.329+0000 7f81397fa700 1 -- 172.21.15.86:0/2348720806 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8130013380 con 0x7f813c0f5410 2024-02-19T22:03:57.329 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:57.329+0000 7f814316f700 1 -- 172.21.15.86:0/2348720806 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f813c105db0 con 0x7f813c0f5410 2024-02-19T22:03:57.329 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:57.329+0000 7f81397fa700 1 -- 172.21.15.86:0/2348720806 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f813001ed80 con 0x7f813c0f5410 2024-02-19T22:03:57.329 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:57.329+0000 7f814316f700 1 -- 172.21.15.86:0/2348720806 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f813c106240 con 0x7f813c0f5410 2024-02-19T22:03:57.330 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:57.330+0000 7f81397fa700 1 -- 172.21.15.86:0/2348720806 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f813001e420 con 0x7f813c0f5410 2024-02-19T22:03:57.330 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:57.330+0000 7f814316f700 1 -- 172.21.15.86:0/2348720806 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f813c045ad0 con 0x7f813c0f5410 2024-02-19T22:03:57.330 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:57.330+0000 7f81397fa700 1 --2- 172.21.15.86:0/2348720806 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f812c02ff60 0x7f812c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:03:57.330 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:57.330+0000 7f81397fa700 1 -- 172.21.15.86:0/2348720806 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8130041000 con 0x7f813c0f5410 2024-02-19T22:03:57.330 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:57.330+0000 7f813bfff700 1 --2- 172.21.15.86:0/2348720806 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f812c02ff60 0x7f812c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:03:57.330 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:57.331+0000 7f813bfff700 1 --2- 172.21.15.86:0/2348720806 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f812c02ff60 0x7f812c032420 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7f81280096a0 tx=0x7f8128006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:03:57.333 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:57.335+0000 7f81397fa700 1 -- 172.21.15.86:0/2348720806 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f81300095f0 con 0x7f813c0f5410 2024-02-19T22:03:57.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:57.539+0000 7f814316f700 1 -- 172.21.15.86:0/2348720806 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f813c1070f0 con 0x7f813c0f5410 2024-02-19T22:03:57.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:57.540+0000 7f81397fa700 1 -- 172.21.15.86:0/2348720806 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f813001cb10 con 0x7f813c0f5410 2024-02-19T22:03:57.539 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T22:03:57.539 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:03:57.540 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:57.541+0000 7f814316f700 1 -- 172.21.15.86:0/2348720806 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f812c02ff60 msgr2=0x7f812c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:03:57.540 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:57.541+0000 7f814316f700 1 --2- 172.21.15.86:0/2348720806 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f812c02ff60 0x7f812c032420 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7f81280096a0 tx=0x7f8128006b40).stop 2024-02-19T22:03:57.540 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:57.541+0000 7f814316f700 1 -- 172.21.15.86:0/2348720806 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f813c0f5410 msgr2=0x7f813c1056c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:03:57.540 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:57.541+0000 7f814316f700 1 --2- 172.21.15.86:0/2348720806 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f813c0f5410 0x7f813c1056c0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7f81300089f0 tx=0x7f813000b040).stop 2024-02-19T22:03:57.540 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:57.541+0000 7f814316f700 1 -- 172.21.15.86:0/2348720806 shutdown_connections 2024-02-19T22:03:57.540 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:57.541+0000 7f814316f700 1 --2- 172.21.15.86:0/2348720806 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f812c02ff60 0x7f812c032420 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:03:57.540 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:57.541+0000 7f814316f700 1 --2- 172.21.15.86:0/2348720806 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f813c0f5410 0x7f813c1056c0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:03:57.540 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:57.541+0000 7f814316f700 1 -- 172.21.15.86:0/2348720806 >> 172.21.15.86:0/2348720806 conn(0x7f813c0f0b70 msgr2=0x7f813c0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:03:57.541 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:57.541+0000 7f814316f700 1 -- 172.21.15.86:0/2348720806 shutdown_connections 2024-02-19T22:03:57.541 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:03:57.541+0000 7f814316f700 1 -- 172.21.15.86:0/2348720806 wait complete. 2024-02-19T22:03:57.541 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T22:03:58.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:03:57 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:03:57.539954+0000 mon.smithi053 (mon.0) 247 : audit [DBG] from='client.? 172.21.15.86:0/2348720806' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:03:59.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:03:58 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:03:57.504543+0000 mgr.smithi053.fyhcdi (mgr.14162) 227 : cluster [DBG] pgmap v215: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:03:59.225 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:03:59.226 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:04:00.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:04:00 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:03:59.504827+0000 mgr.smithi053.fyhcdi (mgr.14162) 228 : cluster [DBG] pgmap v216: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:04:01.169 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:01.168+0000 7f0104455700 1 -- 172.21.15.86:0/34229736 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f00fc0f5410 msgr2=0x7f00fc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:04:01.169 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:01.168+0000 7f0104455700 1 --2- 172.21.15.86:0/34229736 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f00fc0f5410 0x7f00fc0f5830 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 rx=0x7f00ec009fb0 tx=0x7f00ec009b30).stop 2024-02-19T22:04:01.169 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:01.168+0000 7f0104455700 1 -- 172.21.15.86:0/34229736 shutdown_connections 2024-02-19T22:04:01.169 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:01.168+0000 7f0104455700 1 --2- 172.21.15.86:0/34229736 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f00fc0f5410 0x7f00fc0f5830 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:04:01.169 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:01.168+0000 7f0104455700 1 -- 172.21.15.86:0/34229736 >> 172.21.15.86:0/34229736 conn(0x7f00fc0f0b70 msgr2=0x7f00fc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:04:01.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:01.168+0000 7f0104455700 1 -- 172.21.15.86:0/34229736 shutdown_connections 2024-02-19T22:04:01.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:01.168+0000 7f0104455700 1 -- 172.21.15.86:0/34229736 wait complete. 2024-02-19T22:04:01.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:01.168+0000 7f0104455700 1 Processor -- start 2024-02-19T22:04:01.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:01.168+0000 7f0104455700 1 -- start start 2024-02-19T22:04:01.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:01.168+0000 7f0104455700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f00fc0f5410 0x7f00fc1067b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:04:01.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:01.168+0000 7f0104455700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00fc106ca0 con 0x7f00fc0f5410 2024-02-19T22:04:01.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:01.169+0000 7f01021f1700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f00fc0f5410 0x7f00fc1067b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:04:01.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:01.169+0000 7f01021f1700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f00fc0f5410 0x7f00fc1067b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:32820/0 (socket says 172.21.15.86:32820) 2024-02-19T22:04:01.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:01.169+0000 7f01021f1700 1 -- 172.21.15.86:0/4173259963 learned_addr learned my addr 172.21.15.86:0/4173259963 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T22:04:01.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:01.169+0000 7f01021f1700 1 -- 172.21.15.86:0/4173259963 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00ec005040 con 0x7f00fc0f5410 2024-02-19T22:04:01.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:01.169+0000 7f01021f1700 1 --2- 172.21.15.86:0/4173259963 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f00fc0f5410 0x7f00fc1067b0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7f00ec005010 tx=0x7f00ec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:04:01.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:01.169+0000 7f00f2ffd700 1 -- 172.21.15.86:0/4173259963 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f00ec009320 con 0x7f00fc0f5410 2024-02-19T22:04:01.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:01.169+0000 7f0104455700 1 -- 172.21.15.86:0/4173259963 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00fc106ea0 con 0x7f00fc0f5410 2024-02-19T22:04:01.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:01.169+0000 7f00f2ffd700 1 -- 172.21.15.86:0/4173259963 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f00ec013380 con 0x7f00fc0f5410 2024-02-19T22:04:01.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:01.169+0000 7f00f2ffd700 1 -- 172.21.15.86:0/4173259963 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f00ec014e50 con 0x7f00fc0f5410 2024-02-19T22:04:01.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:01.169+0000 7f0104455700 1 -- 172.21.15.86:0/4173259963 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00fc107330 con 0x7f00fc0f5410 2024-02-19T22:04:01.172 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:01.170+0000 7f00f2ffd700 1 -- 172.21.15.86:0/4173259963 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f00ec0095f0 con 0x7f00fc0f5410 2024-02-19T22:04:01.172 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:01.170+0000 7f0104455700 1 -- 172.21.15.86:0/4173259963 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00fc045ad0 con 0x7f00fc0f5410 2024-02-19T22:04:01.172 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:01.170+0000 7f00f2ffd700 1 --2- 172.21.15.86:0/4173259963 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f00e802ff10 0x7f00e80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:04:01.172 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:01.170+0000 7f00f2ffd700 1 -- 172.21.15.86:0/4173259963 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f00ec040f60 con 0x7f00fc0f5410 2024-02-19T22:04:01.172 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:01.170+0000 7f01019f0700 1 --2- 172.21.15.86:0/4173259963 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f00e802ff10 0x7f00e80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:04:01.172 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:01.171+0000 7f01019f0700 1 --2- 172.21.15.86:0/4173259963 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f00e802ff10 0x7f00e80323d0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f00f80096a0 tx=0x7f00f8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:04:01.175 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:01.175+0000 7f00f2ffd700 1 -- 172.21.15.86:0/4173259963 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f00ec014420 con 0x7f00fc0f5410 2024-02-19T22:04:01.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:01.384+0000 7f0104455700 1 -- 172.21.15.86:0/4173259963 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f00fc1081e0 con 0x7f00fc0f5410 2024-02-19T22:04:01.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:01.384+0000 7f00f2ffd700 1 -- 172.21.15.86:0/4173259963 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f00ec014420 con 0x7f00fc0f5410 2024-02-19T22:04:01.384 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T22:04:01.384 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:04:01.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:01.386+0000 7f0104455700 1 -- 172.21.15.86:0/4173259963 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f00e802ff10 msgr2=0x7f00e80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:04:01.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:01.386+0000 7f0104455700 1 --2- 172.21.15.86:0/4173259963 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f00e802ff10 0x7f00e80323d0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f00f80096a0 tx=0x7f00f8006b40).stop 2024-02-19T22:04:01.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:01.386+0000 7f0104455700 1 -- 172.21.15.86:0/4173259963 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f00fc0f5410 msgr2=0x7f00fc1067b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:04:01.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:01.386+0000 7f0104455700 1 --2- 172.21.15.86:0/4173259963 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f00fc0f5410 0x7f00fc1067b0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7f00ec005010 tx=0x7f00ec00b040).stop 2024-02-19T22:04:01.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:01.386+0000 7f0104455700 1 -- 172.21.15.86:0/4173259963 shutdown_connections 2024-02-19T22:04:01.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:01.386+0000 7f0104455700 1 --2- 172.21.15.86:0/4173259963 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f00e802ff10 0x7f00e80323d0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:04:01.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:01.386+0000 7f0104455700 1 --2- 172.21.15.86:0/4173259963 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f00fc0f5410 0x7f00fc1067b0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:04:01.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:01.386+0000 7f0104455700 1 -- 172.21.15.86:0/4173259963 >> 172.21.15.86:0/4173259963 conn(0x7f00fc0f0b70 msgr2=0x7f00fc05fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:04:01.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:01.386+0000 7f0104455700 1 -- 172.21.15.86:0/4173259963 shutdown_connections 2024-02-19T22:04:01.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:01.386+0000 7f0104455700 1 -- 172.21.15.86:0/4173259963 wait complete. 2024-02-19T22:04:01.387 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T22:04:01.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:04:01 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:04:01.385652+0000 mon.smithi053 (mon.0) 248 : audit [DBG] from='client.? 172.21.15.86:0/4173259963' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:04:02.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:04:02 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:04:01.505028+0000 mgr.smithi053.fyhcdi (mgr.14162) 229 : cluster [DBG] pgmap v217: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:04:03.055 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:04:03.055 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:04:04.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:04.972+0000 7fa3bb59e700 1 -- 172.21.15.86:0/2205910341 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa3bc0f51a0 msgr2=0x7fa3bc0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:04:04.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:04.973+0000 7fa3bb59e700 1 --2- 172.21.15.86:0/2205910341 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa3bc0f51a0 0x7fa3bc0f55c0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 rx=0x7fa3b0009fb0 tx=0x7fa3b0009b30).stop 2024-02-19T22:04:04.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:04.973+0000 7fa3bb59e700 1 -- 172.21.15.86:0/2205910341 shutdown_connections 2024-02-19T22:04:04.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:04.973+0000 7fa3bb59e700 1 --2- 172.21.15.86:0/2205910341 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa3bc0f51a0 0x7fa3bc0f55c0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:04:04.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:04.973+0000 7fa3bb59e700 1 -- 172.21.15.86:0/2205910341 >> 172.21.15.86:0/2205910341 conn(0x7fa3bc0f09a0 msgr2=0x7fa3bc0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:04:04.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:04.973+0000 7fa3bb59e700 1 -- 172.21.15.86:0/2205910341 shutdown_connections 2024-02-19T22:04:04.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:04.973+0000 7fa3bb59e700 1 -- 172.21.15.86:0/2205910341 wait complete. 2024-02-19T22:04:04.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:04.973+0000 7fa3bb59e700 1 Processor -- start 2024-02-19T22:04:04.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:04.973+0000 7fa3bb59e700 1 -- start start 2024-02-19T22:04:04.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:04.974+0000 7fa3bb59e700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa3bc0f51a0 0x7fa3bc1092f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:04:04.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:04.974+0000 7fa3bb59e700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3bc060220 con 0x7fa3bc0f51a0 2024-02-19T22:04:04.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:04.974+0000 7fa3ba59c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa3bc0f51a0 0x7fa3bc1092f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:04:04.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:04.974+0000 7fa3ba59c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa3bc0f51a0 0x7fa3bc1092f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:32834/0 (socket says 172.21.15.86:32834) 2024-02-19T22:04:04.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:04.974+0000 7fa3ba59c700 1 -- 172.21.15.86:0/2982011679 learned_addr learned my addr 172.21.15.86:0/2982011679 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T22:04:04.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:04.974+0000 7fa3ba59c700 1 -- 172.21.15.86:0/2982011679 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3b0005040 con 0x7fa3bc0f51a0 2024-02-19T22:04:04.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:04.974+0000 7fa3ba59c700 1 --2- 172.21.15.86:0/2982011679 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa3bc0f51a0 0x7fa3bc1092f0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0x7fa3b00089f0 tx=0x7fa3b000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:04:04.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:04.974+0000 7fa3aaffd700 1 -- 172.21.15.86:0/2982011679 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa3b0009320 con 0x7fa3bc0f51a0 2024-02-19T22:04:04.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:04.974+0000 7fa3bb59e700 1 -- 172.21.15.86:0/2982011679 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3bc060420 con 0x7fa3bc0f51a0 2024-02-19T22:04:04.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:04.974+0000 7fa3aaffd700 1 -- 172.21.15.86:0/2982011679 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa3b0013380 con 0x7fa3bc0f51a0 2024-02-19T22:04:04.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:04.974+0000 7fa3aaffd700 1 -- 172.21.15.86:0/2982011679 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa3b0014e20 con 0x7fa3bc0f51a0 2024-02-19T22:04:04.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:04.974+0000 7fa3bb59e700 1 -- 172.21.15.86:0/2982011679 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3bc1099a0 con 0x7fa3bc0f51a0 2024-02-19T22:04:04.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:04.975+0000 7fa3aaffd700 1 -- 172.21.15.86:0/2982011679 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa3b00095f0 con 0x7fa3bc0f51a0 2024-02-19T22:04:04.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:04.975+0000 7fa3bb59e700 1 -- 172.21.15.86:0/2982011679 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3bc059100 con 0x7fa3bc0f51a0 2024-02-19T22:04:04.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:04.975+0000 7fa3aaffd700 1 --2- 172.21.15.86:0/2982011679 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fa3a402ff10 0x7fa3a40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:04:04.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:04.975+0000 7fa3aaffd700 1 -- 172.21.15.86:0/2982011679 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa3b0040e90 con 0x7fa3bc0f51a0 2024-02-19T22:04:04.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:04.976+0000 7fa3b9d9b700 1 --2- 172.21.15.86:0/2982011679 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fa3a402ff10 0x7fa3a40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:04:04.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:04.976+0000 7fa3b9d9b700 1 --2- 172.21.15.86:0/2982011679 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fa3a402ff10 0x7fa3a40323d0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7fa3ac0096a0 tx=0x7fa3ac006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:04:04.980 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:04.980+0000 7fa3aaffd700 1 -- 172.21.15.86:0/2982011679 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa3b0014420 con 0x7fa3bc0f51a0 2024-02-19T22:04:05.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:04:04 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:04:03.505273+0000 mgr.smithi053.fyhcdi (mgr.14162) 230 : cluster [DBG] pgmap v218: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:04:05.186 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:05.186+0000 7fa3bb59e700 1 -- 172.21.15.86:0/2982011679 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa3bc045ad0 con 0x7fa3bc0f51a0 2024-02-19T22:04:05.186 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:05.187+0000 7fa3aaffd700 1 -- 172.21.15.86:0/2982011679 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa3b001d980 con 0x7fa3bc0f51a0 2024-02-19T22:04:05.187 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T22:04:05.187 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:04:05.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:05.188+0000 7fa3bb59e700 1 -- 172.21.15.86:0/2982011679 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fa3a402ff10 msgr2=0x7fa3a40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:04:05.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:05.188+0000 7fa3bb59e700 1 --2- 172.21.15.86:0/2982011679 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fa3a402ff10 0x7fa3a40323d0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7fa3ac0096a0 tx=0x7fa3ac006b40).stop 2024-02-19T22:04:05.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:05.188+0000 7fa3bb59e700 1 -- 172.21.15.86:0/2982011679 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa3bc0f51a0 msgr2=0x7fa3bc1092f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:04:05.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:05.188+0000 7fa3bb59e700 1 --2- 172.21.15.86:0/2982011679 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa3bc0f51a0 0x7fa3bc1092f0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0x7fa3b00089f0 tx=0x7fa3b000b040).stop 2024-02-19T22:04:05.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:05.188+0000 7fa3bb59e700 1 -- 172.21.15.86:0/2982011679 shutdown_connections 2024-02-19T22:04:05.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:05.188+0000 7fa3bb59e700 1 --2- 172.21.15.86:0/2982011679 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fa3a402ff10 0x7fa3a40323d0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:04:05.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:05.188+0000 7fa3bb59e700 1 --2- 172.21.15.86:0/2982011679 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa3bc0f51a0 0x7fa3bc1092f0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:04:05.189 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:05.188+0000 7fa3bb59e700 1 -- 172.21.15.86:0/2982011679 >> 172.21.15.86:0/2982011679 conn(0x7fa3bc0f09a0 msgr2=0x7fa3bc0f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:04:05.189 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:05.188+0000 7fa3bb59e700 1 -- 172.21.15.86:0/2982011679 shutdown_connections 2024-02-19T22:04:05.189 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:05.188+0000 7fa3bb59e700 1 -- 172.21.15.86:0/2982011679 wait complete. 2024-02-19T22:04:05.189 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T22:04:06.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:04:05 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:04:05.188082+0000 mon.smithi053 (mon.0) 249 : audit [DBG] from='client.? 172.21.15.86:0/2982011679' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:04:06.865 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:04:06.866 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:04:07.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:04:06 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:04:05.505623+0000 mgr.smithi053.fyhcdi (mgr.14162) 231 : cluster [DBG] pgmap v219: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:04:08.663 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:08.662+0000 7fbdffce7700 1 -- 172.21.15.86:0/360175054 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbdf80f5410 msgr2=0x7fbdf80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:04:08.664 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:08.662+0000 7fbdffce7700 1 --2- 172.21.15.86:0/360175054 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbdf80f5410 0x7fbdf80f5830 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 rx=0x7fbdf4009fb0 tx=0x7fbdf4009b30).stop 2024-02-19T22:04:08.664 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:08.662+0000 7fbdffce7700 1 -- 172.21.15.86:0/360175054 shutdown_connections 2024-02-19T22:04:08.664 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:08.662+0000 7fbdffce7700 1 --2- 172.21.15.86:0/360175054 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbdf80f5410 0x7fbdf80f5830 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:04:08.664 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:08.662+0000 7fbdffce7700 1 -- 172.21.15.86:0/360175054 >> 172.21.15.86:0/360175054 conn(0x7fbdf80f0b70 msgr2=0x7fbdf80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:04:08.664 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:08.662+0000 7fbdffce7700 1 -- 172.21.15.86:0/360175054 shutdown_connections 2024-02-19T22:04:08.664 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:08.662+0000 7fbdffce7700 1 -- 172.21.15.86:0/360175054 wait complete. 2024-02-19T22:04:08.664 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:08.663+0000 7fbdffce7700 1 Processor -- start 2024-02-19T22:04:08.664 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:08.663+0000 7fbdffce7700 1 -- start start 2024-02-19T22:04:08.664 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:08.663+0000 7fbdffce7700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbdf80f5410 0x7fbdf8109520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:04:08.664 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:08.663+0000 7fbdffce7700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdf8060240 con 0x7fbdf80f5410 2024-02-19T22:04:08.664 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:08.663+0000 7fbdfda83700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbdf80f5410 0x7fbdf8109520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:04:08.665 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:08.663+0000 7fbdfda83700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbdf80f5410 0x7fbdf8109520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:32864/0 (socket says 172.21.15.86:32864) 2024-02-19T22:04:08.665 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:08.663+0000 7fbdfda83700 1 -- 172.21.15.86:0/2460606050 learned_addr learned my addr 172.21.15.86:0/2460606050 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T22:04:08.665 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:08.663+0000 7fbdfda83700 1 -- 172.21.15.86:0/2460606050 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdf4005040 con 0x7fbdf80f5410 2024-02-19T22:04:08.665 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:08.663+0000 7fbdfda83700 1 --2- 172.21.15.86:0/2460606050 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbdf80f5410 0x7fbdf8109520 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x7fbdf4005010 tx=0x7fbdf400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:04:08.665 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:08.664+0000 7fbdee7fc700 1 -- 172.21.15.86:0/2460606050 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbdf4009320 con 0x7fbdf80f5410 2024-02-19T22:04:08.665 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:08.664+0000 7fbdffce7700 1 -- 172.21.15.86:0/2460606050 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbdf8060440 con 0x7fbdf80f5410 2024-02-19T22:04:08.665 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:08.664+0000 7fbdee7fc700 1 -- 172.21.15.86:0/2460606050 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbdf4013380 con 0x7fbdf80f5410 2024-02-19T22:04:08.665 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:08.664+0000 7fbdee7fc700 1 -- 172.21.15.86:0/2460606050 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbdf4014e50 con 0x7fbdf80f5410 2024-02-19T22:04:08.665 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:08.664+0000 7fbdffce7700 1 -- 172.21.15.86:0/2460606050 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdf8109bd0 con 0x7fbdf80f5410 2024-02-19T22:04:08.666 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:08.665+0000 7fbdee7fc700 1 -- 172.21.15.86:0/2460606050 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbdf4014420 con 0x7fbdf80f5410 2024-02-19T22:04:08.666 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:08.665+0000 7fbdffce7700 1 -- 172.21.15.86:0/2460606050 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbdf8045ad0 con 0x7fbdf80f5410 2024-02-19T22:04:08.666 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:08.665+0000 7fbdee7fc700 1 --2- 172.21.15.86:0/2460606050 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fbde402ff10 0x7fbde40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:04:08.666 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:08.665+0000 7fbdee7fc700 1 -- 172.21.15.86:0/2460606050 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbdf4040e30 con 0x7fbdf80f5410 2024-02-19T22:04:08.666 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:08.665+0000 7fbdfd282700 1 --2- 172.21.15.86:0/2460606050 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fbde402ff10 0x7fbde40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:04:08.666 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:08.665+0000 7fbdfd282700 1 --2- 172.21.15.86:0/2460606050 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fbde402ff10 0x7fbde40323d0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7fbde80096a0 tx=0x7fbde8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:04:08.669 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:08.669+0000 7fbdee7fc700 1 -- 172.21.15.86:0/2460606050 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbdf40095f0 con 0x7fbdf80f5410 2024-02-19T22:04:08.872 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:08.873+0000 7fbdffce7700 1 -- 172.21.15.86:0/2460606050 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbdf810a3f0 con 0x7fbdf80f5410 2024-02-19T22:04:08.873 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:08.873+0000 7fbdee7fc700 1 -- 172.21.15.86:0/2460606050 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fbdf401db70 con 0x7fbdf80f5410 2024-02-19T22:04:08.873 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T22:04:08.873 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:04:08.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:08.875+0000 7fbdffce7700 1 -- 172.21.15.86:0/2460606050 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fbde402ff10 msgr2=0x7fbde40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:04:08.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:08.875+0000 7fbdffce7700 1 --2- 172.21.15.86:0/2460606050 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fbde402ff10 0x7fbde40323d0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7fbde80096a0 tx=0x7fbde8006b40).stop 2024-02-19T22:04:08.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:08.875+0000 7fbdffce7700 1 -- 172.21.15.86:0/2460606050 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbdf80f5410 msgr2=0x7fbdf8109520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:04:08.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:08.875+0000 7fbdffce7700 1 --2- 172.21.15.86:0/2460606050 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbdf80f5410 0x7fbdf8109520 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x7fbdf4005010 tx=0x7fbdf400b040).stop 2024-02-19T22:04:08.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:08.875+0000 7fbdffce7700 1 -- 172.21.15.86:0/2460606050 shutdown_connections 2024-02-19T22:04:08.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:08.875+0000 7fbdffce7700 1 --2- 172.21.15.86:0/2460606050 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fbde402ff10 0x7fbde40323d0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:04:08.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:08.875+0000 7fbdffce7700 1 --2- 172.21.15.86:0/2460606050 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbdf80f5410 0x7fbdf8109520 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:04:08.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:08.875+0000 7fbdffce7700 1 -- 172.21.15.86:0/2460606050 >> 172.21.15.86:0/2460606050 conn(0x7fbdf80f0b70 msgr2=0x7fbdf80f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:04:08.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:08.875+0000 7fbdffce7700 1 -- 172.21.15.86:0/2460606050 shutdown_connections 2024-02-19T22:04:08.876 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:08.875+0000 7fbdffce7700 1 -- 172.21.15.86:0/2460606050 wait complete. 2024-02-19T22:04:08.876 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T22:04:09.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:04:08 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:04:07.505979+0000 mgr.smithi053.fyhcdi (mgr.14162) 232 : cluster [DBG] pgmap v220: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:04:10.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:04:09 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cephadm 2024-02-19T22:04:08.335216+0000 mgr.smithi053.fyhcdi (mgr.14162) 233 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Deploy daemon node-exporter.smithi053 ... 2024-02-19T22:04:10.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:04:09 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: Verifying port 9100 ... 2024-02-19T22:04:10.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:04:09 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: Non-zero exit code 1 from systemctl start ceph-867b6b52-cf71-11ee-95bb-87774f69a715@node-exporter.smithi053 2024-02-19T22:04:10.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:04:09 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: systemctl: stderr Job for ceph-867b6b52-cf71-11ee-95bb-87774f69a715@node-exporter.smithi053.service failed because a timeout was exceeded. 2024-02-19T22:04:10.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:04:09 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: systemctl: stderr See "systemctl status ceph-867b6b52-cf71-11ee-95bb-87774f69a715@node-exporter.smithi053.service" and "journalctl -xe" for details. 2024-02-19T22:04:10.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:04:09 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: Traceback (most recent call last): 2024-02-19T22:04:10.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:04:09 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: File "/var/lib/ceph/867b6b52-cf71-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-02-19T22:04:10.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:04:09 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: main() 2024-02-19T22:04:10.045 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:04:09 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: File "/var/lib/ceph/867b6b52-cf71-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-02-19T22:04:10.045 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:04:09 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: r = ctx.func(ctx) 2024-02-19T22:04:10.045 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:04:09 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: File "/var/lib/ceph/867b6b52-cf71-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-02-19T22:04:10.045 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:04:09 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: return func(ctx) 2024-02-19T22:04:10.045 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:04:09 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: File "/var/lib/ceph/867b6b52-cf71-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4267, in command_deploy 2024-02-19T22:04:10.045 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:04:09 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: ports=daemon_ports) 2024-02-19T22:04:10.045 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:04:09 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: File "/var/lib/ceph/867b6b52-cf71-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2608, in deploy_daemon 2024-02-19T22:04:10.045 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:04:09 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: c, osd_fsid=osd_fsid, ports=ports) 2024-02-19T22:04:10.045 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:04:09 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: File "/var/lib/ceph/867b6b52-cf71-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2806, in deploy_daemon_units 2024-02-19T22:04:10.045 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:04:09 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: call_throws(ctx, ['systemctl', 'start', unit_name]) 2024-02-19T22:04:10.045 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:04:09 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: File "/var/lib/ceph/867b6b52-cf71-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1464, in call_throws 2024-02-19T22:04:10.045 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:04:09 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-02-19T22:04:10.045 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:04:09 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: RuntimeError: Failed command: systemctl start ceph-867b6b52-cf71-11ee-95bb-87774f69a715@node-exporter.smithi053 2024-02-19T22:04:10.046 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:04:09 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: Traceback (most recent call last): 2024-02-19T22:04:10.046 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:04:09 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1144, in _remote_connection 2024-02-19T22:04:10.046 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:04:09 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: yield (conn, connr) 2024-02-19T22:04:10.046 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:04:09 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1059, in _run_cephadm 2024-02-19T22:04:10.046 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:04:09 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: code, '\n'.join(err))) 2024-02-19T22:04:10.046 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:04:09 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Deploy daemon node-exporter.smithi053 ... 2024-02-19T22:04:10.046 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:04:09 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: Verifying port 9100 ... 2024-02-19T22:04:10.046 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:04:09 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: Non-zero exit code 1 from systemctl start ceph-867b6b52-cf71-11ee-95bb-87774f69a715@node-exporter.smithi053 2024-02-19T22:04:10.046 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:04:09 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: systemctl: stderr Job for ceph-867b6b52-cf71-11ee-95bb-87774f69a715@node-exporter.smithi053.service failed because a timeout was exceeded. 2024-02-19T22:04:10.046 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:04:09 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: systemctl: stderr See "systemctl status ceph-867b6b52-cf71-11ee-95bb-87774f69a715@node-exporter.smithi053.service" and "journalctl -xe" for details. 2024-02-19T22:04:10.046 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:04:09 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: Traceback (most recent call last): 2024-02-19T22:04:10.046 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:04:09 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: File "/var/lib/ceph/867b6b52-cf71-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-02-19T22:04:10.046 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:04:09 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: main() 2024-02-19T22:04:10.046 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:04:09 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: File "/var/lib/ceph/867b6b52-cf71-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-02-19T22:04:10.047 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:04:09 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: r = ctx.func(ctx) 2024-02-19T22:04:10.047 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:04:09 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: File "/var/lib/ceph/867b6b52-cf71-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-02-19T22:04:10.047 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:04:09 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: return func(ctx) 2024-02-19T22:04:10.047 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:04:09 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: File "/var/lib/ceph/867b6b52-cf71-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4267, in command_deploy 2024-02-19T22:04:10.047 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:04:09 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: ports=daemon_ports) 2024-02-19T22:04:10.047 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:04:09 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: File "/var/lib/ceph/867b6b52-cf71-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2608, in deploy_daemon 2024-02-19T22:04:10.047 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:04:09 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: c, osd_fsid=osd_fsid, ports=ports) 2024-02-19T22:04:10.047 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:04:09 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: File "/var/lib/ceph/867b6b52-cf71-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2806, in deploy_daemon_units 2024-02-19T22:04:10.047 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:04:09 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: call_throws(ctx, ['systemctl', 'start', unit_name]) 2024-02-19T22:04:10.047 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:04:09 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: File "/var/lib/ceph/867b6b52-cf71-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1464, in call_throws 2024-02-19T22:04:10.047 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:04:09 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-02-19T22:04:10.047 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:04:09 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: RuntimeError: Failed command: systemctl start ceph-867b6b52-cf71-11ee-95bb-87774f69a715@node-exporter.smithi053 2024-02-19T22:04:10.047 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:04:09 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cephadm 2024-02-19T22:04:08.340837+0000 mgr.smithi053.fyhcdi (mgr.14162) 234 : cephadm [INF] Deploying daemon prometheus.smithi053 on smithi053 2024-02-19T22:04:10.047 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:04:09 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:04:08.874740+0000 mon.smithi053 (mon.0) 250 : audit [DBG] from='client.? 172.21.15.86:0/2460606050' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:04:10.473 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:04:10.474 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:04:11.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:04:10 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:04:09.506344+0000 mgr.smithi053.fyhcdi (mgr.14162) 235 : cluster [DBG] pgmap v221: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:04:12.430 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:12.430+0000 7f90cda91700 1 -- 172.21.15.86:0/4169718988 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f90c80f6510 msgr2=0x7f90c80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:04:12.430 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:12.430+0000 7f90cda91700 1 --2- 172.21.15.86:0/4169718988 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f90c80f6510 0x7f90c80f6930 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 rx=0x7f90bc009fb0 tx=0x7f90bc009b30).stop 2024-02-19T22:04:12.430 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:12.430+0000 7f90cda91700 1 -- 172.21.15.86:0/4169718988 shutdown_connections 2024-02-19T22:04:12.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:12.430+0000 7f90cda91700 1 --2- 172.21.15.86:0/4169718988 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f90c80f6510 0x7f90c80f6930 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:04:12.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:12.430+0000 7f90cda91700 1 -- 172.21.15.86:0/4169718988 >> 172.21.15.86:0/4169718988 conn(0x7f90c80f1c70 msgr2=0x7f90c80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:04:12.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:12.430+0000 7f90cda91700 1 -- 172.21.15.86:0/4169718988 shutdown_connections 2024-02-19T22:04:12.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:12.430+0000 7f90cda91700 1 -- 172.21.15.86:0/4169718988 wait complete. 2024-02-19T22:04:12.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:12.430+0000 7f90cda91700 1 Processor -- start 2024-02-19T22:04:12.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:12.431+0000 7f90cda91700 1 -- start start 2024-02-19T22:04:12.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:12.431+0000 7f90cda91700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f90c80f6510 0x7f90c81067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:04:12.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:12.431+0000 7f90cda91700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90c8106cd0 con 0x7f90c80f6510 2024-02-19T22:04:12.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:12.431+0000 7f90c77fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f90c80f6510 0x7f90c81067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:04:12.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:12.431+0000 7f90c77fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f90c80f6510 0x7f90c81067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:45826/0 (socket says 172.21.15.86:45826) 2024-02-19T22:04:12.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:12.431+0000 7f90c77fe700 1 -- 172.21.15.86:0/2841601955 learned_addr learned my addr 172.21.15.86:0/2841601955 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T22:04:12.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:12.431+0000 7f90c77fe700 1 -- 172.21.15.86:0/2841601955 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90bc005040 con 0x7f90c80f6510 2024-02-19T22:04:12.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:12.432+0000 7f90c77fe700 1 --2- 172.21.15.86:0/2841601955 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f90c80f6510 0x7f90c81067e0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0x7f90bc005010 tx=0x7f90bc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:04:12.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:12.432+0000 7f90affff700 1 -- 172.21.15.86:0/2841601955 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f90bc009320 con 0x7f90c80f6510 2024-02-19T22:04:12.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:12.432+0000 7f90cda91700 1 -- 172.21.15.86:0/2841601955 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90c8106ed0 con 0x7f90c80f6510 2024-02-19T22:04:12.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:12.432+0000 7f90affff700 1 -- 172.21.15.86:0/2841601955 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f90bc013380 con 0x7f90c80f6510 2024-02-19T22:04:12.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:12.432+0000 7f90affff700 1 -- 172.21.15.86:0/2841601955 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f90bc014e50 con 0x7f90c80f6510 2024-02-19T22:04:12.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:12.432+0000 7f90cda91700 1 -- 172.21.15.86:0/2841601955 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90c8107360 con 0x7f90c80f6510 2024-02-19T22:04:12.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:12.432+0000 7f90affff700 1 -- 172.21.15.86:0/2841601955 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f90bc014420 con 0x7f90c80f6510 2024-02-19T22:04:12.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:12.433+0000 7f90cda91700 1 -- 172.21.15.86:0/2841601955 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90c8045ad0 con 0x7f90c80f6510 2024-02-19T22:04:12.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:12.433+0000 7f90affff700 1 --2- 172.21.15.86:0/2841601955 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f90b002ff10 0x7f90b00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:04:12.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:12.433+0000 7f90affff700 1 -- 172.21.15.86:0/2841601955 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f90bc040f40 con 0x7f90c80f6510 2024-02-19T22:04:12.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:12.433+0000 7f90c6ffd700 1 --2- 172.21.15.86:0/2841601955 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f90b002ff10 0x7f90b00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:04:12.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:12.433+0000 7f90c6ffd700 1 --2- 172.21.15.86:0/2841601955 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f90b002ff10 0x7f90b00323d0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7f90b80096a0 tx=0x7f90b8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:04:12.437 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:12.437+0000 7f90affff700 1 -- 172.21.15.86:0/2841601955 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f90bc0095f0 con 0x7f90c80f6510 2024-02-19T22:04:12.641 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:12.642+0000 7f90cda91700 1 -- 172.21.15.86:0/2841601955 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f90c8108210 con 0x7f90c80f6510 2024-02-19T22:04:12.642 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:12.642+0000 7f90affff700 1 -- 172.21.15.86:0/2841601955 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f90bc01db70 con 0x7f90c80f6510 2024-02-19T22:04:12.642 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T22:04:12.642 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:04:12.644 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:12.644+0000 7f90cda91700 1 -- 172.21.15.86:0/2841601955 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f90b002ff10 msgr2=0x7f90b00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:04:12.644 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:12.644+0000 7f90cda91700 1 --2- 172.21.15.86:0/2841601955 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f90b002ff10 0x7f90b00323d0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7f90b80096a0 tx=0x7f90b8006b40).stop 2024-02-19T22:04:12.644 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:12.644+0000 7f90cda91700 1 -- 172.21.15.86:0/2841601955 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f90c80f6510 msgr2=0x7f90c81067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:04:12.644 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:12.644+0000 7f90cda91700 1 --2- 172.21.15.86:0/2841601955 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f90c80f6510 0x7f90c81067e0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0x7f90bc005010 tx=0x7f90bc00b040).stop 2024-02-19T22:04:12.644 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:12.644+0000 7f90cda91700 1 -- 172.21.15.86:0/2841601955 shutdown_connections 2024-02-19T22:04:12.644 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:12.644+0000 7f90cda91700 1 --2- 172.21.15.86:0/2841601955 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f90b002ff10 0x7f90b00323d0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:04:12.644 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:12.644+0000 7f90cda91700 1 --2- 172.21.15.86:0/2841601955 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f90c80f6510 0x7f90c81067e0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:04:12.644 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:12.644+0000 7f90cda91700 1 -- 172.21.15.86:0/2841601955 >> 172.21.15.86:0/2841601955 conn(0x7f90c80f1c70 msgr2=0x7f90c80f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:04:12.644 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:12.644+0000 7f90cda91700 1 -- 172.21.15.86:0/2841601955 shutdown_connections 2024-02-19T22:04:12.645 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:12.644+0000 7f90cda91700 1 -- 172.21.15.86:0/2841601955 wait complete. 2024-02-19T22:04:12.645 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T22:04:13.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:04:12 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:04:11.506717+0000 mgr.smithi053.fyhcdi (mgr.14162) 236 : cluster [DBG] pgmap v222: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:04:14.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:04:13 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:04:12.643740+0000 mon.smithi053 (mon.0) 251 : audit [DBG] from='client.? 172.21.15.86:0/2841601955' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:04:14.426 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:04:14.427 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:04:15.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:04:14 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:04:13.507004+0000 mgr.smithi053.fyhcdi (mgr.14162) 237 : cluster [DBG] pgmap v223: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:04:16.343 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:16.341+0000 7f906c096700 1 -- 172.21.15.86:0/3844680746 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f90640f6a80 msgr2=0x7f90640f6ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:04:16.343 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:16.341+0000 7f906c096700 1 --2- 172.21.15.86:0/3844680746 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f90640f6a80 0x7f90640f6ea0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 rx=0x7f9054009fb0 tx=0x7f9054009b30).stop 2024-02-19T22:04:16.343 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:16.341+0000 7f906c096700 1 -- 172.21.15.86:0/3844680746 shutdown_connections 2024-02-19T22:04:16.343 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:16.341+0000 7f906c096700 1 --2- 172.21.15.86:0/3844680746 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f90640f6a80 0x7f90640f6ea0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:04:16.343 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:16.341+0000 7f906c096700 1 -- 172.21.15.86:0/3844680746 >> 172.21.15.86:0/3844680746 conn(0x7f90640f02e0 msgr2=0x7f90640f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:04:16.343 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:16.341+0000 7f906c096700 1 -- 172.21.15.86:0/3844680746 shutdown_connections 2024-02-19T22:04:16.343 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:16.341+0000 7f906c096700 1 -- 172.21.15.86:0/3844680746 wait complete. 2024-02-19T22:04:16.343 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:16.342+0000 7f906c096700 1 Processor -- start 2024-02-19T22:04:16.343 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:16.342+0000 7f906c096700 1 -- start start 2024-02-19T22:04:16.343 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:16.342+0000 7f906c096700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f90640f6a80 0x7f90641056a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:04:16.343 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:16.342+0000 7f906c096700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9064105b90 con 0x7f90640f6a80 2024-02-19T22:04:16.343 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:16.342+0000 7f9069e32700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f90640f6a80 0x7f90641056a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:04:16.344 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:16.342+0000 7f9069e32700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f90640f6a80 0x7f90641056a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:45846/0 (socket says 172.21.15.86:45846) 2024-02-19T22:04:16.344 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:16.342+0000 7f9069e32700 1 -- 172.21.15.86:0/1612989660 learned_addr learned my addr 172.21.15.86:0/1612989660 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T22:04:16.344 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:16.342+0000 7f9069e32700 1 -- 172.21.15.86:0/1612989660 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9054005040 con 0x7f90640f6a80 2024-02-19T22:04:16.344 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:16.343+0000 7f9069e32700 1 --2- 172.21.15.86:0/1612989660 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f90640f6a80 0x7f90641056a0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7f90540089f0 tx=0x7f905400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:04:16.344 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:16.343+0000 7f905a7fc700 1 -- 172.21.15.86:0/1612989660 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9054009320 con 0x7f90640f6a80 2024-02-19T22:04:16.344 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:16.343+0000 7f906c096700 1 -- 172.21.15.86:0/1612989660 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9064105d90 con 0x7f90640f6a80 2024-02-19T22:04:16.344 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:16.343+0000 7f905a7fc700 1 -- 172.21.15.86:0/1612989660 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9054013380 con 0x7f90640f6a80 2024-02-19T22:04:16.344 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:16.343+0000 7f905a7fc700 1 -- 172.21.15.86:0/1612989660 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9054014e20 con 0x7f90640f6a80 2024-02-19T22:04:16.344 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:16.343+0000 7f906c096700 1 -- 172.21.15.86:0/1612989660 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9064106220 con 0x7f90640f6a80 2024-02-19T22:04:16.345 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:16.344+0000 7f905a7fc700 1 -- 172.21.15.86:0/1612989660 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9054014420 con 0x7f90640f6a80 2024-02-19T22:04:16.345 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:16.344+0000 7f906c096700 1 -- 172.21.15.86:0/1612989660 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9064045ad0 con 0x7f90640f6a80 2024-02-19T22:04:16.345 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:16.344+0000 7f905a7fc700 1 --2- 172.21.15.86:0/1612989660 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f905002ff10 0x7f90500323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:04:16.345 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:16.344+0000 7f905a7fc700 1 -- 172.21.15.86:0/1612989660 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9054042da0 con 0x7f90640f6a80 2024-02-19T22:04:16.345 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:16.344+0000 7f9069631700 1 --2- 172.21.15.86:0/1612989660 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f905002ff10 0x7f90500323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:04:16.345 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:16.345+0000 7f9069631700 1 --2- 172.21.15.86:0/1612989660 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f905002ff10 0x7f90500323d0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f90600096a0 tx=0x7f9060006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:04:16.349 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:16.349+0000 7f905a7fc700 1 -- 172.21.15.86:0/1612989660 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9054014790 con 0x7f90640f6a80 2024-02-19T22:04:16.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:16.554+0000 7f906c096700 1 -- 172.21.15.86:0/1612989660 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9064107070 con 0x7f90640f6a80 2024-02-19T22:04:16.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:16.555+0000 7f905a7fc700 1 -- 172.21.15.86:0/1612989660 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f905401f7f0 con 0x7f90640f6a80 2024-02-19T22:04:16.554 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T22:04:16.555 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:04:16.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:16.556+0000 7f906c096700 1 -- 172.21.15.86:0/1612989660 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f905002ff10 msgr2=0x7f90500323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:04:16.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:16.556+0000 7f906c096700 1 --2- 172.21.15.86:0/1612989660 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f905002ff10 0x7f90500323d0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f90600096a0 tx=0x7f9060006b40).stop 2024-02-19T22:04:16.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:16.556+0000 7f906c096700 1 -- 172.21.15.86:0/1612989660 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f90640f6a80 msgr2=0x7f90641056a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:04:16.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:16.556+0000 7f906c096700 1 --2- 172.21.15.86:0/1612989660 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f90640f6a80 0x7f90641056a0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7f90540089f0 tx=0x7f905400b040).stop 2024-02-19T22:04:16.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:16.556+0000 7f906c096700 1 -- 172.21.15.86:0/1612989660 shutdown_connections 2024-02-19T22:04:16.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:16.556+0000 7f906c096700 1 --2- 172.21.15.86:0/1612989660 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f905002ff10 0x7f90500323d0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:04:16.557 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:16.556+0000 7f906c096700 1 --2- 172.21.15.86:0/1612989660 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f90640f6a80 0x7f90641056a0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:04:16.557 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:16.556+0000 7f906c096700 1 -- 172.21.15.86:0/1612989660 >> 172.21.15.86:0/1612989660 conn(0x7f90640f02e0 msgr2=0x7f90640f0d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:04:16.557 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:16.556+0000 7f906c096700 1 -- 172.21.15.86:0/1612989660 shutdown_connections 2024-02-19T22:04:16.557 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:16.556+0000 7f906c096700 1 -- 172.21.15.86:0/1612989660 wait complete. 2024-02-19T22:04:16.557 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T22:04:17.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:04:16 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:04:15.507206+0000 mgr.smithi053.fyhcdi (mgr.14162) 238 : cluster [DBG] pgmap v224: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:04:17.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:04:16 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:04:16.555705+0000 mon.smithi053 (mon.0) 252 : audit [DBG] from='client.? 172.21.15.86:0/1612989660' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:04:18.365 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:04:18.366 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:04:19.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:04:18 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:04:17.507405+0000 mgr.smithi053.fyhcdi (mgr.14162) 239 : cluster [DBG] pgmap v225: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:04:20.175 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:20.171+0000 7fd199829700 1 -- 172.21.15.86:0/1719318160 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd1940f51a0 msgr2=0x7fd1940f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:04:20.175 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:20.171+0000 7fd199829700 1 --2- 172.21.15.86:0/1719318160 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd1940f51a0 0x7fd1940f55c0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 rx=0x7fd188009fb0 tx=0x7fd188009b30).stop 2024-02-19T22:04:20.175 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:20.172+0000 7fd199829700 1 -- 172.21.15.86:0/1719318160 shutdown_connections 2024-02-19T22:04:20.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:20.172+0000 7fd199829700 1 --2- 172.21.15.86:0/1719318160 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd1940f51a0 0x7fd1940f55c0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:04:20.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:20.172+0000 7fd199829700 1 -- 172.21.15.86:0/1719318160 >> 172.21.15.86:0/1719318160 conn(0x7fd1940f09a0 msgr2=0x7fd1940f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:04:20.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:20.172+0000 7fd199829700 1 -- 172.21.15.86:0/1719318160 shutdown_connections 2024-02-19T22:04:20.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:20.172+0000 7fd199829700 1 -- 172.21.15.86:0/1719318160 wait complete. 2024-02-19T22:04:20.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:20.172+0000 7fd199829700 1 Processor -- start 2024-02-19T22:04:20.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:20.172+0000 7fd199829700 1 -- start start 2024-02-19T22:04:20.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:20.172+0000 7fd199829700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd1940f51a0 0x7fd1940f2f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:04:20.177 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:20.172+0000 7fd199829700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1940f13a0 con 0x7fd1940f51a0 2024-02-19T22:04:20.177 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:20.173+0000 7fd198827700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd1940f51a0 0x7fd1940f2f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:04:20.177 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:20.173+0000 7fd198827700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd1940f51a0 0x7fd1940f2f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:34930/0 (socket says 172.21.15.86:34930) 2024-02-19T22:04:20.177 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:20.173+0000 7fd198827700 1 -- 172.21.15.86:0/1426137837 learned_addr learned my addr 172.21.15.86:0/1426137837 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T22:04:20.177 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:20.173+0000 7fd198827700 1 -- 172.21.15.86:0/1426137837 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd188005040 con 0x7fd1940f51a0 2024-02-19T22:04:20.177 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:20.173+0000 7fd198827700 1 --2- 172.21.15.86:0/1426137837 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd1940f51a0 0x7fd1940f2f20 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0x7fd1880089f0 tx=0x7fd18800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:04:20.177 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:20.173+0000 7fd1917fa700 1 -- 172.21.15.86:0/1426137837 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd188009320 con 0x7fd1940f51a0 2024-02-19T22:04:20.177 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:20.173+0000 7fd1917fa700 1 -- 172.21.15.86:0/1426137837 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd188013380 con 0x7fd1940f51a0 2024-02-19T22:04:20.177 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:20.173+0000 7fd199829700 1 -- 172.21.15.86:0/1426137837 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1940f1540 con 0x7fd1940f51a0 2024-02-19T22:04:20.177 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:20.173+0000 7fd1917fa700 1 -- 172.21.15.86:0/1426137837 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd188014e20 con 0x7fd1940f51a0 2024-02-19T22:04:20.178 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:20.173+0000 7fd199829700 1 -- 172.21.15.86:0/1426137837 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1940f19d0 con 0x7fd1940f51a0 2024-02-19T22:04:20.178 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:20.174+0000 7fd1917fa700 1 -- 172.21.15.86:0/1426137837 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd1880095f0 con 0x7fd1940f51a0 2024-02-19T22:04:20.178 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:20.174+0000 7fd199829700 1 -- 172.21.15.86:0/1426137837 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd194059100 con 0x7fd1940f51a0 2024-02-19T22:04:20.178 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:20.174+0000 7fd1917fa700 1 --2- 172.21.15.86:0/1426137837 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fd18402ff60 0x7fd184032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:04:20.178 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:20.174+0000 7fd1917fa700 1 -- 172.21.15.86:0/1426137837 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd188040e40 con 0x7fd1940f51a0 2024-02-19T22:04:20.178 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:20.174+0000 7fd193fff700 1 --2- 172.21.15.86:0/1426137837 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fd18402ff60 0x7fd184032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:04:20.178 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:20.175+0000 7fd193fff700 1 --2- 172.21.15.86:0/1426137837 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fd18402ff60 0x7fd184032420 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7fd1800096a0 tx=0x7fd180006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:04:20.178 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:20.179+0000 7fd1917fa700 1 -- 172.21.15.86:0/1426137837 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd188014420 con 0x7fd1940f51a0 2024-02-19T22:04:20.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:20.384+0000 7fd199829700 1 -- 172.21.15.86:0/1426137837 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd194045ad0 con 0x7fd1940f51a0 2024-02-19T22:04:20.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:20.385+0000 7fd1917fa700 1 -- 172.21.15.86:0/1426137837 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd188010230 con 0x7fd1940f51a0 2024-02-19T22:04:20.385 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T22:04:20.385 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:04:20.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:20.386+0000 7fd199829700 1 -- 172.21.15.86:0/1426137837 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fd18402ff60 msgr2=0x7fd184032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:04:20.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:20.386+0000 7fd199829700 1 --2- 172.21.15.86:0/1426137837 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fd18402ff60 0x7fd184032420 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7fd1800096a0 tx=0x7fd180006b40).stop 2024-02-19T22:04:20.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:20.386+0000 7fd199829700 1 -- 172.21.15.86:0/1426137837 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd1940f51a0 msgr2=0x7fd1940f2f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:04:20.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:20.386+0000 7fd199829700 1 --2- 172.21.15.86:0/1426137837 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd1940f51a0 0x7fd1940f2f20 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0x7fd1880089f0 tx=0x7fd18800b040).stop 2024-02-19T22:04:20.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:20.386+0000 7fd199829700 1 -- 172.21.15.86:0/1426137837 shutdown_connections 2024-02-19T22:04:20.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:20.386+0000 7fd199829700 1 --2- 172.21.15.86:0/1426137837 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fd18402ff60 0x7fd184032420 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:04:20.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:20.386+0000 7fd199829700 1 --2- 172.21.15.86:0/1426137837 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd1940f51a0 0x7fd1940f2f20 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:04:20.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:20.386+0000 7fd199829700 1 -- 172.21.15.86:0/1426137837 >> 172.21.15.86:0/1426137837 conn(0x7fd1940f09a0 msgr2=0x7fd19405fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:04:20.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:20.386+0000 7fd199829700 1 -- 172.21.15.86:0/1426137837 shutdown_connections 2024-02-19T22:04:20.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:20.386+0000 7fd199829700 1 -- 172.21.15.86:0/1426137837 wait complete. 2024-02-19T22:04:20.387 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T22:04:20.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:04:20 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:04:19.507688+0000 mgr.smithi053.fyhcdi (mgr.14162) 240 : cluster [DBG] pgmap v226: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:04:20.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:04:20 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:04:20.386033+0000 mon.smithi053 (mon.0) 253 : audit [DBG] from='client.? 172.21.15.86:0/1426137837' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:04:22.165 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:04:22.165 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:04:23.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:04:22 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:04:21.508034+0000 mgr.smithi053.fyhcdi (mgr.14162) 241 : cluster [DBG] pgmap v227: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:04:24.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:24.083+0000 7f64bfcf0700 1 -- 172.21.15.86:0/1213654483 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f64b80f6510 msgr2=0x7f64b80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:04:24.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:24.084+0000 7f64bfcf0700 1 --2- 172.21.15.86:0/1213654483 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f64b80f6510 0x7f64b80f6930 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 rx=0x7f64a8009fb0 tx=0x7f64a8009b30).stop 2024-02-19T22:04:24.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:24.084+0000 7f64bfcf0700 1 -- 172.21.15.86:0/1213654483 shutdown_connections 2024-02-19T22:04:24.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:24.084+0000 7f64bfcf0700 1 --2- 172.21.15.86:0/1213654483 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f64b80f6510 0x7f64b80f6930 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:04:24.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:24.084+0000 7f64bfcf0700 1 -- 172.21.15.86:0/1213654483 >> 172.21.15.86:0/1213654483 conn(0x7f64b80f1c70 msgr2=0x7f64b80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:04:24.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:24.084+0000 7f64bfcf0700 1 -- 172.21.15.86:0/1213654483 shutdown_connections 2024-02-19T22:04:24.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:24.084+0000 7f64bfcf0700 1 -- 172.21.15.86:0/1213654483 wait complete. 2024-02-19T22:04:24.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:24.084+0000 7f64bfcf0700 1 Processor -- start 2024-02-19T22:04:24.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:24.084+0000 7f64bfcf0700 1 -- start start 2024-02-19T22:04:24.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:24.085+0000 7f64bfcf0700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f64b80f6510 0x7f64b81067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:04:24.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:24.085+0000 7f64bfcf0700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64b8106cd0 con 0x7f64b80f6510 2024-02-19T22:04:24.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:24.085+0000 7f64bda8c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f64b80f6510 0x7f64b81067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:04:24.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:24.085+0000 7f64bda8c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f64b80f6510 0x7f64b81067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:34948/0 (socket says 172.21.15.86:34948) 2024-02-19T22:04:24.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:24.085+0000 7f64bda8c700 1 -- 172.21.15.86:0/272793869 learned_addr learned my addr 172.21.15.86:0/272793869 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T22:04:24.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:24.085+0000 7f64bda8c700 1 -- 172.21.15.86:0/272793869 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64a8005040 con 0x7f64b80f6510 2024-02-19T22:04:24.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:24.085+0000 7f64bda8c700 1 --2- 172.21.15.86:0/272793869 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f64b80f6510 0x7f64b81067e0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7f64a80089f0 tx=0x7f64a800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:04:24.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:24.085+0000 7f64ae7fc700 1 -- 172.21.15.86:0/272793869 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f64a8009320 con 0x7f64b80f6510 2024-02-19T22:04:24.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:24.085+0000 7f64ae7fc700 1 -- 172.21.15.86:0/272793869 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f64a8013380 con 0x7f64b80f6510 2024-02-19T22:04:24.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:24.085+0000 7f64bfcf0700 1 -- 172.21.15.86:0/272793869 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64b8106ed0 con 0x7f64b80f6510 2024-02-19T22:04:24.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:24.085+0000 7f64ae7fc700 1 -- 172.21.15.86:0/272793869 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f64a801ed80 con 0x7f64b80f6510 2024-02-19T22:04:24.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:24.086+0000 7f64bfcf0700 1 -- 172.21.15.86:0/272793869 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64b8107360 con 0x7f64b80f6510 2024-02-19T22:04:24.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:24.086+0000 7f64ae7fc700 1 -- 172.21.15.86:0/272793869 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f64a801e420 con 0x7f64b80f6510 2024-02-19T22:04:24.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:24.086+0000 7f64bfcf0700 1 -- 172.21.15.86:0/272793869 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64b8045ad0 con 0x7f64b80f6510 2024-02-19T22:04:24.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:24.087+0000 7f64ae7fc700 1 --2- 172.21.15.86:0/272793869 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f64a402ff60 0x7f64a4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:04:24.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:24.087+0000 7f64ae7fc700 1 -- 172.21.15.86:0/272793869 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f64a8041090 con 0x7f64b80f6510 2024-02-19T22:04:24.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:24.087+0000 7f64bd28b700 1 --2- 172.21.15.86:0/272793869 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f64a402ff60 0x7f64a4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:04:24.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:24.087+0000 7f64bd28b700 1 --2- 172.21.15.86:0/272793869 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f64a402ff60 0x7f64a4032420 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f64b40096a0 tx=0x7f64b4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:04:24.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:24.091+0000 7f64ae7fc700 1 -- 172.21.15.86:0/272793869 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f64a80095f0 con 0x7f64b80f6510 2024-02-19T22:04:24.294 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:24.294+0000 7f64bfcf0700 1 -- 172.21.15.86:0/272793869 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f64b8108210 con 0x7f64b80f6510 2024-02-19T22:04:24.295 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:24.295+0000 7f64ae7fc700 1 -- 172.21.15.86:0/272793869 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f64a8004490 con 0x7f64b80f6510 2024-02-19T22:04:24.295 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T22:04:24.295 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:04:24.297 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:24.297+0000 7f64bfcf0700 1 -- 172.21.15.86:0/272793869 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f64a402ff60 msgr2=0x7f64a4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:04:24.297 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:24.297+0000 7f64bfcf0700 1 --2- 172.21.15.86:0/272793869 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f64a402ff60 0x7f64a4032420 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f64b40096a0 tx=0x7f64b4006b40).stop 2024-02-19T22:04:24.297 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:24.297+0000 7f64bfcf0700 1 -- 172.21.15.86:0/272793869 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f64b80f6510 msgr2=0x7f64b81067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:04:24.297 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:24.297+0000 7f64bfcf0700 1 --2- 172.21.15.86:0/272793869 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f64b80f6510 0x7f64b81067e0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7f64a80089f0 tx=0x7f64a800b040).stop 2024-02-19T22:04:24.297 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:24.297+0000 7f64bfcf0700 1 -- 172.21.15.86:0/272793869 shutdown_connections 2024-02-19T22:04:24.297 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:24.297+0000 7f64bfcf0700 1 --2- 172.21.15.86:0/272793869 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f64a402ff60 0x7f64a4032420 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:04:24.297 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:24.297+0000 7f64bfcf0700 1 --2- 172.21.15.86:0/272793869 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f64b80f6510 0x7f64b81067e0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:04:24.297 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:24.297+0000 7f64bfcf0700 1 -- 172.21.15.86:0/272793869 >> 172.21.15.86:0/272793869 conn(0x7f64b80f1c70 msgr2=0x7f64b80f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:04:24.297 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:24.297+0000 7f64bfcf0700 1 -- 172.21.15.86:0/272793869 shutdown_connections 2024-02-19T22:04:24.298 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:24.297+0000 7f64bfcf0700 1 -- 172.21.15.86:0/272793869 wait complete. 2024-02-19T22:04:24.298 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T22:04:25.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:04:24 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:04:23.508337+0000 mgr.smithi053.fyhcdi (mgr.14162) 242 : cluster [DBG] pgmap v228: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:04:25.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:04:24 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:04:24.296448+0000 mon.smithi053 (mon.0) 254 : audit [DBG] from='client.? 172.21.15.86:0/272793869' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:04:25.939 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:04:25.940 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:04:27.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:04:26 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:04:25.508675+0000 mgr.smithi053.fyhcdi (mgr.14162) 243 : cluster [DBG] pgmap v229: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:04:27.995 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:27.852+0000 7f1cf2260700 1 -- 172.21.15.86:0/2553352794 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1cec0f6480 msgr2=0x7f1cec0f68a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:04:27.995 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:27.852+0000 7f1cf2260700 1 --2- 172.21.15.86:0/2553352794 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1cec0f6480 0x7f1cec0f68a0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 rx=0x7f1ce0009fb0 tx=0x7f1ce0009b30).stop 2024-02-19T22:04:27.995 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:27.852+0000 7f1cf2260700 1 -- 172.21.15.86:0/2553352794 shutdown_connections 2024-02-19T22:04:27.995 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:27.852+0000 7f1cf2260700 1 --2- 172.21.15.86:0/2553352794 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1cec0f6480 0x7f1cec0f68a0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:04:27.995 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:27.852+0000 7f1cf2260700 1 -- 172.21.15.86:0/2553352794 >> 172.21.15.86:0/2553352794 conn(0x7f1cec0f1c00 msgr2=0x7f1cec0f4020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:04:27.996 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:27.852+0000 7f1cf2260700 1 -- 172.21.15.86:0/2553352794 shutdown_connections 2024-02-19T22:04:27.996 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:27.852+0000 7f1cf2260700 1 -- 172.21.15.86:0/2553352794 wait complete. 2024-02-19T22:04:27.996 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:27.853+0000 7f1cf2260700 1 Processor -- start 2024-02-19T22:04:27.996 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:27.853+0000 7f1cf2260700 1 -- start start 2024-02-19T22:04:27.996 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:27.853+0000 7f1cf2260700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1cec0f6480 0x7f1cec1067f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:04:27.996 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:27.853+0000 7f1cf2260700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cec106ce0 con 0x7f1cec0f6480 2024-02-19T22:04:27.996 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:27.854+0000 7f1ceb7fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1cec0f6480 0x7f1cec1067f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:04:27.996 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:27.854+0000 7f1ceb7fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1cec0f6480 0x7f1cec1067f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:34960/0 (socket says 172.21.15.86:34960) 2024-02-19T22:04:27.996 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:27.854+0000 7f1ceb7fe700 1 -- 172.21.15.86:0/4133054635 learned_addr learned my addr 172.21.15.86:0/4133054635 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T22:04:27.996 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:27.854+0000 7f1ceb7fe700 1 -- 172.21.15.86:0/4133054635 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ce0005040 con 0x7f1cec0f6480 2024-02-19T22:04:27.996 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:27.854+0000 7f1ceb7fe700 1 --2- 172.21.15.86:0/4133054635 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1cec0f6480 0x7f1cec1067f0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7f1ce0005010 tx=0x7f1ce000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:04:27.996 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:27.854+0000 7f1cd3fff700 1 -- 172.21.15.86:0/4133054635 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1ce0009320 con 0x7f1cec0f6480 2024-02-19T22:04:27.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:27.854+0000 7f1cf2260700 1 -- 172.21.15.86:0/4133054635 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cec106ee0 con 0x7f1cec0f6480 2024-02-19T22:04:27.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:27.854+0000 7f1cd3fff700 1 -- 172.21.15.86:0/4133054635 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1ce0013380 con 0x7f1cec0f6480 2024-02-19T22:04:27.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:27.854+0000 7f1cd3fff700 1 -- 172.21.15.86:0/4133054635 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1ce0014e50 con 0x7f1cec0f6480 2024-02-19T22:04:27.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:27.854+0000 7f1cf2260700 1 -- 172.21.15.86:0/4133054635 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cec107370 con 0x7f1cec0f6480 2024-02-19T22:04:27.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:27.855+0000 7f1cd3fff700 1 -- 172.21.15.86:0/4133054635 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1ce0014420 con 0x7f1cec0f6480 2024-02-19T22:04:27.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:27.855+0000 7f1cf2260700 1 -- 172.21.15.86:0/4133054635 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1cec045ad0 con 0x7f1cec0f6480 2024-02-19T22:04:27.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:27.855+0000 7f1cd3fff700 1 --2- 172.21.15.86:0/4133054635 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f1cd402ff10 0x7f1cd40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:04:27.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:27.855+0000 7f1cd3fff700 1 -- 172.21.15.86:0/4133054635 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1ce0040e30 con 0x7f1cec0f6480 2024-02-19T22:04:27.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:27.855+0000 7f1ceaffd700 1 --2- 172.21.15.86:0/4133054635 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f1cd402ff10 0x7f1cd40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:04:27.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:27.856+0000 7f1ceaffd700 1 --2- 172.21.15.86:0/4133054635 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f1cd402ff10 0x7f1cd40323d0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f1cdc0096a0 tx=0x7f1cdc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:04:27.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:27.860+0000 7f1cd3fff700 1 -- 172.21.15.86:0/4133054635 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1ce00095f0 con 0x7f1cec0f6480 2024-02-19T22:04:28.067 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:28.068+0000 7f1cf2260700 1 -- 172.21.15.86:0/4133054635 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1cec0fc090 con 0x7f1cec0f6480 2024-02-19T22:04:28.068 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:28.068+0000 7f1cd3fff700 1 -- 172.21.15.86:0/4133054635 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1ce001db70 con 0x7f1cec0f6480 2024-02-19T22:04:28.068 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T22:04:28.068 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:04:28.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:28.070+0000 7f1cf2260700 1 -- 172.21.15.86:0/4133054635 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f1cd402ff10 msgr2=0x7f1cd40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:04:28.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:28.070+0000 7f1cf2260700 1 --2- 172.21.15.86:0/4133054635 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f1cd402ff10 0x7f1cd40323d0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f1cdc0096a0 tx=0x7f1cdc006b40).stop 2024-02-19T22:04:28.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:28.070+0000 7f1cf2260700 1 -- 172.21.15.86:0/4133054635 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1cec0f6480 msgr2=0x7f1cec1067f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:04:28.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:28.070+0000 7f1cf2260700 1 --2- 172.21.15.86:0/4133054635 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1cec0f6480 0x7f1cec1067f0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7f1ce0005010 tx=0x7f1ce000b040).stop 2024-02-19T22:04:28.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:28.070+0000 7f1cf2260700 1 -- 172.21.15.86:0/4133054635 shutdown_connections 2024-02-19T22:04:28.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:28.070+0000 7f1cf2260700 1 --2- 172.21.15.86:0/4133054635 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f1cd402ff10 0x7f1cd40323d0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:04:28.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:28.070+0000 7f1cf2260700 1 --2- 172.21.15.86:0/4133054635 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1cec0f6480 0x7f1cec1067f0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:04:28.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:28.070+0000 7f1cf2260700 1 -- 172.21.15.86:0/4133054635 >> 172.21.15.86:0/4133054635 conn(0x7f1cec0f1c00 msgr2=0x7f1cec0f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:04:28.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:28.070+0000 7f1cf2260700 1 -- 172.21.15.86:0/4133054635 shutdown_connections 2024-02-19T22:04:28.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:28.070+0000 7f1cf2260700 1 -- 172.21.15.86:0/4133054635 wait complete. 2024-02-19T22:04:28.071 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T22:04:29.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:04:28 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:04:27.508940+0000 mgr.smithi053.fyhcdi (mgr.14162) 244 : cluster [DBG] pgmap v230: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:04:29.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:04:28 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:04:28.069552+0000 mon.smithi053 (mon.0) 255 : audit [DBG] from='client.? 172.21.15.86:0/4133054635' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:04:29.839 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:04:29.840 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:04:30.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:04:30 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:04:29.509220+0000 mgr.smithi053.fyhcdi (mgr.14162) 245 : cluster [DBG] pgmap v231: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:04:31.740 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:31.739+0000 7fa215c9c700 1 -- 172.21.15.86:0/1880556267 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa2100f51a0 msgr2=0x7fa2100f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:04:31.740 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:31.739+0000 7fa215c9c700 1 --2- 172.21.15.86:0/1880556267 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa2100f51a0 0x7fa2100f55c0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 rx=0x7fa204009fb0 tx=0x7fa204009b30).stop 2024-02-19T22:04:31.740 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:31.739+0000 7fa215c9c700 1 -- 172.21.15.86:0/1880556267 shutdown_connections 2024-02-19T22:04:31.740 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:31.739+0000 7fa215c9c700 1 --2- 172.21.15.86:0/1880556267 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa2100f51a0 0x7fa2100f55c0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:04:31.741 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:31.739+0000 7fa215c9c700 1 -- 172.21.15.86:0/1880556267 >> 172.21.15.86:0/1880556267 conn(0x7fa2100f09a0 msgr2=0x7fa2100f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:04:31.741 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:31.739+0000 7fa215c9c700 1 -- 172.21.15.86:0/1880556267 shutdown_connections 2024-02-19T22:04:31.741 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:31.739+0000 7fa215c9c700 1 -- 172.21.15.86:0/1880556267 wait complete. 2024-02-19T22:04:31.741 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:31.739+0000 7fa215c9c700 1 Processor -- start 2024-02-19T22:04:31.741 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:31.739+0000 7fa215c9c700 1 -- start start 2024-02-19T22:04:31.741 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:31.740+0000 7fa215c9c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa2100f51a0 0x7fa2101092d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:04:31.741 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:31.740+0000 7fa215c9c700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa210060240 con 0x7fa2100f51a0 2024-02-19T22:04:31.741 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:31.740+0000 7fa214c9a700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa2100f51a0 0x7fa2101092d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:04:31.741 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:31.740+0000 7fa214c9a700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa2100f51a0 0x7fa2101092d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:57688/0 (socket says 172.21.15.86:57688) 2024-02-19T22:04:31.741 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:31.740+0000 7fa214c9a700 1 -- 172.21.15.86:0/1595981235 learned_addr learned my addr 172.21.15.86:0/1595981235 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T22:04:31.741 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:31.740+0000 7fa214c9a700 1 -- 172.21.15.86:0/1595981235 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa204005040 con 0x7fa2100f51a0 2024-02-19T22:04:31.741 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:31.740+0000 7fa214c9a700 1 --2- 172.21.15.86:0/1595981235 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa2100f51a0 0x7fa2101092d0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0x7fa2040089f0 tx=0x7fa20400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:04:31.741 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:31.740+0000 7fa20d7fa700 1 -- 172.21.15.86:0/1595981235 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa204009320 con 0x7fa2100f51a0 2024-02-19T22:04:31.742 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:31.740+0000 7fa215c9c700 1 -- 172.21.15.86:0/1595981235 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa210060440 con 0x7fa2100f51a0 2024-02-19T22:04:31.742 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:31.741+0000 7fa20d7fa700 1 -- 172.21.15.86:0/1595981235 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa204013380 con 0x7fa2100f51a0 2024-02-19T22:04:31.742 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:31.741+0000 7fa20d7fa700 1 -- 172.21.15.86:0/1595981235 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa204014e20 con 0x7fa2100f51a0 2024-02-19T22:04:31.742 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:31.741+0000 7fa215c9c700 1 -- 172.21.15.86:0/1595981235 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa210109980 con 0x7fa2100f51a0 2024-02-19T22:04:31.743 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:31.741+0000 7fa20d7fa700 1 -- 172.21.15.86:0/1595981235 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa2040263a0 con 0x7fa2100f51a0 2024-02-19T22:04:31.743 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:31.741+0000 7fa20d7fa700 1 --2- 172.21.15.86:0/1595981235 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fa20002ff10 0x7fa2000323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:04:31.743 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:31.741+0000 7fa20d7fa700 1 -- 172.21.15.86:0/1595981235 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa204040ec0 con 0x7fa2100f51a0 2024-02-19T22:04:31.743 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:31.742+0000 7fa215c9c700 1 -- 172.21.15.86:0/1595981235 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa210045ad0 con 0x7fa2100f51a0 2024-02-19T22:04:31.743 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:31.742+0000 7fa20ffff700 1 --2- 172.21.15.86:0/1595981235 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fa20002ff10 0x7fa2000323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:04:31.743 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:31.742+0000 7fa20ffff700 1 --2- 172.21.15.86:0/1595981235 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fa20002ff10 0x7fa2000323d0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7fa1fc0096a0 tx=0x7fa1fc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:04:31.746 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:31.746+0000 7fa20d7fa700 1 -- 172.21.15.86:0/1595981235 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa204012920 con 0x7fa2100f51a0 2024-02-19T22:04:31.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:31.949+0000 7fa215c9c700 1 -- 172.21.15.86:0/1595981235 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa21010a060 con 0x7fa2100f51a0 2024-02-19T22:04:31.949 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:31.949+0000 7fa20d7fa700 1 -- 172.21.15.86:0/1595981235 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa20402b020 con 0x7fa2100f51a0 2024-02-19T22:04:31.949 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T22:04:31.949 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:04:31.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:31.950+0000 7fa215c9c700 1 -- 172.21.15.86:0/1595981235 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fa20002ff10 msgr2=0x7fa2000323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:04:31.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:31.950+0000 7fa215c9c700 1 --2- 172.21.15.86:0/1595981235 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fa20002ff10 0x7fa2000323d0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7fa1fc0096a0 tx=0x7fa1fc006b40).stop 2024-02-19T22:04:31.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:31.950+0000 7fa215c9c700 1 -- 172.21.15.86:0/1595981235 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa2100f51a0 msgr2=0x7fa2101092d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:04:31.951 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:31.950+0000 7fa215c9c700 1 --2- 172.21.15.86:0/1595981235 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa2100f51a0 0x7fa2101092d0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0x7fa2040089f0 tx=0x7fa20400b040).stop 2024-02-19T22:04:31.951 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:31.951+0000 7fa215c9c700 1 -- 172.21.15.86:0/1595981235 shutdown_connections 2024-02-19T22:04:31.951 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:31.951+0000 7fa215c9c700 1 --2- 172.21.15.86:0/1595981235 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fa20002ff10 0x7fa2000323d0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:04:31.951 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:31.951+0000 7fa215c9c700 1 --2- 172.21.15.86:0/1595981235 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa2100f51a0 0x7fa2101092d0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:04:31.951 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:31.951+0000 7fa215c9c700 1 -- 172.21.15.86:0/1595981235 >> 172.21.15.86:0/1595981235 conn(0x7fa2100f09a0 msgr2=0x7fa2100f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:04:31.951 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:31.951+0000 7fa215c9c700 1 -- 172.21.15.86:0/1595981235 shutdown_connections 2024-02-19T22:04:31.951 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:31.951+0000 7fa215c9c700 1 -- 172.21.15.86:0/1595981235 wait complete. 2024-02-19T22:04:31.951 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T22:04:33.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:04:32 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:04:31.509567+0000 mgr.smithi053.fyhcdi (mgr.14162) 246 : cluster [DBG] pgmap v232: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:04:33.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:04:32 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:04:31.950429+0000 mon.smithi053 (mon.0) 256 : audit [DBG] from='client.? 172.21.15.86:0/1595981235' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:04:33.655 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:04:33.655 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:04:35.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:04:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:04:33.509846+0000 mgr.smithi053.fyhcdi (mgr.14162) 247 : cluster [DBG] pgmap v233: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:04:35.603 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:35.588+0000 7ff6dc97e700 1 -- 172.21.15.86:0/352102739 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff6d80f6510 msgr2=0x7ff6d80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:04:35.603 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:35.588+0000 7ff6dc97e700 1 --2- 172.21.15.86:0/352102739 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff6d80f6510 0x7ff6d80f6930 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 rx=0x7ff6cc009fb0 tx=0x7ff6cc009b30).stop 2024-02-19T22:04:35.603 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:35.588+0000 7ff6dc97e700 1 -- 172.21.15.86:0/352102739 shutdown_connections 2024-02-19T22:04:35.603 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:35.588+0000 7ff6dc97e700 1 --2- 172.21.15.86:0/352102739 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff6d80f6510 0x7ff6d80f6930 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:04:35.603 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:35.588+0000 7ff6dc97e700 1 -- 172.21.15.86:0/352102739 >> 172.21.15.86:0/352102739 conn(0x7ff6d80f1c70 msgr2=0x7ff6d80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:04:35.603 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:35.588+0000 7ff6dc97e700 1 -- 172.21.15.86:0/352102739 shutdown_connections 2024-02-19T22:04:35.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:35.588+0000 7ff6dc97e700 1 -- 172.21.15.86:0/352102739 wait complete. 2024-02-19T22:04:35.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:35.589+0000 7ff6dc97e700 1 Processor -- start 2024-02-19T22:04:35.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:35.589+0000 7ff6dc97e700 1 -- start start 2024-02-19T22:04:35.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:35.589+0000 7ff6dc97e700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff6d80f6510 0x7ff6d81067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:04:35.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:35.589+0000 7ff6dc97e700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6d8106cd0 con 0x7ff6d80f6510 2024-02-19T22:04:35.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:35.589+0000 7ff6d659c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff6d80f6510 0x7ff6d81067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:04:35.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:35.589+0000 7ff6d659c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff6d80f6510 0x7ff6d81067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:57716/0 (socket says 172.21.15.86:57716) 2024-02-19T22:04:35.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:35.589+0000 7ff6d659c700 1 -- 172.21.15.86:0/1605030993 learned_addr learned my addr 172.21.15.86:0/1605030993 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T22:04:35.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:35.590+0000 7ff6d659c700 1 -- 172.21.15.86:0/1605030993 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6cc005040 con 0x7ff6d80f6510 2024-02-19T22:04:35.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:35.590+0000 7ff6d659c700 1 --2- 172.21.15.86:0/1605030993 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff6d80f6510 0x7ff6d81067e0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0x7ff6cc005010 tx=0x7ff6cc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:04:35.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:35.590+0000 7ff6c6ffd700 1 -- 172.21.15.86:0/1605030993 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff6cc009320 con 0x7ff6d80f6510 2024-02-19T22:04:35.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:35.590+0000 7ff6c6ffd700 1 -- 172.21.15.86:0/1605030993 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff6cc013380 con 0x7ff6d80f6510 2024-02-19T22:04:35.605 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:35.590+0000 7ff6c6ffd700 1 -- 172.21.15.86:0/1605030993 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff6cc014e50 con 0x7ff6d80f6510 2024-02-19T22:04:35.605 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:35.590+0000 7ff6dc97e700 1 -- 172.21.15.86:0/1605030993 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6d8106ed0 con 0x7ff6d80f6510 2024-02-19T22:04:35.605 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:35.590+0000 7ff6dc97e700 1 -- 172.21.15.86:0/1605030993 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6d8107360 con 0x7ff6d80f6510 2024-02-19T22:04:35.605 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:35.591+0000 7ff6c6ffd700 1 -- 172.21.15.86:0/1605030993 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff6cc0095f0 con 0x7ff6d80f6510 2024-02-19T22:04:35.605 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:35.591+0000 7ff6c6ffd700 1 --2- 172.21.15.86:0/1605030993 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7ff6c002ff10 0x7ff6c00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:04:35.605 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:35.591+0000 7ff6dc97e700 1 -- 172.21.15.86:0/1605030993 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6d8045ad0 con 0x7ff6d80f6510 2024-02-19T22:04:35.605 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:35.591+0000 7ff6c6ffd700 1 -- 172.21.15.86:0/1605030993 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff6cc040e60 con 0x7ff6d80f6510 2024-02-19T22:04:35.605 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:35.591+0000 7ff6d5d9b700 1 --2- 172.21.15.86:0/1605030993 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7ff6c002ff10 0x7ff6c00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:04:35.605 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:35.591+0000 7ff6d5d9b700 1 --2- 172.21.15.86:0/1605030993 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7ff6c002ff10 0x7ff6c00323d0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7ff6c80096a0 tx=0x7ff6c8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:04:35.605 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:35.596+0000 7ff6c6ffd700 1 -- 172.21.15.86:0/1605030993 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff6cc0672b0 con 0x7ff6d80f6510 2024-02-19T22:04:35.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:35.801+0000 7ff6dc97e700 1 -- 172.21.15.86:0/1605030993 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff6d8108210 con 0x7ff6d80f6510 2024-02-19T22:04:35.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:35.802+0000 7ff6c6ffd700 1 -- 172.21.15.86:0/1605030993 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff6cc014420 con 0x7ff6d80f6510 2024-02-19T22:04:35.802 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T22:04:35.802 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:04:35.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:35.803+0000 7ff6dc97e700 1 -- 172.21.15.86:0/1605030993 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7ff6c002ff10 msgr2=0x7ff6c00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:04:35.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:35.803+0000 7ff6dc97e700 1 --2- 172.21.15.86:0/1605030993 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7ff6c002ff10 0x7ff6c00323d0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7ff6c80096a0 tx=0x7ff6c8006b40).stop 2024-02-19T22:04:35.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:35.803+0000 7ff6dc97e700 1 -- 172.21.15.86:0/1605030993 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff6d80f6510 msgr2=0x7ff6d81067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:04:35.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:35.803+0000 7ff6dc97e700 1 --2- 172.21.15.86:0/1605030993 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff6d80f6510 0x7ff6d81067e0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0x7ff6cc005010 tx=0x7ff6cc00b040).stop 2024-02-19T22:04:35.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:35.804+0000 7ff6dc97e700 1 -- 172.21.15.86:0/1605030993 shutdown_connections 2024-02-19T22:04:35.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:35.804+0000 7ff6dc97e700 1 --2- 172.21.15.86:0/1605030993 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7ff6c002ff10 0x7ff6c00323d0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:04:35.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:35.804+0000 7ff6dc97e700 1 --2- 172.21.15.86:0/1605030993 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff6d80f6510 0x7ff6d81067e0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:04:35.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:35.804+0000 7ff6dc97e700 1 -- 172.21.15.86:0/1605030993 >> 172.21.15.86:0/1605030993 conn(0x7ff6d80f1c70 msgr2=0x7ff6d80f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:04:35.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:35.804+0000 7ff6dc97e700 1 -- 172.21.15.86:0/1605030993 shutdown_connections 2024-02-19T22:04:35.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:35.804+0000 7ff6dc97e700 1 -- 172.21.15.86:0/1605030993 wait complete. 2024-02-19T22:04:35.804 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T22:04:37.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:04:36 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:04:35.510225+0000 mgr.smithi053.fyhcdi (mgr.14162) 248 : cluster [DBG] pgmap v234: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:04:37.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:04:36 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:04:35.604286+0000 mon.smithi053 (mon.0) 257 : audit [INF] from='mgr.14162 172.21.15.53:0/2860358250' entity='mgr.smithi053.fyhcdi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi053.fyhcdi/mirror_snapshot_schedule"}]: dispatch 2024-02-19T22:04:37.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:04:36 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:04:35.607429+0000 mon.smithi053 (mon.0) 258 : audit [INF] from='mgr.14162 172.21.15.53:0/2860358250' entity='mgr.smithi053.fyhcdi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi053.fyhcdi/trash_purge_schedule"}]: dispatch 2024-02-19T22:04:37.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:04:36 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:04:35.803305+0000 mon.smithi053 (mon.0) 259 : audit [DBG] from='client.? 172.21.15.86:0/1605030993' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:04:37.491 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:04:37.492 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:04:39.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:04:38 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:04:37.510538+0000 mgr.smithi053.fyhcdi (mgr.14162) 249 : cluster [DBG] pgmap v235: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:04:39.528 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:39.527+0000 7f20c214c700 1 -- 172.21.15.86:0/1960080398 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f20bc0f5410 msgr2=0x7f20bc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:04:39.528 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:39.527+0000 7f20c214c700 1 --2- 172.21.15.86:0/1960080398 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f20bc0f5410 0x7f20bc0f5830 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 rx=0x7f20b0009fb0 tx=0x7f20b0009b30).stop 2024-02-19T22:04:39.528 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:39.527+0000 7f20c214c700 1 -- 172.21.15.86:0/1960080398 shutdown_connections 2024-02-19T22:04:39.528 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:39.527+0000 7f20c214c700 1 --2- 172.21.15.86:0/1960080398 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f20bc0f5410 0x7f20bc0f5830 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:04:39.528 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:39.527+0000 7f20c214c700 1 -- 172.21.15.86:0/1960080398 >> 172.21.15.86:0/1960080398 conn(0x7f20bc0f0b70 msgr2=0x7f20bc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:04:39.528 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:39.527+0000 7f20c214c700 1 -- 172.21.15.86:0/1960080398 shutdown_connections 2024-02-19T22:04:39.528 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:39.527+0000 7f20c214c700 1 -- 172.21.15.86:0/1960080398 wait complete. 2024-02-19T22:04:39.528 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:39.528+0000 7f20c214c700 1 Processor -- start 2024-02-19T22:04:39.528 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:39.528+0000 7f20c214c700 1 -- start start 2024-02-19T22:04:39.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:39.528+0000 7f20c214c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f20bc0f5410 0x7f20bc1067d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:04:39.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:39.528+0000 7f20c214c700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20bc106cc0 con 0x7f20bc0f5410 2024-02-19T22:04:39.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:39.528+0000 7f20bb7fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f20bc0f5410 0x7f20bc1067d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:04:39.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:39.528+0000 7f20bb7fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f20bc0f5410 0x7f20bc1067d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:46006/0 (socket says 172.21.15.86:46006) 2024-02-19T22:04:39.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:39.528+0000 7f20bb7fe700 1 -- 172.21.15.86:0/697672630 learned_addr learned my addr 172.21.15.86:0/697672630 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T22:04:39.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:39.529+0000 7f20bb7fe700 1 -- 172.21.15.86:0/697672630 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20b0005040 con 0x7f20bc0f5410 2024-02-19T22:04:39.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:39.529+0000 7f20bb7fe700 1 --2- 172.21.15.86:0/697672630 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f20bc0f5410 0x7f20bc1067d0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0x7f20b0005010 tx=0x7f20b000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:04:39.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:39.529+0000 7f20a3fff700 1 -- 172.21.15.86:0/697672630 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f20b0009320 con 0x7f20bc0f5410 2024-02-19T22:04:39.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:39.529+0000 7f20c214c700 1 -- 172.21.15.86:0/697672630 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20bc106ec0 con 0x7f20bc0f5410 2024-02-19T22:04:39.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:39.529+0000 7f20a3fff700 1 -- 172.21.15.86:0/697672630 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f20b0013380 con 0x7f20bc0f5410 2024-02-19T22:04:39.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:39.529+0000 7f20a3fff700 1 -- 172.21.15.86:0/697672630 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f20b0014e50 con 0x7f20bc0f5410 2024-02-19T22:04:39.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:39.529+0000 7f20c214c700 1 -- 172.21.15.86:0/697672630 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20bc107350 con 0x7f20bc0f5410 2024-02-19T22:04:39.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:39.530+0000 7f20a3fff700 1 -- 172.21.15.86:0/697672630 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f20b0014420 con 0x7f20bc0f5410 2024-02-19T22:04:39.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:39.530+0000 7f20c214c700 1 -- 172.21.15.86:0/697672630 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20bc045ad0 con 0x7f20bc0f5410 2024-02-19T22:04:39.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:39.530+0000 7f20a3fff700 1 --2- 172.21.15.86:0/697672630 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f20a402ff10 0x7f20a40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:04:39.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:39.530+0000 7f20a3fff700 1 -- 172.21.15.86:0/697672630 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f20b0040f70 con 0x7f20bc0f5410 2024-02-19T22:04:39.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:39.530+0000 7f20baffd700 1 --2- 172.21.15.86:0/697672630 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f20a402ff10 0x7f20a40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:04:39.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:39.530+0000 7f20baffd700 1 --2- 172.21.15.86:0/697672630 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f20a402ff10 0x7f20a40323d0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f20ac0096a0 tx=0x7f20ac006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:04:39.533 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:39.535+0000 7f20a3fff700 1 -- 172.21.15.86:0/697672630 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f20b00095f0 con 0x7f20bc0f5410 2024-02-19T22:04:39.740 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:39.741+0000 7f20c214c700 1 -- 172.21.15.86:0/697672630 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f20bc108180 con 0x7f20bc0f5410 2024-02-19T22:04:39.740 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:39.742+0000 7f20a3fff700 1 -- 172.21.15.86:0/697672630 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f20b001db70 con 0x7f20bc0f5410 2024-02-19T22:04:39.741 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T22:04:39.741 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:04:39.742 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:39.744+0000 7f20c214c700 1 -- 172.21.15.86:0/697672630 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f20a402ff10 msgr2=0x7f20a40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:04:39.742 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:39.744+0000 7f20c214c700 1 --2- 172.21.15.86:0/697672630 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f20a402ff10 0x7f20a40323d0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f20ac0096a0 tx=0x7f20ac006b40).stop 2024-02-19T22:04:39.742 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:39.744+0000 7f20c214c700 1 -- 172.21.15.86:0/697672630 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f20bc0f5410 msgr2=0x7f20bc1067d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:04:39.742 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:39.744+0000 7f20c214c700 1 --2- 172.21.15.86:0/697672630 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f20bc0f5410 0x7f20bc1067d0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0x7f20b0005010 tx=0x7f20b000b040).stop 2024-02-19T22:04:39.743 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:39.744+0000 7f20c214c700 1 -- 172.21.15.86:0/697672630 shutdown_connections 2024-02-19T22:04:39.743 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:39.744+0000 7f20c214c700 1 --2- 172.21.15.86:0/697672630 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f20a402ff10 0x7f20a40323d0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:04:39.743 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:39.744+0000 7f20c214c700 1 --2- 172.21.15.86:0/697672630 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f20bc0f5410 0x7f20bc1067d0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:04:39.743 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:39.744+0000 7f20c214c700 1 -- 172.21.15.86:0/697672630 >> 172.21.15.86:0/697672630 conn(0x7f20bc0f0b70 msgr2=0x7f20bc05fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:04:39.743 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:39.744+0000 7f20c214c700 1 -- 172.21.15.86:0/697672630 shutdown_connections 2024-02-19T22:04:39.743 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:39.744+0000 7f20c214c700 1 -- 172.21.15.86:0/697672630 wait complete. 2024-02-19T22:04:39.743 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T22:04:40.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:04:40 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:04:39.510927+0000 mgr.smithi053.fyhcdi (mgr.14162) 250 : cluster [DBG] pgmap v236: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:04:40.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:04:40 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:04:39.742334+0000 mon.smithi053 (mon.0) 260 : audit [DBG] from='client.? 172.21.15.86:0/697672630' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:04:41.564 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:04:41.564 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:04:43.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:04:42 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:04:41.511246+0000 mgr.smithi053.fyhcdi (mgr.14162) 251 : cluster [DBG] pgmap v237: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:04:43.470 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:43.470+0000 7fd3fca12700 1 -- 172.21.15.86:0/3518465900 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd3f80f5410 msgr2=0x7fd3f80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:04:43.470 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:43.470+0000 7fd3fca12700 1 --2- 172.21.15.86:0/3518465900 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd3f80f5410 0x7fd3f80f5830 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 rx=0x7fd3ec009fb0 tx=0x7fd3ec009b30).stop 2024-02-19T22:04:43.471 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:43.470+0000 7fd3fca12700 1 -- 172.21.15.86:0/3518465900 shutdown_connections 2024-02-19T22:04:43.471 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:43.470+0000 7fd3fca12700 1 --2- 172.21.15.86:0/3518465900 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd3f80f5410 0x7fd3f80f5830 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:04:43.471 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:43.470+0000 7fd3fca12700 1 -- 172.21.15.86:0/3518465900 >> 172.21.15.86:0/3518465900 conn(0x7fd3f80f0b70 msgr2=0x7fd3f80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:04:43.471 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:43.470+0000 7fd3fca12700 1 -- 172.21.15.86:0/3518465900 shutdown_connections 2024-02-19T22:04:43.471 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:43.470+0000 7fd3fca12700 1 -- 172.21.15.86:0/3518465900 wait complete. 2024-02-19T22:04:43.471 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:43.471+0000 7fd3fca12700 1 Processor -- start 2024-02-19T22:04:43.471 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:43.471+0000 7fd3fca12700 1 -- start start 2024-02-19T22:04:43.471 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:43.471+0000 7fd3fca12700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd3f80f5410 0x7fd3f80fd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:04:43.471 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:43.471+0000 7fd3fca12700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3f80fb960 con 0x7fd3f80f5410 2024-02-19T22:04:43.471 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:43.471+0000 7fd3f659c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd3f80f5410 0x7fd3f80fd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:04:43.471 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:43.471+0000 7fd3f659c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd3f80f5410 0x7fd3f80fd4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:46014/0 (socket says 172.21.15.86:46014) 2024-02-19T22:04:43.471 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:43.471+0000 7fd3f659c700 1 -- 172.21.15.86:0/731975609 learned_addr learned my addr 172.21.15.86:0/731975609 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T22:04:43.471 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:43.471+0000 7fd3f659c700 1 -- 172.21.15.86:0/731975609 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3ec005040 con 0x7fd3f80f5410 2024-02-19T22:04:43.472 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:43.472+0000 7fd3f659c700 1 --2- 172.21.15.86:0/731975609 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd3f80f5410 0x7fd3f80fd4e0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0x7fd3ec005010 tx=0x7fd3ec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:04:43.472 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:43.472+0000 7fd3e6ffd700 1 -- 172.21.15.86:0/731975609 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd3ec009320 con 0x7fd3f80f5410 2024-02-19T22:04:43.472 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:43.472+0000 7fd3fca12700 1 -- 172.21.15.86:0/731975609 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3f80fbb00 con 0x7fd3f80f5410 2024-02-19T22:04:43.472 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:43.472+0000 7fd3e6ffd700 1 -- 172.21.15.86:0/731975609 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd3ec013380 con 0x7fd3f80f5410 2024-02-19T22:04:43.472 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:43.472+0000 7fd3e6ffd700 1 -- 172.21.15.86:0/731975609 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd3ec014e50 con 0x7fd3f80f5410 2024-02-19T22:04:43.472 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:43.472+0000 7fd3fca12700 1 -- 172.21.15.86:0/731975609 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3f80fbf90 con 0x7fd3f80f5410 2024-02-19T22:04:43.473 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:43.473+0000 7fd3e6ffd700 1 -- 172.21.15.86:0/731975609 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd3ec014420 con 0x7fd3f80f5410 2024-02-19T22:04:43.473 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:43.473+0000 7fd3fca12700 1 -- 172.21.15.86:0/731975609 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3f8059100 con 0x7fd3f80f5410 2024-02-19T22:04:43.473 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:43.473+0000 7fd3e6ffd700 1 --2- 172.21.15.86:0/731975609 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fd3e002ff10 0x7fd3e00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:04:43.473 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:43.473+0000 7fd3e6ffd700 1 -- 172.21.15.86:0/731975609 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd3ec040e30 con 0x7fd3f80f5410 2024-02-19T22:04:43.473 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:43.473+0000 7fd3f5d9b700 1 --2- 172.21.15.86:0/731975609 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fd3e002ff10 0x7fd3e00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:04:43.473 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:43.473+0000 7fd3f5d9b700 1 --2- 172.21.15.86:0/731975609 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fd3e002ff10 0x7fd3e00323d0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7fd3e80096a0 tx=0x7fd3e8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:04:43.476 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:43.478+0000 7fd3e6ffd700 1 -- 172.21.15.86:0/731975609 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd3ec0095f0 con 0x7fd3f80f5410 2024-02-19T22:04:43.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:43.681+0000 7fd3fca12700 1 -- 172.21.15.86:0/731975609 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd3f80fafa0 con 0x7fd3f80f5410 2024-02-19T22:04:43.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:43.682+0000 7fd3e6ffd700 1 -- 172.21.15.86:0/731975609 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd3ec01db70 con 0x7fd3f80f5410 2024-02-19T22:04:43.681 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T22:04:43.681 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:04:43.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:43.683+0000 7fd3fca12700 1 -- 172.21.15.86:0/731975609 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fd3e002ff10 msgr2=0x7fd3e00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:04:43.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:43.683+0000 7fd3fca12700 1 --2- 172.21.15.86:0/731975609 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fd3e002ff10 0x7fd3e00323d0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7fd3e80096a0 tx=0x7fd3e8006b40).stop 2024-02-19T22:04:43.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:43.683+0000 7fd3fca12700 1 -- 172.21.15.86:0/731975609 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd3f80f5410 msgr2=0x7fd3f80fd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:04:43.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:43.683+0000 7fd3fca12700 1 --2- 172.21.15.86:0/731975609 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd3f80f5410 0x7fd3f80fd4e0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0x7fd3ec005010 tx=0x7fd3ec00b040).stop 2024-02-19T22:04:43.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:43.683+0000 7fd3fca12700 1 -- 172.21.15.86:0/731975609 shutdown_connections 2024-02-19T22:04:43.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:43.683+0000 7fd3fca12700 1 --2- 172.21.15.86:0/731975609 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fd3e002ff10 0x7fd3e00323d0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:04:43.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:43.683+0000 7fd3fca12700 1 --2- 172.21.15.86:0/731975609 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd3f80f5410 0x7fd3f80fd4e0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:04:43.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:43.683+0000 7fd3fca12700 1 -- 172.21.15.86:0/731975609 >> 172.21.15.86:0/731975609 conn(0x7fd3f80f0b70 msgr2=0x7fd3f805fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:04:43.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:43.683+0000 7fd3fca12700 1 -- 172.21.15.86:0/731975609 shutdown_connections 2024-02-19T22:04:43.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:43.683+0000 7fd3fca12700 1 -- 172.21.15.86:0/731975609 wait complete. 2024-02-19T22:04:43.684 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T22:04:45.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:04:44 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:04:43.511612+0000 mgr.smithi053.fyhcdi (mgr.14162) 252 : cluster [DBG] pgmap v238: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:04:45.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:04:44 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:04:43.682153+0000 mon.smithi053 (mon.0) 261 : audit [DBG] from='client.? 172.21.15.86:0/731975609' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:04:45.395 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:04:45.395 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:04:47.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:04:46 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:04:45.511993+0000 mgr.smithi053.fyhcdi (mgr.14162) 253 : cluster [DBG] pgmap v239: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:04:47.281 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:47.280+0000 7ff4996ff700 1 -- 172.21.15.86:0/2768334587 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff4940f62a0 msgr2=0x7ff4940f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:04:47.281 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:47.280+0000 7ff4996ff700 1 --2- 172.21.15.86:0/2768334587 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff4940f62a0 0x7ff4940f66c0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 rx=0x7ff484009fb0 tx=0x7ff484009b30).stop 2024-02-19T22:04:47.281 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:47.280+0000 7ff4996ff700 1 -- 172.21.15.86:0/2768334587 shutdown_connections 2024-02-19T22:04:47.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:47.280+0000 7ff4996ff700 1 --2- 172.21.15.86:0/2768334587 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff4940f62a0 0x7ff4940f66c0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:04:47.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:47.280+0000 7ff4996ff700 1 -- 172.21.15.86:0/2768334587 >> 172.21.15.86:0/2768334587 conn(0x7ff4940f1aa0 msgr2=0x7ff4940f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:04:47.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:47.280+0000 7ff4996ff700 1 -- 172.21.15.86:0/2768334587 shutdown_connections 2024-02-19T22:04:47.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:47.280+0000 7ff4996ff700 1 -- 172.21.15.86:0/2768334587 wait complete. 2024-02-19T22:04:47.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:47.280+0000 7ff4996ff700 1 Processor -- start 2024-02-19T22:04:47.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:47.280+0000 7ff4996ff700 1 -- start start 2024-02-19T22:04:47.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:47.281+0000 7ff4996ff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff4940f62a0 0x7ff49406b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:04:47.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:47.281+0000 7ff4996ff700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff494069e20 con 0x7ff4940f62a0 2024-02-19T22:04:47.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:47.281+0000 7ff493fff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff4940f62a0 0x7ff49406b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:04:47.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:47.281+0000 7ff493fff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff4940f62a0 0x7ff49406b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:46026/0 (socket says 172.21.15.86:46026) 2024-02-19T22:04:47.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:47.281+0000 7ff493fff700 1 -- 172.21.15.86:0/2094472440 learned_addr learned my addr 172.21.15.86:0/2094472440 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T22:04:47.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:47.281+0000 7ff493fff700 1 -- 172.21.15.86:0/2094472440 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff484005040 con 0x7ff4940f62a0 2024-02-19T22:04:47.283 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:47.281+0000 7ff493fff700 1 --2- 172.21.15.86:0/2094472440 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff4940f62a0 0x7ff49406b9a0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0x7ff4840089f0 tx=0x7ff48400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:04:47.283 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:47.281+0000 7ff490ff9700 1 -- 172.21.15.86:0/2094472440 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff484009320 con 0x7ff4940f62a0 2024-02-19T22:04:47.283 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:47.282+0000 7ff4996ff700 1 -- 172.21.15.86:0/2094472440 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff494069fc0 con 0x7ff4940f62a0 2024-02-19T22:04:47.283 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:47.282+0000 7ff490ff9700 1 -- 172.21.15.86:0/2094472440 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff484013380 con 0x7ff4940f62a0 2024-02-19T22:04:47.283 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:47.282+0000 7ff4996ff700 1 -- 172.21.15.86:0/2094472440 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff49406a450 con 0x7ff4940f62a0 2024-02-19T22:04:47.283 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:47.282+0000 7ff490ff9700 1 -- 172.21.15.86:0/2094472440 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff484014e20 con 0x7ff4940f62a0 2024-02-19T22:04:47.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:47.282+0000 7ff490ff9700 1 -- 172.21.15.86:0/2094472440 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff4840263a0 con 0x7ff4940f62a0 2024-02-19T22:04:47.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:47.283+0000 7ff4996ff700 1 -- 172.21.15.86:0/2094472440 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff494059100 con 0x7ff4940f62a0 2024-02-19T22:04:47.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:47.283+0000 7ff490ff9700 1 --2- 172.21.15.86:0/2094472440 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7ff47c02ff10 0x7ff47c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:04:47.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:47.283+0000 7ff490ff9700 1 -- 172.21.15.86:0/2094472440 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff4840412a0 con 0x7ff4940f62a0 2024-02-19T22:04:47.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:47.283+0000 7ff4937fe700 1 --2- 172.21.15.86:0/2094472440 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7ff47c02ff10 0x7ff47c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:04:47.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:47.283+0000 7ff4937fe700 1 --2- 172.21.15.86:0/2094472440 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7ff47c02ff10 0x7ff47c0323d0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7ff4880096a0 tx=0x7ff488006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:04:47.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:47.288+0000 7ff490ff9700 1 -- 172.21.15.86:0/2094472440 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff484012330 con 0x7ff4940f62a0 2024-02-19T22:04:47.493 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:47.494+0000 7ff4996ff700 1 -- 172.21.15.86:0/2094472440 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff4940fbe30 con 0x7ff4940f62a0 2024-02-19T22:04:47.494 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:47.495+0000 7ff490ff9700 1 -- 172.21.15.86:0/2094472440 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff484012330 con 0x7ff4940f62a0 2024-02-19T22:04:47.495 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T22:04:47.495 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:04:47.496 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:47.496+0000 7ff4996ff700 1 -- 172.21.15.86:0/2094472440 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7ff47c02ff10 msgr2=0x7ff47c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:04:47.496 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:47.496+0000 7ff4996ff700 1 --2- 172.21.15.86:0/2094472440 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7ff47c02ff10 0x7ff47c0323d0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7ff4880096a0 tx=0x7ff488006b40).stop 2024-02-19T22:04:47.496 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:47.496+0000 7ff4996ff700 1 -- 172.21.15.86:0/2094472440 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff4940f62a0 msgr2=0x7ff49406b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:04:47.496 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:47.496+0000 7ff4996ff700 1 --2- 172.21.15.86:0/2094472440 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff4940f62a0 0x7ff49406b9a0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0x7ff4840089f0 tx=0x7ff48400b040).stop 2024-02-19T22:04:47.496 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:47.496+0000 7ff4996ff700 1 -- 172.21.15.86:0/2094472440 shutdown_connections 2024-02-19T22:04:47.496 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:47.496+0000 7ff4996ff700 1 --2- 172.21.15.86:0/2094472440 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7ff47c02ff10 0x7ff47c0323d0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:04:47.497 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:47.496+0000 7ff4996ff700 1 --2- 172.21.15.86:0/2094472440 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff4940f62a0 0x7ff49406b9a0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:04:47.497 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:47.496+0000 7ff4996ff700 1 -- 172.21.15.86:0/2094472440 >> 172.21.15.86:0/2094472440 conn(0x7ff4940f1aa0 msgr2=0x7ff4940f2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:04:47.497 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:47.496+0000 7ff4996ff700 1 -- 172.21.15.86:0/2094472440 shutdown_connections 2024-02-19T22:04:47.497 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:47.496+0000 7ff4996ff700 1 -- 172.21.15.86:0/2094472440 wait complete. 2024-02-19T22:04:47.497 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T22:04:48.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:04:47 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:04:47.495911+0000 mon.smithi053 (mon.0) 262 : audit [DBG] from='client.? 172.21.15.86:0/2094472440' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:04:49.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:04:48 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:04:47.512318+0000 mgr.smithi053.fyhcdi (mgr.14162) 254 : cluster [DBG] pgmap v240: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:04:49.240 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:04:49.241 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:04:50.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:04:50 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:04:49.512524+0000 mgr.smithi053.fyhcdi (mgr.14162) 255 : cluster [DBG] pgmap v241: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:04:51.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:51.068+0000 7f2011d9b700 1 -- 172.21.15.86:0/4223200024 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2004013990 con 0x7f20140f6510 2024-02-19T22:04:51.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:51.069+0000 7f20191f9700 1 -- 172.21.15.86:0/4223200024 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f20140f6510 msgr2=0x7f20140f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:04:51.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:51.069+0000 7f20191f9700 1 --2- 172.21.15.86:0/4223200024 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f20140f6510 0x7f20140f6930 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 rx=0x7f2004009fb0 tx=0x7f2004009b30).stop 2024-02-19T22:04:51.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:51.069+0000 7f20191f9700 1 -- 172.21.15.86:0/4223200024 shutdown_connections 2024-02-19T22:04:51.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:51.069+0000 7f20191f9700 1 --2- 172.21.15.86:0/4223200024 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f20140f6510 0x7f20140f6930 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:04:51.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:51.069+0000 7f20191f9700 1 -- 172.21.15.86:0/4223200024 >> 172.21.15.86:0/4223200024 conn(0x7f20140f1c70 msgr2=0x7f20140f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:04:51.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:51.069+0000 7f20191f9700 1 -- 172.21.15.86:0/4223200024 shutdown_connections 2024-02-19T22:04:51.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:51.069+0000 7f20191f9700 1 -- 172.21.15.86:0/4223200024 wait complete. 2024-02-19T22:04:51.112 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:51.070+0000 7f20191f9700 1 Processor -- start 2024-02-19T22:04:51.112 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:51.070+0000 7f20191f9700 1 -- start start 2024-02-19T22:04:51.112 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:51.070+0000 7f20191f9700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f20140f6510 0x7f20141067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:04:51.112 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:51.070+0000 7f20191f9700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2014106cd0 con 0x7f20140f6510 2024-02-19T22:04:51.112 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:51.070+0000 7f2012d9d700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f20140f6510 0x7f20141067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:04:51.112 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:51.070+0000 7f2012d9d700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f20140f6510 0x7f20141067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:55512/0 (socket says 172.21.15.86:55512) 2024-02-19T22:04:51.112 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:51.070+0000 7f2012d9d700 1 -- 172.21.15.86:0/1158255627 learned_addr learned my addr 172.21.15.86:0/1158255627 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T22:04:51.112 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:51.070+0000 7f2012d9d700 1 -- 172.21.15.86:0/1158255627 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2004005040 con 0x7f20140f6510 2024-02-19T22:04:51.112 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:51.070+0000 7f2012d9d700 1 --2- 172.21.15.86:0/1158255627 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f20140f6510 0x7f20141067e0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0x7f2004005010 tx=0x7f200400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:04:51.112 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:51.071+0000 7f1ffb7fe700 1 -- 172.21.15.86:0/1158255627 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2004009320 con 0x7f20140f6510 2024-02-19T22:04:51.112 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:51.071+0000 7f20191f9700 1 -- 172.21.15.86:0/1158255627 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2014106ed0 con 0x7f20140f6510 2024-02-19T22:04:51.112 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:51.071+0000 7f1ffb7fe700 1 -- 172.21.15.86:0/1158255627 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2004013380 con 0x7f20140f6510 2024-02-19T22:04:51.112 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:51.071+0000 7f20191f9700 1 -- 172.21.15.86:0/1158255627 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2014107360 con 0x7f20140f6510 2024-02-19T22:04:51.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:51.071+0000 7f1ffb7fe700 1 -- 172.21.15.86:0/1158255627 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2004014e50 con 0x7f20140f6510 2024-02-19T22:04:51.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:51.071+0000 7f20191f9700 1 -- 172.21.15.86:0/1158255627 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ff4004fa0 con 0x7f20140f6510 2024-02-19T22:04:51.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:51.071+0000 7f1ffb7fe700 1 -- 172.21.15.86:0/1158255627 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f20040095f0 con 0x7f20140f6510 2024-02-19T22:04:51.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:51.072+0000 7f1ffb7fe700 1 --2- 172.21.15.86:0/1158255627 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f1ffc02ff10 0x7f1ffc0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:04:51.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:51.072+0000 7f1ffb7fe700 1 -- 172.21.15.86:0/1158255627 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2004040de0 con 0x7f20140f6510 2024-02-19T22:04:51.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:51.072+0000 7f201259c700 1 --2- 172.21.15.86:0/1158255627 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f1ffc02ff10 0x7f1ffc0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:04:51.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:51.072+0000 7f201259c700 1 --2- 172.21.15.86:0/1158255627 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f1ffc02ff10 0x7f1ffc0323d0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f20080096a0 tx=0x7f2008006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:04:51.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:51.076+0000 7f1ffb7fe700 1 -- 172.21.15.86:0/1158255627 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2004014420 con 0x7f20140f6510 2024-02-19T22:04:51.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:51.282+0000 7f20191f9700 1 -- 172.21.15.86:0/1158255627 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1ff4005670 con 0x7f20140f6510 2024-02-19T22:04:51.283 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:51.283+0000 7f1ffb7fe700 1 -- 172.21.15.86:0/1158255627 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2004010230 con 0x7f20140f6510 2024-02-19T22:04:51.283 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T22:04:51.283 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:04:51.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:51.285+0000 7f20191f9700 1 -- 172.21.15.86:0/1158255627 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f1ffc02ff10 msgr2=0x7f1ffc0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:04:51.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:51.285+0000 7f20191f9700 1 --2- 172.21.15.86:0/1158255627 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f1ffc02ff10 0x7f1ffc0323d0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f20080096a0 tx=0x7f2008006b40).stop 2024-02-19T22:04:51.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:51.285+0000 7f20191f9700 1 -- 172.21.15.86:0/1158255627 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f20140f6510 msgr2=0x7f20141067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:04:51.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:51.285+0000 7f20191f9700 1 --2- 172.21.15.86:0/1158255627 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f20140f6510 0x7f20141067e0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0x7f2004005010 tx=0x7f200400b040).stop 2024-02-19T22:04:51.285 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:51.285+0000 7f20191f9700 1 -- 172.21.15.86:0/1158255627 shutdown_connections 2024-02-19T22:04:51.285 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:51.285+0000 7f20191f9700 1 --2- 172.21.15.86:0/1158255627 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f1ffc02ff10 0x7f1ffc0323d0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:04:51.285 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:51.285+0000 7f20191f9700 1 --2- 172.21.15.86:0/1158255627 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f20140f6510 0x7f20141067e0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:04:51.285 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:51.285+0000 7f20191f9700 1 -- 172.21.15.86:0/1158255627 >> 172.21.15.86:0/1158255627 conn(0x7f20140f1c70 msgr2=0x7f20140f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:04:51.285 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:51.285+0000 7f20191f9700 1 -- 172.21.15.86:0/1158255627 shutdown_connections 2024-02-19T22:04:51.285 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:51.285+0000 7f20191f9700 1 -- 172.21.15.86:0/1158255627 wait complete. 2024-02-19T22:04:51.285 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T22:04:51.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:04:51 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:04:51.284351+0000 mon.smithi053 (mon.0) 263 : audit [DBG] from='client.? 172.21.15.86:0/1158255627' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:04:52.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:04:52 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:04:51.512750+0000 mgr.smithi053.fyhcdi (mgr.14162) 256 : cluster [DBG] pgmap v242: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:04:52.885 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:04:52.885 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:04:54.746 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:54.742+0000 7f9e390b5700 1 -- 172.21.15.86:0/3765850310 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9e340f51a0 msgr2=0x7f9e340f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:04:54.746 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:54.742+0000 7f9e390b5700 1 --2- 172.21.15.86:0/3765850310 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9e340f51a0 0x7f9e340f55c0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 rx=0x7f9e24009fb0 tx=0x7f9e24009b30).stop 2024-02-19T22:04:54.746 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:54.742+0000 7f9e390b5700 1 -- 172.21.15.86:0/3765850310 shutdown_connections 2024-02-19T22:04:54.746 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:54.742+0000 7f9e390b5700 1 --2- 172.21.15.86:0/3765850310 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9e340f51a0 0x7f9e340f55c0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:04:54.746 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:54.742+0000 7f9e390b5700 1 -- 172.21.15.86:0/3765850310 >> 172.21.15.86:0/3765850310 conn(0x7f9e340f09a0 msgr2=0x7f9e340f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:04:54.746 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:54.742+0000 7f9e390b5700 1 -- 172.21.15.86:0/3765850310 shutdown_connections 2024-02-19T22:04:54.746 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:54.742+0000 7f9e390b5700 1 -- 172.21.15.86:0/3765850310 wait complete. 2024-02-19T22:04:54.746 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:54.742+0000 7f9e390b5700 1 Processor -- start 2024-02-19T22:04:54.746 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:54.743+0000 7f9e390b5700 1 -- start start 2024-02-19T22:04:54.746 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:54.743+0000 7f9e390b5700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9e340f51a0 0x7f9e341054c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:04:54.746 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:54.743+0000 7f9e390b5700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e341059b0 con 0x7f9e340f51a0 2024-02-19T22:04:54.747 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:54.743+0000 7f9e33fff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9e340f51a0 0x7f9e341054c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:04:54.747 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:54.743+0000 7f9e33fff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9e340f51a0 0x7f9e341054c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:55526/0 (socket says 172.21.15.86:55526) 2024-02-19T22:04:54.747 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:54.743+0000 7f9e33fff700 1 -- 172.21.15.86:0/1220112904 learned_addr learned my addr 172.21.15.86:0/1220112904 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T22:04:54.747 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:54.743+0000 7f9e33fff700 1 -- 172.21.15.86:0/1220112904 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e24005040 con 0x7f9e340f51a0 2024-02-19T22:04:54.747 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:54.744+0000 7f9e33fff700 1 --2- 172.21.15.86:0/1220112904 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9e340f51a0 0x7f9e341054c0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0x7f9e240089f0 tx=0x7f9e2400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:04:54.747 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:54.744+0000 7f9e30ff9700 1 -- 172.21.15.86:0/1220112904 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9e24009320 con 0x7f9e340f51a0 2024-02-19T22:04:54.747 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:54.744+0000 7f9e390b5700 1 -- 172.21.15.86:0/1220112904 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e34105bb0 con 0x7f9e340f51a0 2024-02-19T22:04:54.747 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:54.744+0000 7f9e30ff9700 1 -- 172.21.15.86:0/1220112904 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9e24013380 con 0x7f9e340f51a0 2024-02-19T22:04:54.747 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:54.744+0000 7f9e30ff9700 1 -- 172.21.15.86:0/1220112904 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9e24014e20 con 0x7f9e340f51a0 2024-02-19T22:04:54.747 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:54.744+0000 7f9e390b5700 1 -- 172.21.15.86:0/1220112904 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e34106040 con 0x7f9e340f51a0 2024-02-19T22:04:54.747 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:54.745+0000 7f9e30ff9700 1 -- 172.21.15.86:0/1220112904 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9e24014420 con 0x7f9e340f51a0 2024-02-19T22:04:54.747 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:54.745+0000 7f9e390b5700 1 -- 172.21.15.86:0/1220112904 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e34045ad0 con 0x7f9e340f51a0 2024-02-19T22:04:54.747 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:54.745+0000 7f9e30ff9700 1 --2- 172.21.15.86:0/1220112904 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f9e1c02ff10 0x7f9e1c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:04:54.747 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:54.745+0000 7f9e30ff9700 1 -- 172.21.15.86:0/1220112904 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9e24040fd0 con 0x7f9e340f51a0 2024-02-19T22:04:54.748 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:54.745+0000 7f9e337fe700 1 --2- 172.21.15.86:0/1220112904 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f9e1c02ff10 0x7f9e1c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:04:54.748 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:54.745+0000 7f9e337fe700 1 --2- 172.21.15.86:0/1220112904 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f9e1c02ff10 0x7f9e1c0323d0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7f9e280096a0 tx=0x7f9e28006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:04:54.749 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:54.750+0000 7f9e30ff9700 1 -- 172.21.15.86:0/1220112904 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9e240095f0 con 0x7f9e340f51a0 2024-02-19T22:04:54.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:54.954+0000 7f9e390b5700 1 -- 172.21.15.86:0/1220112904 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9e34106ef0 con 0x7f9e340f51a0 2024-02-19T22:04:54.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:54.955+0000 7f9e30ff9700 1 -- 172.21.15.86:0/1220112904 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9e2401db70 con 0x7f9e340f51a0 2024-02-19T22:04:54.955 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T22:04:54.955 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:04:54.956 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:54.956+0000 7f9e390b5700 1 -- 172.21.15.86:0/1220112904 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f9e1c02ff10 msgr2=0x7f9e1c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:04:54.956 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:54.956+0000 7f9e390b5700 1 --2- 172.21.15.86:0/1220112904 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f9e1c02ff10 0x7f9e1c0323d0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7f9e280096a0 tx=0x7f9e28006b40).stop 2024-02-19T22:04:54.956 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:54.956+0000 7f9e390b5700 1 -- 172.21.15.86:0/1220112904 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9e340f51a0 msgr2=0x7f9e341054c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:04:54.956 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:54.956+0000 7f9e390b5700 1 --2- 172.21.15.86:0/1220112904 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9e340f51a0 0x7f9e341054c0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0x7f9e240089f0 tx=0x7f9e2400b040).stop 2024-02-19T22:04:54.956 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:54.956+0000 7f9e390b5700 1 -- 172.21.15.86:0/1220112904 shutdown_connections 2024-02-19T22:04:54.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:54.956+0000 7f9e390b5700 1 --2- 172.21.15.86:0/1220112904 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f9e1c02ff10 0x7f9e1c0323d0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:04:54.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:54.956+0000 7f9e390b5700 1 --2- 172.21.15.86:0/1220112904 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9e340f51a0 0x7f9e341054c0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:04:54.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:54.956+0000 7f9e390b5700 1 -- 172.21.15.86:0/1220112904 >> 172.21.15.86:0/1220112904 conn(0x7f9e340f09a0 msgr2=0x7f9e340f13e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:04:54.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:54.956+0000 7f9e390b5700 1 -- 172.21.15.86:0/1220112904 shutdown_connections 2024-02-19T22:04:54.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:54.956+0000 7f9e390b5700 1 -- 172.21.15.86:0/1220112904 wait complete. 2024-02-19T22:04:54.957 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T22:04:55.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:04:54 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:04:53.513103+0000 mgr.smithi053.fyhcdi (mgr.14162) 257 : cluster [DBG] pgmap v243: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:04:56.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:04:55 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:04:54.956217+0000 mon.smithi053 (mon.0) 264 : audit [DBG] from='client.? 172.21.15.86:0/1220112904' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:04:56.709 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:04:56.709 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:04:57.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:04:56 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:04:55.513458+0000 mgr.smithi053.fyhcdi (mgr.14162) 258 : cluster [DBG] pgmap v244: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:04:58.777 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:58.696+0000 7f1a68320700 1 -- 172.21.15.86:0/574434946 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1a600f5410 msgr2=0x7f1a600f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:04:58.777 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:58.696+0000 7f1a68320700 1 --2- 172.21.15.86:0/574434946 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1a600f5410 0x7f1a600f5830 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 rx=0x7f1a58009960 tx=0x7f1a58004d10).stop 2024-02-19T22:04:58.778 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:58.696+0000 7f1a68320700 1 -- 172.21.15.86:0/574434946 shutdown_connections 2024-02-19T22:04:58.778 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:58.696+0000 7f1a68320700 1 --2- 172.21.15.86:0/574434946 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1a600f5410 0x7f1a600f5830 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:04:58.778 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:58.696+0000 7f1a68320700 1 -- 172.21.15.86:0/574434946 >> 172.21.15.86:0/574434946 conn(0x7f1a600f0b70 msgr2=0x7f1a600f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:04:58.778 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:58.696+0000 7f1a68320700 1 -- 172.21.15.86:0/574434946 shutdown_connections 2024-02-19T22:04:58.778 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:58.696+0000 7f1a68320700 1 -- 172.21.15.86:0/574434946 wait complete. 2024-02-19T22:04:58.778 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:58.696+0000 7f1a68320700 1 Processor -- start 2024-02-19T22:04:58.778 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:58.696+0000 7f1a68320700 1 -- start start 2024-02-19T22:04:58.778 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:58.696+0000 7f1a68320700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1a600f5410 0x7f1a6010b940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:04:58.778 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:58.696+0000 7f1a68320700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a60107330 con 0x7f1a600f5410 2024-02-19T22:04:58.778 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:58.697+0000 7f1a660bc700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1a600f5410 0x7f1a6010b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:04:58.778 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:58.697+0000 7f1a660bc700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1a600f5410 0x7f1a6010b940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:55544/0 (socket says 172.21.15.86:55544) 2024-02-19T22:04:58.778 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:58.697+0000 7f1a660bc700 1 -- 172.21.15.86:0/4072409164 learned_addr learned my addr 172.21.15.86:0/4072409164 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T22:04:58.778 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:58.697+0000 7f1a660bc700 1 -- 172.21.15.86:0/4072409164 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a58009640 con 0x7f1a600f5410 2024-02-19T22:04:58.778 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:58.697+0000 7f1a660bc700 1 --2- 172.21.15.86:0/4072409164 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1a600f5410 0x7f1a6010b940 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7f1a58004b70 tx=0x7f1a58004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:04:58.779 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:58.697+0000 7f1a5effd700 1 -- 172.21.15.86:0/4072409164 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1a58018070 con 0x7f1a600f5410 2024-02-19T22:04:58.779 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:58.697+0000 7f1a68320700 1 -- 172.21.15.86:0/4072409164 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a60060220 con 0x7f1a600f5410 2024-02-19T22:04:58.779 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:58.697+0000 7f1a5effd700 1 -- 172.21.15.86:0/4072409164 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1a5800b970 con 0x7f1a600f5410 2024-02-19T22:04:58.779 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:58.697+0000 7f1a5effd700 1 -- 172.21.15.86:0/4072409164 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1a5801c560 con 0x7f1a600f5410 2024-02-19T22:04:58.779 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:58.697+0000 7f1a68320700 1 -- 172.21.15.86:0/4072409164 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a6010be30 con 0x7f1a600f5410 2024-02-19T22:04:58.779 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:58.698+0000 7f1a5effd700 1 -- 172.21.15.86:0/4072409164 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1a5801c6c0 con 0x7f1a600f5410 2024-02-19T22:04:58.779 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:58.698+0000 7f1a5effd700 1 --2- 172.21.15.86:0/4072409164 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f1a4802ff60 0x7f1a48032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:04:58.779 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:58.698+0000 7f1a5effd700 1 -- 172.21.15.86:0/4072409164 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1a58041270 con 0x7f1a600f5410 2024-02-19T22:04:58.779 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:58.698+0000 7f1a68320700 1 -- 172.21.15.86:0/4072409164 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a60045ad0 con 0x7f1a600f5410 2024-02-19T22:04:58.779 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:58.698+0000 7f1a658bb700 1 --2- 172.21.15.86:0/4072409164 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f1a4802ff60 0x7f1a48032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:04:58.779 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:58.699+0000 7f1a658bb700 1 --2- 172.21.15.86:0/4072409164 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f1a4802ff60 0x7f1a48032420 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7f1a540096a0 tx=0x7f1a54006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:04:58.779 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:58.703+0000 7f1a5effd700 1 -- 172.21.15.86:0/4072409164 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1a580127f0 con 0x7f1a600f5410 2024-02-19T22:04:58.908 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:58.908+0000 7f1a68320700 1 -- 172.21.15.86:0/4072409164 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1a6010c570 con 0x7f1a600f5410 2024-02-19T22:04:58.909 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:58.909+0000 7f1a5effd700 1 -- 172.21.15.86:0/4072409164 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1a58019230 con 0x7f1a600f5410 2024-02-19T22:04:58.909 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T22:04:58.909 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:04:58.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:58.910+0000 7f1a68320700 1 -- 172.21.15.86:0/4072409164 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f1a4802ff60 msgr2=0x7f1a48032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:04:58.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:58.911+0000 7f1a68320700 1 --2- 172.21.15.86:0/4072409164 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f1a4802ff60 0x7f1a48032420 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7f1a540096a0 tx=0x7f1a54006b40).stop 2024-02-19T22:04:58.911 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:58.911+0000 7f1a68320700 1 -- 172.21.15.86:0/4072409164 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1a600f5410 msgr2=0x7f1a6010b940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:04:58.911 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:58.911+0000 7f1a68320700 1 --2- 172.21.15.86:0/4072409164 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1a600f5410 0x7f1a6010b940 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7f1a58004b70 tx=0x7f1a58004d10).stop 2024-02-19T22:04:58.911 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:58.911+0000 7f1a68320700 1 -- 172.21.15.86:0/4072409164 shutdown_connections 2024-02-19T22:04:58.911 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:58.911+0000 7f1a68320700 1 --2- 172.21.15.86:0/4072409164 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f1a4802ff60 0x7f1a48032420 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:04:58.911 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:58.911+0000 7f1a68320700 1 --2- 172.21.15.86:0/4072409164 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1a600f5410 0x7f1a6010b940 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:04:58.911 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:58.911+0000 7f1a68320700 1 -- 172.21.15.86:0/4072409164 >> 172.21.15.86:0/4072409164 conn(0x7f1a600f0b70 msgr2=0x7f1a600f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:04:58.911 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:58.911+0000 7f1a68320700 1 -- 172.21.15.86:0/4072409164 shutdown_connections 2024-02-19T22:04:58.911 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:04:58.911+0000 7f1a68320700 1 -- 172.21.15.86:0/4072409164 wait complete. 2024-02-19T22:04:58.911 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T22:04:59.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:04:58 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:04:57.513850+0000 mgr.smithi053.fyhcdi (mgr.14162) 259 : cluster [DBG] pgmap v245: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:05:00.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:04:59 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:04:58.910388+0000 mon.smithi053 (mon.0) 265 : audit [DBG] from='client.? 172.21.15.86:0/4072409164' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:05:00.712 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:05:00.713 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:05:01.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:05:00 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:04:59.514231+0000 mgr.smithi053.fyhcdi (mgr.14162) 260 : cluster [DBG] pgmap v246: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:05:02.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:02.707+0000 7ff74caed700 1 -- 172.21.15.86:0/3392447223 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff7480f5410 msgr2=0x7ff7480f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:05:02.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:02.707+0000 7ff74caed700 1 --2- 172.21.15.86:0/3392447223 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff7480f5410 0x7ff7480f5830 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 rx=0x7ff73c009fb0 tx=0x7ff73c009b30).stop 2024-02-19T22:05:02.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:02.708+0000 7ff74caed700 1 -- 172.21.15.86:0/3392447223 shutdown_connections 2024-02-19T22:05:02.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:02.708+0000 7ff74caed700 1 --2- 172.21.15.86:0/3392447223 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff7480f5410 0x7ff7480f5830 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:05:02.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:02.708+0000 7ff74caed700 1 -- 172.21.15.86:0/3392447223 >> 172.21.15.86:0/3392447223 conn(0x7ff7480f0b70 msgr2=0x7ff7480f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:05:02.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:02.708+0000 7ff74caed700 1 -- 172.21.15.86:0/3392447223 shutdown_connections 2024-02-19T22:05:02.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:02.708+0000 7ff74caed700 1 -- 172.21.15.86:0/3392447223 wait complete. 2024-02-19T22:05:02.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:02.708+0000 7ff74caed700 1 Processor -- start 2024-02-19T22:05:02.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:02.708+0000 7ff74caed700 1 -- start start 2024-02-19T22:05:02.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:02.708+0000 7ff74caed700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff7480f5410 0x7ff74810b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:05:02.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:02.708+0000 7ff74caed700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7481072f0 con 0x7ff7480f5410 2024-02-19T22:05:02.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:02.708+0000 7ff74659c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff7480f5410 0x7ff74810b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:05:02.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:02.708+0000 7ff74659c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff7480f5410 0x7ff74810b900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:39864/0 (socket says 172.21.15.86:39864) 2024-02-19T22:05:02.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:02.708+0000 7ff74659c700 1 -- 172.21.15.86:0/2976597505 learned_addr learned my addr 172.21.15.86:0/2976597505 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T22:05:02.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:02.709+0000 7ff74659c700 1 -- 172.21.15.86:0/2976597505 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff73c005040 con 0x7ff7480f5410 2024-02-19T22:05:02.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:02.709+0000 7ff74659c700 1 --2- 172.21.15.86:0/2976597505 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff7480f5410 0x7ff74810b900 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 rx=0x7ff73c005010 tx=0x7ff73c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:05:02.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:02.709+0000 7ff736ffd700 1 -- 172.21.15.86:0/2976597505 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff73c009320 con 0x7ff7480f5410 2024-02-19T22:05:02.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:02.709+0000 7ff74caed700 1 -- 172.21.15.86:0/2976597505 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff748060220 con 0x7ff7480f5410 2024-02-19T22:05:02.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:02.709+0000 7ff736ffd700 1 -- 172.21.15.86:0/2976597505 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff73c013380 con 0x7ff7480f5410 2024-02-19T22:05:02.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:02.709+0000 7ff736ffd700 1 -- 172.21.15.86:0/2976597505 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff73c014e50 con 0x7ff7480f5410 2024-02-19T22:05:02.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:02.709+0000 7ff74caed700 1 -- 172.21.15.86:0/2976597505 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff74810bdf0 con 0x7ff7480f5410 2024-02-19T22:05:02.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:02.710+0000 7ff736ffd700 1 -- 172.21.15.86:0/2976597505 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff73c0095f0 con 0x7ff7480f5410 2024-02-19T22:05:02.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:02.710+0000 7ff74caed700 1 -- 172.21.15.86:0/2976597505 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff748045ad0 con 0x7ff7480f5410 2024-02-19T22:05:02.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:02.710+0000 7ff736ffd700 1 --2- 172.21.15.86:0/2976597505 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7ff73002ff10 0x7ff7300323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:05:02.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:02.710+0000 7ff736ffd700 1 -- 172.21.15.86:0/2976597505 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff73c040e90 con 0x7ff7480f5410 2024-02-19T22:05:02.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:02.710+0000 7ff745d9b700 1 --2- 172.21.15.86:0/2976597505 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7ff73002ff10 0x7ff7300323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:05:02.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:02.710+0000 7ff745d9b700 1 --2- 172.21.15.86:0/2976597505 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7ff73002ff10 0x7ff7300323d0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7ff7380096a0 tx=0x7ff738006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:05:02.714 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:02.715+0000 7ff736ffd700 1 -- 172.21.15.86:0/2976597505 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff73c014420 con 0x7ff7480f5410 2024-02-19T22:05:02.919 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:02.919+0000 7ff74caed700 1 -- 172.21.15.86:0/2976597505 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff74810c5b0 con 0x7ff7480f5410 2024-02-19T22:05:02.919 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:02.919+0000 7ff736ffd700 1 -- 172.21.15.86:0/2976597505 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff73c010230 con 0x7ff7480f5410 2024-02-19T22:05:02.919 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T22:05:02.919 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:05:02.921 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:02.921+0000 7ff74caed700 1 -- 172.21.15.86:0/2976597505 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7ff73002ff10 msgr2=0x7ff7300323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:05:02.921 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:02.921+0000 7ff74caed700 1 --2- 172.21.15.86:0/2976597505 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7ff73002ff10 0x7ff7300323d0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7ff7380096a0 tx=0x7ff738006b40).stop 2024-02-19T22:05:02.921 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:02.921+0000 7ff74caed700 1 -- 172.21.15.86:0/2976597505 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff7480f5410 msgr2=0x7ff74810b900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:05:02.921 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:02.921+0000 7ff74caed700 1 --2- 172.21.15.86:0/2976597505 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff7480f5410 0x7ff74810b900 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 rx=0x7ff73c005010 tx=0x7ff73c00b040).stop 2024-02-19T22:05:02.921 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:02.921+0000 7ff74caed700 1 -- 172.21.15.86:0/2976597505 shutdown_connections 2024-02-19T22:05:02.921 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:02.921+0000 7ff74caed700 1 --2- 172.21.15.86:0/2976597505 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7ff73002ff10 0x7ff7300323d0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:05:02.921 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:02.921+0000 7ff74caed700 1 --2- 172.21.15.86:0/2976597505 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff7480f5410 0x7ff74810b900 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:05:02.921 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:02.921+0000 7ff74caed700 1 -- 172.21.15.86:0/2976597505 >> 172.21.15.86:0/2976597505 conn(0x7ff7480f0b70 msgr2=0x7ff7480f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:05:02.921 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:02.921+0000 7ff74caed700 1 -- 172.21.15.86:0/2976597505 shutdown_connections 2024-02-19T22:05:02.922 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:02.921+0000 7ff74caed700 1 -- 172.21.15.86:0/2976597505 wait complete. 2024-02-19T22:05:02.922 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T22:05:03.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:05:02 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:05:01.514549+0000 mgr.smithi053.fyhcdi (mgr.14162) 261 : cluster [DBG] pgmap v247: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:05:04.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:05:03 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:05:02.920641+0000 mon.smithi053 (mon.0) 266 : audit [DBG] from='client.? 172.21.15.86:0/2976597505' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:05:04.575 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:05:04.576 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:05:05.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:05:04 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:05:03.514862+0000 mgr.smithi053.fyhcdi (mgr.14162) 262 : cluster [DBG] pgmap v248: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:05:06.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:06.485+0000 7f314b41c700 1 -- 172.21.15.86:0/2023281622 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f31440f5410 msgr2=0x7f31440f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:05:06.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:06.485+0000 7f314b41c700 1 --2- 172.21.15.86:0/2023281622 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f31440f5410 0x7f31440f5830 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 rx=0x7f3134009fb0 tx=0x7f3134009b30).stop 2024-02-19T22:05:06.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:06.485+0000 7f314b41c700 1 -- 172.21.15.86:0/2023281622 shutdown_connections 2024-02-19T22:05:06.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:06.485+0000 7f314b41c700 1 --2- 172.21.15.86:0/2023281622 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f31440f5410 0x7f31440f5830 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:05:06.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:06.485+0000 7f314b41c700 1 -- 172.21.15.86:0/2023281622 >> 172.21.15.86:0/2023281622 conn(0x7f31440f0b70 msgr2=0x7f31440f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:05:06.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:06.485+0000 7f314b41c700 1 -- 172.21.15.86:0/2023281622 shutdown_connections 2024-02-19T22:05:06.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:06.485+0000 7f314b41c700 1 -- 172.21.15.86:0/2023281622 wait complete. 2024-02-19T22:05:06.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:06.486+0000 7f314b41c700 1 Processor -- start 2024-02-19T22:05:06.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:06.486+0000 7f314b41c700 1 -- start start 2024-02-19T22:05:06.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:06.486+0000 7f314b41c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f31440f5410 0x7f31441056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:05:06.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:06.486+0000 7f314b41c700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3144105bd0 con 0x7f31440f5410 2024-02-19T22:05:06.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:06.487+0000 7f31491b8700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f31440f5410 0x7f31441056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:05:06.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:06.487+0000 7f31491b8700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f31440f5410 0x7f31441056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:39870/0 (socket says 172.21.15.86:39870) 2024-02-19T22:05:06.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:06.487+0000 7f31491b8700 1 -- 172.21.15.86:0/3469868576 learned_addr learned my addr 172.21.15.86:0/3469868576 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T22:05:06.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:06.487+0000 7f31491b8700 1 -- 172.21.15.86:0/3469868576 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3134005040 con 0x7f31440f5410 2024-02-19T22:05:06.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:06.487+0000 7f31491b8700 1 --2- 172.21.15.86:0/3469868576 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f31440f5410 0x7f31441056e0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 rx=0x7f31340089f0 tx=0x7f313400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:05:06.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:06.487+0000 7f3139ffb700 1 -- 172.21.15.86:0/3469868576 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3134009320 con 0x7f31440f5410 2024-02-19T22:05:06.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:06.487+0000 7f314b41c700 1 -- 172.21.15.86:0/3469868576 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3144105dd0 con 0x7f31440f5410 2024-02-19T22:05:06.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:06.487+0000 7f3139ffb700 1 -- 172.21.15.86:0/3469868576 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3134013380 con 0x7f31440f5410 2024-02-19T22:05:06.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:06.487+0000 7f3139ffb700 1 -- 172.21.15.86:0/3469868576 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f313401ed80 con 0x7f31440f5410 2024-02-19T22:05:06.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:06.487+0000 7f314b41c700 1 -- 172.21.15.86:0/3469868576 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3144106260 con 0x7f31440f5410 2024-02-19T22:05:06.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:06.488+0000 7f3139ffb700 1 -- 172.21.15.86:0/3469868576 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f313401e420 con 0x7f31440f5410 2024-02-19T22:05:06.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:06.488+0000 7f314b41c700 1 -- 172.21.15.86:0/3469868576 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3144045ad0 con 0x7f31440f5410 2024-02-19T22:05:06.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:06.488+0000 7f3139ffb700 1 --2- 172.21.15.86:0/3469868576 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f313002ff60 0x7f3130032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:05:06.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:06.488+0000 7f3139ffb700 1 -- 172.21.15.86:0/3469868576 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3134041080 con 0x7f31440f5410 2024-02-19T22:05:06.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:06.489+0000 7f31489b7700 1 --2- 172.21.15.86:0/3469868576 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f313002ff60 0x7f3130032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:05:06.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:06.489+0000 7f31489b7700 1 --2- 172.21.15.86:0/3469868576 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f313002ff60 0x7f3130032420 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7f31400096a0 tx=0x7f3140006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:05:06.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:06.493+0000 7f3139ffb700 1 -- 172.21.15.86:0/3469868576 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f31340095f0 con 0x7f31440f5410 2024-02-19T22:05:06.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:06.697+0000 7f314b41c700 1 -- 172.21.15.86:0/3469868576 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3144107110 con 0x7f31440f5410 2024-02-19T22:05:06.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:06.698+0000 7f3139ffb700 1 -- 172.21.15.86:0/3469868576 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f313401cb10 con 0x7f31440f5410 2024-02-19T22:05:06.698 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T22:05:06.698 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:05:06.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:06.699+0000 7f314b41c700 1 -- 172.21.15.86:0/3469868576 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f313002ff60 msgr2=0x7f3130032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:05:06.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:06.699+0000 7f314b41c700 1 --2- 172.21.15.86:0/3469868576 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f313002ff60 0x7f3130032420 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7f31400096a0 tx=0x7f3140006b40).stop 2024-02-19T22:05:06.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:06.700+0000 7f314b41c700 1 -- 172.21.15.86:0/3469868576 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f31440f5410 msgr2=0x7f31441056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:05:06.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:06.700+0000 7f314b41c700 1 --2- 172.21.15.86:0/3469868576 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f31440f5410 0x7f31441056e0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 rx=0x7f31340089f0 tx=0x7f313400b040).stop 2024-02-19T22:05:06.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:06.700+0000 7f314b41c700 1 -- 172.21.15.86:0/3469868576 shutdown_connections 2024-02-19T22:05:06.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:06.700+0000 7f314b41c700 1 --2- 172.21.15.86:0/3469868576 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f313002ff60 0x7f3130032420 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:05:06.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:06.700+0000 7f314b41c700 1 --2- 172.21.15.86:0/3469868576 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f31440f5410 0x7f31441056e0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:05:06.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:06.700+0000 7f314b41c700 1 -- 172.21.15.86:0/3469868576 >> 172.21.15.86:0/3469868576 conn(0x7f31440f0b70 msgr2=0x7f31440f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:05:06.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:06.700+0000 7f314b41c700 1 -- 172.21.15.86:0/3469868576 shutdown_connections 2024-02-19T22:05:06.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:06.700+0000 7f314b41c700 1 -- 172.21.15.86:0/3469868576 wait complete. 2024-02-19T22:05:06.700 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T22:05:07.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:05:06 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:05:05.515288+0000 mgr.smithi053.fyhcdi (mgr.14162) 263 : cluster [DBG] pgmap v249: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:05:08.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:05:07 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:05:06.699297+0000 mon.smithi053 (mon.0) 267 : audit [DBG] from='client.? 172.21.15.86:0/3469868576' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:05:08.400 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:05:08.400 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:05:09.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:05:08 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:05:07.515657+0000 mgr.smithi053.fyhcdi (mgr.14162) 264 : cluster [DBG] pgmap v250: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:05:10.341 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:10.341+0000 7fca70ac2700 1 -- 172.21.15.86:0/2032666866 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fca6c0f5410 msgr2=0x7fca6c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:05:10.341 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:10.341+0000 7fca70ac2700 1 --2- 172.21.15.86:0/2032666866 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fca6c0f5410 0x7fca6c0f5830 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 rx=0x7fca54009fb0 tx=0x7fca54009b30).stop 2024-02-19T22:05:10.341 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:10.341+0000 7fca70ac2700 1 -- 172.21.15.86:0/2032666866 shutdown_connections 2024-02-19T22:05:10.342 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:10.341+0000 7fca70ac2700 1 --2- 172.21.15.86:0/2032666866 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fca6c0f5410 0x7fca6c0f5830 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:05:10.342 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:10.341+0000 7fca70ac2700 1 -- 172.21.15.86:0/2032666866 >> 172.21.15.86:0/2032666866 conn(0x7fca6c0f0b70 msgr2=0x7fca6c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:05:10.342 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:10.342+0000 7fca70ac2700 1 -- 172.21.15.86:0/2032666866 shutdown_connections 2024-02-19T22:05:10.342 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:10.342+0000 7fca70ac2700 1 -- 172.21.15.86:0/2032666866 wait complete. 2024-02-19T22:05:10.344 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:10.342+0000 7fca70ac2700 1 Processor -- start 2024-02-19T22:05:10.344 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:10.342+0000 7fca70ac2700 1 -- start start 2024-02-19T22:05:10.344 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:10.342+0000 7fca70ac2700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fca6c0f5410 0x7fca6c1056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:05:10.344 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:10.342+0000 7fca70ac2700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca6c105bd0 con 0x7fca6c0f5410 2024-02-19T22:05:10.344 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:10.342+0000 7fca6a59c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fca6c0f5410 0x7fca6c1056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:05:10.344 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:10.342+0000 7fca6a59c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fca6c0f5410 0x7fca6c1056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:35130/0 (socket says 172.21.15.86:35130) 2024-02-19T22:05:10.345 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:10.342+0000 7fca6a59c700 1 -- 172.21.15.86:0/3624711429 learned_addr learned my addr 172.21.15.86:0/3624711429 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T22:05:10.345 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:10.343+0000 7fca6a59c700 1 -- 172.21.15.86:0/3624711429 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca54005040 con 0x7fca6c0f5410 2024-02-19T22:05:10.345 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:10.343+0000 7fca6a59c700 1 --2- 172.21.15.86:0/3624711429 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fca6c0f5410 0x7fca6c1056e0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 rx=0x7fca54005010 tx=0x7fca5400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:05:10.345 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:10.343+0000 7fca62ffd700 1 -- 172.21.15.86:0/3624711429 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fca54009320 con 0x7fca6c0f5410 2024-02-19T22:05:10.345 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:10.343+0000 7fca70ac2700 1 -- 172.21.15.86:0/3624711429 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca6c105dd0 con 0x7fca6c0f5410 2024-02-19T22:05:10.345 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:10.343+0000 7fca62ffd700 1 -- 172.21.15.86:0/3624711429 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fca54013380 con 0x7fca6c0f5410 2024-02-19T22:05:10.345 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:10.343+0000 7fca62ffd700 1 -- 172.21.15.86:0/3624711429 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fca54014e50 con 0x7fca6c0f5410 2024-02-19T22:05:10.345 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:10.343+0000 7fca70ac2700 1 -- 172.21.15.86:0/3624711429 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca6c106260 con 0x7fca6c0f5410 2024-02-19T22:05:10.345 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:10.344+0000 7fca62ffd700 1 -- 172.21.15.86:0/3624711429 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fca54014420 con 0x7fca6c0f5410 2024-02-19T22:05:10.345 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:10.344+0000 7fca70ac2700 1 -- 172.21.15.86:0/3624711429 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca6c045ad0 con 0x7fca6c0f5410 2024-02-19T22:05:10.347 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:10.344+0000 7fca62ffd700 1 --2- 172.21.15.86:0/3624711429 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fca5802ff10 0x7fca580323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:05:10.347 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:10.344+0000 7fca62ffd700 1 -- 172.21.15.86:0/3624711429 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fca54040f30 con 0x7fca6c0f5410 2024-02-19T22:05:10.347 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:10.344+0000 7fca69d9b700 1 --2- 172.21.15.86:0/3624711429 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fca5802ff10 0x7fca580323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:05:10.347 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:10.345+0000 7fca69d9b700 1 --2- 172.21.15.86:0/3624711429 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fca5802ff10 0x7fca580323d0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7fca5c0096a0 tx=0x7fca5c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:05:10.348 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:10.349+0000 7fca62ffd700 1 -- 172.21.15.86:0/3624711429 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fca540095f0 con 0x7fca6c0f5410 2024-02-19T22:05:10.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:10.555+0000 7fca70ac2700 1 -- 172.21.15.86:0/3624711429 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fca6c1070b0 con 0x7fca6c0f5410 2024-02-19T22:05:10.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:10.556+0000 7fca62ffd700 1 -- 172.21.15.86:0/3624711429 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fca5401db70 con 0x7fca6c0f5410 2024-02-19T22:05:10.556 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T22:05:10.556 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:05:10.557 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:10.557+0000 7fca70ac2700 1 -- 172.21.15.86:0/3624711429 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fca5802ff10 msgr2=0x7fca580323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:05:10.557 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:10.558+0000 7fca70ac2700 1 --2- 172.21.15.86:0/3624711429 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fca5802ff10 0x7fca580323d0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7fca5c0096a0 tx=0x7fca5c006b40).stop 2024-02-19T22:05:10.557 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:10.558+0000 7fca70ac2700 1 -- 172.21.15.86:0/3624711429 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fca6c0f5410 msgr2=0x7fca6c1056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:05:10.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:10.558+0000 7fca70ac2700 1 --2- 172.21.15.86:0/3624711429 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fca6c0f5410 0x7fca6c1056e0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 rx=0x7fca54005010 tx=0x7fca5400b040).stop 2024-02-19T22:05:10.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:10.558+0000 7fca70ac2700 1 -- 172.21.15.86:0/3624711429 shutdown_connections 2024-02-19T22:05:10.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:10.558+0000 7fca70ac2700 1 --2- 172.21.15.86:0/3624711429 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fca5802ff10 0x7fca580323d0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:05:10.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:10.558+0000 7fca70ac2700 1 --2- 172.21.15.86:0/3624711429 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fca6c0f5410 0x7fca6c1056e0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:05:10.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:10.558+0000 7fca70ac2700 1 -- 172.21.15.86:0/3624711429 >> 172.21.15.86:0/3624711429 conn(0x7fca6c0f0b70 msgr2=0x7fca6c0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:05:10.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:10.558+0000 7fca70ac2700 1 -- 172.21.15.86:0/3624711429 shutdown_connections 2024-02-19T22:05:10.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:10.558+0000 7fca70ac2700 1 -- 172.21.15.86:0/3624711429 wait complete. 2024-02-19T22:05:10.558 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T22:05:10.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:05:10 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:05:09.516058+0000 mgr.smithi053.fyhcdi (mgr.14162) 265 : cluster [DBG] pgmap v251: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:05:11.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:05:11 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:05:10.557406+0000 mon.smithi053 (mon.0) 268 : audit [DBG] from='client.? 172.21.15.86:0/3624711429' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:05:12.343 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:05:12.344 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:05:12.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:05:12 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:05:11.516339+0000 mgr.smithi053.fyhcdi (mgr.14162) 266 : cluster [DBG] pgmap v252: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:05:14.234 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:14.233+0000 7fd79b138700 1 -- 172.21.15.86:0/2868039691 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd7940f6510 msgr2=0x7fd7940f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:05:14.234 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:14.233+0000 7fd79b138700 1 --2- 172.21.15.86:0/2868039691 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd7940f6510 0x7fd7940f6930 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 rx=0x7fd788009fb0 tx=0x7fd788009b30).stop 2024-02-19T22:05:14.234 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:14.233+0000 7fd79b138700 1 -- 172.21.15.86:0/2868039691 shutdown_connections 2024-02-19T22:05:14.234 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:14.233+0000 7fd79b138700 1 --2- 172.21.15.86:0/2868039691 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd7940f6510 0x7fd7940f6930 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:05:14.234 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:14.233+0000 7fd79b138700 1 -- 172.21.15.86:0/2868039691 >> 172.21.15.86:0/2868039691 conn(0x7fd7940f1c70 msgr2=0x7fd7940f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:05:14.234 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:14.233+0000 7fd79b138700 1 -- 172.21.15.86:0/2868039691 shutdown_connections 2024-02-19T22:05:14.234 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:14.233+0000 7fd79b138700 1 -- 172.21.15.86:0/2868039691 wait complete. 2024-02-19T22:05:14.234 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:14.233+0000 7fd79b138700 1 Processor -- start 2024-02-19T22:05:14.234 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:14.233+0000 7fd79b138700 1 -- start start 2024-02-19T22:05:14.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:14.233+0000 7fd79b138700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd7940f6510 0x7fd7941067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:05:14.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:14.233+0000 7fd79b138700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd794106cd0 con 0x7fd7940f6510 2024-02-19T22:05:14.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:14.234+0000 7fd798ed4700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd7940f6510 0x7fd7941067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:05:14.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:14.234+0000 7fd798ed4700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd7940f6510 0x7fd7941067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:35140/0 (socket says 172.21.15.86:35140) 2024-02-19T22:05:14.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:14.234+0000 7fd798ed4700 1 -- 172.21.15.86:0/3643234756 learned_addr learned my addr 172.21.15.86:0/3643234756 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T22:05:14.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:14.234+0000 7fd798ed4700 1 -- 172.21.15.86:0/3643234756 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd788005040 con 0x7fd7940f6510 2024-02-19T22:05:14.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:14.234+0000 7fd798ed4700 1 --2- 172.21.15.86:0/3643234756 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd7940f6510 0x7fd7941067e0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0x7fd788005010 tx=0x7fd78800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:05:14.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:14.234+0000 7fd7917fa700 1 -- 172.21.15.86:0/3643234756 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd788009320 con 0x7fd7940f6510 2024-02-19T22:05:14.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:14.234+0000 7fd7917fa700 1 -- 172.21.15.86:0/3643234756 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd788013380 con 0x7fd7940f6510 2024-02-19T22:05:14.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:14.234+0000 7fd79b138700 1 -- 172.21.15.86:0/3643234756 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd794106ed0 con 0x7fd7940f6510 2024-02-19T22:05:14.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:14.234+0000 7fd7917fa700 1 -- 172.21.15.86:0/3643234756 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd788014e50 con 0x7fd7940f6510 2024-02-19T22:05:14.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:14.234+0000 7fd79b138700 1 -- 172.21.15.86:0/3643234756 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd794107360 con 0x7fd7940f6510 2024-02-19T22:05:14.236 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:14.235+0000 7fd7917fa700 1 -- 172.21.15.86:0/3643234756 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd7880095f0 con 0x7fd7940f6510 2024-02-19T22:05:14.236 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:14.235+0000 7fd79b138700 1 -- 172.21.15.86:0/3643234756 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd794045ad0 con 0x7fd7940f6510 2024-02-19T22:05:14.236 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:14.235+0000 7fd7917fa700 1 --2- 172.21.15.86:0/3643234756 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fd78402ff10 0x7fd7840323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:05:14.237 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:14.235+0000 7fd7917fa700 1 -- 172.21.15.86:0/3643234756 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd788040f60 con 0x7fd7940f6510 2024-02-19T22:05:14.237 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:14.235+0000 7fd793fff700 1 --2- 172.21.15.86:0/3643234756 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fd78402ff10 0x7fd7840323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:05:14.237 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:14.236+0000 7fd793fff700 1 --2- 172.21.15.86:0/3643234756 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fd78402ff10 0x7fd7840323d0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7fd7800096a0 tx=0x7fd780006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:05:14.239 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:14.240+0000 7fd7917fa700 1 -- 172.21.15.86:0/3643234756 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd788014420 con 0x7fd7940f6510 2024-02-19T22:05:14.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:14.446+0000 7fd79b138700 1 -- 172.21.15.86:0/3643234756 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd794108210 con 0x7fd7940f6510 2024-02-19T22:05:14.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:14.447+0000 7fd7917fa700 1 -- 172.21.15.86:0/3643234756 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd788010230 con 0x7fd7940f6510 2024-02-19T22:05:14.447 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T22:05:14.447 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:05:14.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:14.449+0000 7fd79b138700 1 -- 172.21.15.86:0/3643234756 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fd78402ff10 msgr2=0x7fd7840323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:05:14.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:14.449+0000 7fd79b138700 1 --2- 172.21.15.86:0/3643234756 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fd78402ff10 0x7fd7840323d0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7fd7800096a0 tx=0x7fd780006b40).stop 2024-02-19T22:05:14.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:14.449+0000 7fd79b138700 1 -- 172.21.15.86:0/3643234756 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd7940f6510 msgr2=0x7fd7941067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:05:14.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:14.449+0000 7fd79b138700 1 --2- 172.21.15.86:0/3643234756 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd7940f6510 0x7fd7941067e0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0x7fd788005010 tx=0x7fd78800b040).stop 2024-02-19T22:05:14.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:14.449+0000 7fd79b138700 1 -- 172.21.15.86:0/3643234756 shutdown_connections 2024-02-19T22:05:14.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:14.449+0000 7fd79b138700 1 --2- 172.21.15.86:0/3643234756 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fd78402ff10 0x7fd7840323d0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:05:14.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:14.449+0000 7fd79b138700 1 --2- 172.21.15.86:0/3643234756 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd7940f6510 0x7fd7941067e0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:05:14.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:14.449+0000 7fd79b138700 1 -- 172.21.15.86:0/3643234756 >> 172.21.15.86:0/3643234756 conn(0x7fd7940f1c70 msgr2=0x7fd7940f26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:05:14.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:14.449+0000 7fd79b138700 1 -- 172.21.15.86:0/3643234756 shutdown_connections 2024-02-19T22:05:14.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:14.449+0000 7fd79b138700 1 -- 172.21.15.86:0/3643234756 wait complete. 2024-02-19T22:05:14.449 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T22:05:15.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:05:14 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:05:13.516556+0000 mgr.smithi053.fyhcdi (mgr.14162) 267 : cluster [DBG] pgmap v253: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:05:15.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:05:14 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:05:14.448280+0000 mon.smithi053 (mon.0) 269 : audit [DBG] from='client.? 172.21.15.86:0/3643234756' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:05:16.101 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:05:16.101 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:05:17.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:05:16 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:05:15.516801+0000 mgr.smithi053.fyhcdi (mgr.14162) 268 : cluster [DBG] pgmap v254: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:05:18.024 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:18.024+0000 7f88116f2700 1 -- 172.21.15.86:0/2811129696 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f880c0f6510 msgr2=0x7f880c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:05:18.024 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:18.024+0000 7f88116f2700 1 --2- 172.21.15.86:0/2811129696 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f880c0f6510 0x7f880c0f6930 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 rx=0x7f87fc009fb0 tx=0x7f87fc009b30).stop 2024-02-19T22:05:18.024 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:18.024+0000 7f88116f2700 1 -- 172.21.15.86:0/2811129696 shutdown_connections 2024-02-19T22:05:18.024 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:18.024+0000 7f88116f2700 1 --2- 172.21.15.86:0/2811129696 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f880c0f6510 0x7f880c0f6930 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:05:18.024 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:18.024+0000 7f88116f2700 1 -- 172.21.15.86:0/2811129696 >> 172.21.15.86:0/2811129696 conn(0x7f880c0f1c70 msgr2=0x7f880c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:05:18.024 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:18.024+0000 7f88116f2700 1 -- 172.21.15.86:0/2811129696 shutdown_connections 2024-02-19T22:05:18.024 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:18.024+0000 7f88116f2700 1 -- 172.21.15.86:0/2811129696 wait complete. 2024-02-19T22:05:18.026 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:18.025+0000 7f88116f2700 1 Processor -- start 2024-02-19T22:05:18.026 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:18.025+0000 7f88116f2700 1 -- start start 2024-02-19T22:05:18.027 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:18.025+0000 7f88116f2700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f880c0f6510 0x7f880c06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:05:18.027 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:18.025+0000 7f88116f2700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f880c069e20 con 0x7f880c0f6510 2024-02-19T22:05:18.027 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:18.025+0000 7f880affd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f880c0f6510 0x7f880c06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:05:18.027 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:18.025+0000 7f880affd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f880c0f6510 0x7f880c06b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:35148/0 (socket says 172.21.15.86:35148) 2024-02-19T22:05:18.027 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:18.025+0000 7f880affd700 1 -- 172.21.15.86:0/4184646057 learned_addr learned my addr 172.21.15.86:0/4184646057 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T22:05:18.027 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:18.025+0000 7f880affd700 1 -- 172.21.15.86:0/4184646057 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87fc005040 con 0x7f880c0f6510 2024-02-19T22:05:18.027 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:18.026+0000 7f880affd700 1 --2- 172.21.15.86:0/4184646057 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f880c0f6510 0x7f880c06b9a0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 rx=0x7f87fc0089f0 tx=0x7f87fc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:05:18.027 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:18.026+0000 7f87f37fe700 1 -- 172.21.15.86:0/4184646057 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f87fc009320 con 0x7f880c0f6510 2024-02-19T22:05:18.027 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:18.026+0000 7f88116f2700 1 -- 172.21.15.86:0/4184646057 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f880c069fc0 con 0x7f880c0f6510 2024-02-19T22:05:18.027 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:18.026+0000 7f87f37fe700 1 -- 172.21.15.86:0/4184646057 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f87fc013380 con 0x7f880c0f6510 2024-02-19T22:05:18.027 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:18.026+0000 7f87f37fe700 1 -- 172.21.15.86:0/4184646057 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f87fc01ed80 con 0x7f880c0f6510 2024-02-19T22:05:18.027 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:18.026+0000 7f88116f2700 1 -- 172.21.15.86:0/4184646057 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f880c06a450 con 0x7f880c0f6510 2024-02-19T22:05:18.027 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:18.027+0000 7f87f37fe700 1 -- 172.21.15.86:0/4184646057 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f87fc01e420 con 0x7f880c0f6510 2024-02-19T22:05:18.027 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:18.027+0000 7f88116f2700 1 -- 172.21.15.86:0/4184646057 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f880c059100 con 0x7f880c0f6510 2024-02-19T22:05:18.027 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:18.027+0000 7f87f37fe700 1 --2- 172.21.15.86:0/4184646057 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f87f402ff60 0x7f87f4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:05:18.028 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:18.027+0000 7f87f37fe700 1 -- 172.21.15.86:0/4184646057 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f87fc041060 con 0x7f880c0f6510 2024-02-19T22:05:18.028 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:18.027+0000 7f880a7fc700 1 --2- 172.21.15.86:0/4184646057 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f87f402ff60 0x7f87f4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:05:18.028 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:18.027+0000 7f880a7fc700 1 --2- 172.21.15.86:0/4184646057 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f87f402ff60 0x7f87f4032420 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f88000096a0 tx=0x7f8800006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:05:18.031 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:18.032+0000 7f87f37fe700 1 -- 172.21.15.86:0/4184646057 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f87fc0095f0 con 0x7f880c0f6510 2024-02-19T22:05:18.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:18.235+0000 7f88116f2700 1 -- 172.21.15.86:0/4184646057 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f880c045ad0 con 0x7f880c0f6510 2024-02-19T22:05:18.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:18.236+0000 7f87f37fe700 1 -- 172.21.15.86:0/4184646057 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f87fc01cb10 con 0x7f880c0f6510 2024-02-19T22:05:18.236 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T22:05:18.236 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:05:18.237 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:18.237+0000 7f88116f2700 1 -- 172.21.15.86:0/4184646057 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f87f402ff60 msgr2=0x7f87f4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:05:18.237 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:18.237+0000 7f88116f2700 1 --2- 172.21.15.86:0/4184646057 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f87f402ff60 0x7f87f4032420 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f88000096a0 tx=0x7f8800006b40).stop 2024-02-19T22:05:18.238 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:18.237+0000 7f88116f2700 1 -- 172.21.15.86:0/4184646057 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f880c0f6510 msgr2=0x7f880c06b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:05:18.238 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:18.237+0000 7f88116f2700 1 --2- 172.21.15.86:0/4184646057 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f880c0f6510 0x7f880c06b9a0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 rx=0x7f87fc0089f0 tx=0x7f87fc00b040).stop 2024-02-19T22:05:18.238 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:18.237+0000 7f88116f2700 1 -- 172.21.15.86:0/4184646057 shutdown_connections 2024-02-19T22:05:18.238 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:18.237+0000 7f88116f2700 1 --2- 172.21.15.86:0/4184646057 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f87f402ff60 0x7f87f4032420 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:05:18.238 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:18.237+0000 7f88116f2700 1 --2- 172.21.15.86:0/4184646057 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f880c0f6510 0x7f880c06b9a0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:05:18.238 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:18.237+0000 7f88116f2700 1 -- 172.21.15.86:0/4184646057 >> 172.21.15.86:0/4184646057 conn(0x7f880c0f1c70 msgr2=0x7f880c0f26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:05:18.238 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:18.237+0000 7f88116f2700 1 -- 172.21.15.86:0/4184646057 shutdown_connections 2024-02-19T22:05:18.238 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:18.237+0000 7f88116f2700 1 -- 172.21.15.86:0/4184646057 wait complete. 2024-02-19T22:05:18.238 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T22:05:19.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:05:18 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:05:17.517059+0000 mgr.smithi053.fyhcdi (mgr.14162) 269 : cluster [DBG] pgmap v255: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:05:19.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:05:18 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:05:18.236976+0000 mon.smithi053 (mon.0) 270 : audit [DBG] from='client.? 172.21.15.86:0/4184646057' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:05:19.829 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:05:19.829 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:05:20.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:05:20 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:05:19.517417+0000 mgr.smithi053.fyhcdi (mgr.14162) 270 : cluster [DBG] pgmap v256: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:05:21.755 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:21.754+0000 7fe582246700 1 -- 172.21.15.86:0/3351917160 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe57c0f6510 msgr2=0x7fe57c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:05:21.755 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:21.754+0000 7fe582246700 1 --2- 172.21.15.86:0/3351917160 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe57c0f6510 0x7fe57c0f6930 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 rx=0x7fe570009fb0 tx=0x7fe570009b30).stop 2024-02-19T22:05:21.756 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:21.754+0000 7fe582246700 1 -- 172.21.15.86:0/3351917160 shutdown_connections 2024-02-19T22:05:21.756 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:21.754+0000 7fe582246700 1 --2- 172.21.15.86:0/3351917160 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe57c0f6510 0x7fe57c0f6930 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:05:21.756 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:21.754+0000 7fe582246700 1 -- 172.21.15.86:0/3351917160 >> 172.21.15.86:0/3351917160 conn(0x7fe57c0f1c70 msgr2=0x7fe57c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:05:21.756 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:21.754+0000 7fe582246700 1 -- 172.21.15.86:0/3351917160 shutdown_connections 2024-02-19T22:05:21.756 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:21.754+0000 7fe582246700 1 -- 172.21.15.86:0/3351917160 wait complete. 2024-02-19T22:05:21.756 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:21.755+0000 7fe582246700 1 Processor -- start 2024-02-19T22:05:21.756 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:21.755+0000 7fe582246700 1 -- start start 2024-02-19T22:05:21.756 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:21.755+0000 7fe582246700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe57c0f6510 0x7fe57c1067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:05:21.756 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:21.755+0000 7fe582246700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe57c106cd0 con 0x7fe57c0f6510 2024-02-19T22:05:21.756 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:21.755+0000 7fe57b7fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe57c0f6510 0x7fe57c1067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:05:21.756 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:21.755+0000 7fe57b7fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe57c0f6510 0x7fe57c1067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:37374/0 (socket says 172.21.15.86:37374) 2024-02-19T22:05:21.756 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:21.755+0000 7fe57b7fe700 1 -- 172.21.15.86:0/732726152 learned_addr learned my addr 172.21.15.86:0/732726152 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T22:05:21.756 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:21.756+0000 7fe57b7fe700 1 -- 172.21.15.86:0/732726152 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe570005040 con 0x7fe57c0f6510 2024-02-19T22:05:21.756 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:21.756+0000 7fe57b7fe700 1 --2- 172.21.15.86:0/732726152 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe57c0f6510 0x7fe57c1067e0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7fe570005010 tx=0x7fe57000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:05:21.758 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:21.756+0000 7fe563fff700 1 -- 172.21.15.86:0/732726152 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe570009320 con 0x7fe57c0f6510 2024-02-19T22:05:21.758 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:21.756+0000 7fe582246700 1 -- 172.21.15.86:0/732726152 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe57c106ed0 con 0x7fe57c0f6510 2024-02-19T22:05:21.758 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:21.756+0000 7fe563fff700 1 -- 172.21.15.86:0/732726152 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe570013380 con 0x7fe57c0f6510 2024-02-19T22:05:21.758 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:21.756+0000 7fe563fff700 1 -- 172.21.15.86:0/732726152 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe570014e50 con 0x7fe57c0f6510 2024-02-19T22:05:21.758 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:21.756+0000 7fe582246700 1 -- 172.21.15.86:0/732726152 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe57c107360 con 0x7fe57c0f6510 2024-02-19T22:05:21.758 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:21.757+0000 7fe563fff700 1 -- 172.21.15.86:0/732726152 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe5700095f0 con 0x7fe57c0f6510 2024-02-19T22:05:21.758 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:21.757+0000 7fe582246700 1 -- 172.21.15.86:0/732726152 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe57c045ad0 con 0x7fe57c0f6510 2024-02-19T22:05:21.758 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:21.757+0000 7fe563fff700 1 --2- 172.21.15.86:0/732726152 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fe56402ff10 0x7fe5640323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:05:21.758 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:21.757+0000 7fe563fff700 1 -- 172.21.15.86:0/732726152 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe570040f80 con 0x7fe57c0f6510 2024-02-19T22:05:21.758 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:21.757+0000 7fe57affd700 1 --2- 172.21.15.86:0/732726152 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fe56402ff10 0x7fe5640323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:05:21.759 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:21.758+0000 7fe57affd700 1 --2- 172.21.15.86:0/732726152 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fe56402ff10 0x7fe5640323d0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7fe56c0096a0 tx=0x7fe56c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:05:21.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:21.762+0000 7fe563fff700 1 -- 172.21.15.86:0/732726152 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe570014420 con 0x7fe57c0f6510 2024-02-19T22:05:21.968 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:21.968+0000 7fe582246700 1 -- 172.21.15.86:0/732726152 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe57c108210 con 0x7fe57c0f6510 2024-02-19T22:05:21.969 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:21.969+0000 7fe563fff700 1 -- 172.21.15.86:0/732726152 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe570010230 con 0x7fe57c0f6510 2024-02-19T22:05:21.969 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T22:05:21.969 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:05:21.970 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:21.971+0000 7fe582246700 1 -- 172.21.15.86:0/732726152 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fe56402ff10 msgr2=0x7fe5640323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:05:21.971 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:21.971+0000 7fe582246700 1 --2- 172.21.15.86:0/732726152 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fe56402ff10 0x7fe5640323d0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7fe56c0096a0 tx=0x7fe56c006b40).stop 2024-02-19T22:05:21.971 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:21.971+0000 7fe582246700 1 -- 172.21.15.86:0/732726152 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe57c0f6510 msgr2=0x7fe57c1067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:05:21.971 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:21.971+0000 7fe582246700 1 --2- 172.21.15.86:0/732726152 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe57c0f6510 0x7fe57c1067e0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7fe570005010 tx=0x7fe57000b040).stop 2024-02-19T22:05:21.971 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:21.971+0000 7fe582246700 1 -- 172.21.15.86:0/732726152 shutdown_connections 2024-02-19T22:05:21.971 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:21.971+0000 7fe582246700 1 --2- 172.21.15.86:0/732726152 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fe56402ff10 0x7fe5640323d0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:05:21.971 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:21.971+0000 7fe582246700 1 --2- 172.21.15.86:0/732726152 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe57c0f6510 0x7fe57c1067e0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:05:21.971 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:21.971+0000 7fe582246700 1 -- 172.21.15.86:0/732726152 >> 172.21.15.86:0/732726152 conn(0x7fe57c0f1c70 msgr2=0x7fe57c0f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:05:21.971 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:21.971+0000 7fe582246700 1 -- 172.21.15.86:0/732726152 shutdown_connections 2024-02-19T22:05:21.971 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:21.971+0000 7fe582246700 1 -- 172.21.15.86:0/732726152 wait complete. 2024-02-19T22:05:21.971 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T22:05:23.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:05:22 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:05:21.517821+0000 mgr.smithi053.fyhcdi (mgr.14162) 271 : cluster [DBG] pgmap v257: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:05:23.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:05:22 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:05:21.970167+0000 mon.smithi053 (mon.0) 271 : audit [DBG] from='client.? 172.21.15.86:0/732726152' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:05:23.635 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:05:23.636 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:05:25.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:05:24 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:05:23.518229+0000 mgr.smithi053.fyhcdi (mgr.14162) 272 : cluster [DBG] pgmap v258: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:05:25.595 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:25.593+0000 7f3d6cf3e700 1 -- 172.21.15.86:0/1124422012 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3d680f5410 msgr2=0x7f3d680f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:05:25.595 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:25.593+0000 7f3d6cf3e700 1 --2- 172.21.15.86:0/1124422012 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3d680f5410 0x7f3d680f5830 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 rx=0x7f3d5c009fb0 tx=0x7f3d5c009b30).stop 2024-02-19T22:05:25.595 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:25.593+0000 7f3d6cf3e700 1 -- 172.21.15.86:0/1124422012 shutdown_connections 2024-02-19T22:05:25.596 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:25.593+0000 7f3d6cf3e700 1 --2- 172.21.15.86:0/1124422012 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3d680f5410 0x7f3d680f5830 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:05:25.596 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:25.593+0000 7f3d6cf3e700 1 -- 172.21.15.86:0/1124422012 >> 172.21.15.86:0/1124422012 conn(0x7f3d680f0b70 msgr2=0x7f3d680f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:05:25.596 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:25.593+0000 7f3d6cf3e700 1 -- 172.21.15.86:0/1124422012 shutdown_connections 2024-02-19T22:05:25.596 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:25.593+0000 7f3d6cf3e700 1 -- 172.21.15.86:0/1124422012 wait complete. 2024-02-19T22:05:25.596 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:25.593+0000 7f3d6cf3e700 1 Processor -- start 2024-02-19T22:05:25.596 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:25.593+0000 7f3d6cf3e700 1 -- start start 2024-02-19T22:05:25.596 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:25.594+0000 7f3d6cf3e700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3d680f5410 0x7f3d681094e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:05:25.596 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:25.594+0000 7f3d6cf3e700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d68060240 con 0x7f3d680f5410 2024-02-19T22:05:25.596 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:25.594+0000 7f3d6659c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3d680f5410 0x7f3d681094e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:05:25.596 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:25.594+0000 7f3d6659c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3d680f5410 0x7f3d681094e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:37402/0 (socket says 172.21.15.86:37402) 2024-02-19T22:05:25.596 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:25.594+0000 7f3d6659c700 1 -- 172.21.15.86:0/2371668700 learned_addr learned my addr 172.21.15.86:0/2371668700 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T22:05:25.596 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:25.594+0000 7f3d6659c700 1 -- 172.21.15.86:0/2371668700 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d5c005040 con 0x7f3d680f5410 2024-02-19T22:05:25.596 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:25.594+0000 7f3d6659c700 1 --2- 172.21.15.86:0/2371668700 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3d680f5410 0x7f3d681094e0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0x7f3d5c0089f0 tx=0x7f3d5c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:05:25.597 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:25.595+0000 7f3d56ffd700 1 -- 172.21.15.86:0/2371668700 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3d5c009320 con 0x7f3d680f5410 2024-02-19T22:05:25.597 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:25.595+0000 7f3d6cf3e700 1 -- 172.21.15.86:0/2371668700 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d68060440 con 0x7f3d680f5410 2024-02-19T22:05:25.597 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:25.595+0000 7f3d56ffd700 1 -- 172.21.15.86:0/2371668700 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3d5c013380 con 0x7f3d680f5410 2024-02-19T22:05:25.597 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:25.595+0000 7f3d56ffd700 1 -- 172.21.15.86:0/2371668700 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3d5c01ed80 con 0x7f3d680f5410 2024-02-19T22:05:25.597 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:25.595+0000 7f3d6cf3e700 1 -- 172.21.15.86:0/2371668700 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d68109b90 con 0x7f3d680f5410 2024-02-19T22:05:25.597 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:25.595+0000 7f3d56ffd700 1 -- 172.21.15.86:0/2371668700 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3d5c01e420 con 0x7f3d680f5410 2024-02-19T22:05:25.597 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:25.596+0000 7f3d6cf3e700 1 -- 172.21.15.86:0/2371668700 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d68045ad0 con 0x7f3d680f5410 2024-02-19T22:05:25.597 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:25.596+0000 7f3d56ffd700 1 --2- 172.21.15.86:0/2371668700 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f3d5002ff60 0x7f3d50032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:05:25.597 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:25.596+0000 7f3d56ffd700 1 -- 172.21.15.86:0/2371668700 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3d5c040e70 con 0x7f3d680f5410 2024-02-19T22:05:25.597 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:25.596+0000 7f3d65d9b700 1 --2- 172.21.15.86:0/2371668700 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f3d5002ff60 0x7f3d50032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:05:25.597 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:25.596+0000 7f3d65d9b700 1 --2- 172.21.15.86:0/2371668700 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f3d5002ff60 0x7f3d50032420 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f3d580096a0 tx=0x7f3d58006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:05:25.599 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:25.600+0000 7f3d56ffd700 1 -- 172.21.15.86:0/2371668700 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3d5c00e2a0 con 0x7f3d680f5410 2024-02-19T22:05:25.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:25.809+0000 7f3d6cf3e700 1 -- 172.21.15.86:0/2371668700 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3d6810a3b0 con 0x7f3d680f5410 2024-02-19T22:05:25.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:25.810+0000 7f3d56ffd700 1 -- 172.21.15.86:0/2371668700 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3d5c00e2a0 con 0x7f3d680f5410 2024-02-19T22:05:25.809 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T22:05:25.809 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:05:25.810 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:25.811+0000 7f3d6cf3e700 1 -- 172.21.15.86:0/2371668700 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f3d5002ff60 msgr2=0x7f3d50032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:05:25.810 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:25.811+0000 7f3d6cf3e700 1 --2- 172.21.15.86:0/2371668700 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f3d5002ff60 0x7f3d50032420 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f3d580096a0 tx=0x7f3d58006b40).stop 2024-02-19T22:05:25.810 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:25.811+0000 7f3d6cf3e700 1 -- 172.21.15.86:0/2371668700 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3d680f5410 msgr2=0x7f3d681094e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:05:25.810 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:25.811+0000 7f3d6cf3e700 1 --2- 172.21.15.86:0/2371668700 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3d680f5410 0x7f3d681094e0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0x7f3d5c0089f0 tx=0x7f3d5c00b040).stop 2024-02-19T22:05:25.810 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:25.811+0000 7f3d6cf3e700 1 -- 172.21.15.86:0/2371668700 shutdown_connections 2024-02-19T22:05:25.810 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:25.811+0000 7f3d6cf3e700 1 --2- 172.21.15.86:0/2371668700 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f3d5002ff60 0x7f3d50032420 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:05:25.811 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:25.811+0000 7f3d6cf3e700 1 --2- 172.21.15.86:0/2371668700 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3d680f5410 0x7f3d681094e0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:05:25.811 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:25.811+0000 7f3d6cf3e700 1 -- 172.21.15.86:0/2371668700 >> 172.21.15.86:0/2371668700 conn(0x7f3d680f0b70 msgr2=0x7f3d680f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:05:25.811 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:25.811+0000 7f3d6cf3e700 1 -- 172.21.15.86:0/2371668700 shutdown_connections 2024-02-19T22:05:25.811 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:25.811+0000 7f3d6cf3e700 1 -- 172.21.15.86:0/2371668700 wait complete. 2024-02-19T22:05:25.811 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T22:05:27.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:05:26 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:05:25.518508+0000 mgr.smithi053.fyhcdi (mgr.14162) 273 : cluster [DBG] pgmap v259: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:05:27.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:05:26 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:05:25.810191+0000 mon.smithi053 (mon.0) 272 : audit [DBG] from='client.? 172.21.15.86:0/2371668700' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:05:27.571 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:05:27.571 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:05:29.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:05:28 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:05:27.518842+0000 mgr.smithi053.fyhcdi (mgr.14162) 274 : cluster [DBG] pgmap v260: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:05:29.633 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:29.629+0000 7fdbc7f03700 1 -- 172.21.15.86:0/244636968 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdbc00f5410 msgr2=0x7fdbc00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:05:29.634 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:29.629+0000 7fdbc7f03700 1 --2- 172.21.15.86:0/244636968 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdbc00f5410 0x7fdbc00f5830 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 rx=0x7fdbb0009fb0 tx=0x7fdbb0009b30).stop 2024-02-19T22:05:29.634 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:29.629+0000 7fdbc7f03700 1 -- 172.21.15.86:0/244636968 shutdown_connections 2024-02-19T22:05:29.634 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:29.629+0000 7fdbc7f03700 1 --2- 172.21.15.86:0/244636968 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdbc00f5410 0x7fdbc00f5830 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:05:29.634 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:29.629+0000 7fdbc7f03700 1 -- 172.21.15.86:0/244636968 >> 172.21.15.86:0/244636968 conn(0x7fdbc00f0b70 msgr2=0x7fdbc00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:05:29.634 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:29.629+0000 7fdbc7f03700 1 -- 172.21.15.86:0/244636968 shutdown_connections 2024-02-19T22:05:29.634 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:29.629+0000 7fdbc7f03700 1 -- 172.21.15.86:0/244636968 wait complete. 2024-02-19T22:05:29.634 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:29.630+0000 7fdbc7f03700 1 Processor -- start 2024-02-19T22:05:29.634 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:29.630+0000 7fdbc7f03700 1 -- start start 2024-02-19T22:05:29.634 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:29.630+0000 7fdbc7f03700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdbc00f5410 0x7fdbc01067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:05:29.634 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:29.630+0000 7fdbc7f03700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbc0106cd0 con 0x7fdbc00f5410 2024-02-19T22:05:29.634 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:29.630+0000 7fdbc5c9f700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdbc00f5410 0x7fdbc01067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:05:29.635 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:29.630+0000 7fdbc5c9f700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdbc00f5410 0x7fdbc01067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:43282/0 (socket says 172.21.15.86:43282) 2024-02-19T22:05:29.635 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:29.630+0000 7fdbc5c9f700 1 -- 172.21.15.86:0/481875811 learned_addr learned my addr 172.21.15.86:0/481875811 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T22:05:29.635 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:29.630+0000 7fdbc5c9f700 1 -- 172.21.15.86:0/481875811 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbb0005040 con 0x7fdbc00f5410 2024-02-19T22:05:29.635 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:29.631+0000 7fdbc5c9f700 1 --2- 172.21.15.86:0/481875811 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdbc00f5410 0x7fdbc01067e0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0x7fdbb0005010 tx=0x7fdbb000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:05:29.635 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:29.631+0000 7fdbb67fc700 1 -- 172.21.15.86:0/481875811 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdbb0009320 con 0x7fdbc00f5410 2024-02-19T22:05:29.635 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:29.631+0000 7fdbc7f03700 1 -- 172.21.15.86:0/481875811 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbc0106ed0 con 0x7fdbc00f5410 2024-02-19T22:05:29.635 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:29.631+0000 7fdbb67fc700 1 -- 172.21.15.86:0/481875811 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdbb0013380 con 0x7fdbc00f5410 2024-02-19T22:05:29.635 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:29.631+0000 7fdbb67fc700 1 -- 172.21.15.86:0/481875811 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdbb0014e50 con 0x7fdbc00f5410 2024-02-19T22:05:29.635 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:29.631+0000 7fdbc7f03700 1 -- 172.21.15.86:0/481875811 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbc0107360 con 0x7fdbc00f5410 2024-02-19T22:05:29.635 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:29.632+0000 7fdbb67fc700 1 -- 172.21.15.86:0/481875811 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdbb00095f0 con 0x7fdbc00f5410 2024-02-19T22:05:29.635 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:29.632+0000 7fdbc7f03700 1 -- 172.21.15.86:0/481875811 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbc0045ad0 con 0x7fdbc00f5410 2024-02-19T22:05:29.635 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:29.632+0000 7fdbb67fc700 1 --2- 172.21.15.86:0/481875811 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fdbac02ff10 0x7fdbac0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:05:29.635 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:29.632+0000 7fdbb67fc700 1 -- 172.21.15.86:0/481875811 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdbb0040ef0 con 0x7fdbc00f5410 2024-02-19T22:05:29.636 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:29.632+0000 7fdbc549e700 1 --2- 172.21.15.86:0/481875811 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fdbac02ff10 0x7fdbac0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:05:29.636 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:29.632+0000 7fdbc549e700 1 --2- 172.21.15.86:0/481875811 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fdbac02ff10 0x7fdbac0323d0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7fdbbc0096a0 tx=0x7fdbbc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:05:29.636 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:29.636+0000 7fdbb67fc700 1 -- 172.21.15.86:0/481875811 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdbb0014420 con 0x7fdbc00f5410 2024-02-19T22:05:29.842 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:29.843+0000 7fdbc7f03700 1 -- 172.21.15.86:0/481875811 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdbc00fafa0 con 0x7fdbc00f5410 2024-02-19T22:05:29.843 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:29.844+0000 7fdbb67fc700 1 -- 172.21.15.86:0/481875811 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fdbb0010230 con 0x7fdbc00f5410 2024-02-19T22:05:29.843 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T22:05:29.843 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:05:29.844 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:29.846+0000 7fdbc7f03700 1 -- 172.21.15.86:0/481875811 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fdbac02ff10 msgr2=0x7fdbac0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:05:29.845 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:29.846+0000 7fdbc7f03700 1 --2- 172.21.15.86:0/481875811 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fdbac02ff10 0x7fdbac0323d0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7fdbbc0096a0 tx=0x7fdbbc006b40).stop 2024-02-19T22:05:29.845 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:29.846+0000 7fdbc7f03700 1 -- 172.21.15.86:0/481875811 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdbc00f5410 msgr2=0x7fdbc01067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:05:29.845 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:29.846+0000 7fdbc7f03700 1 --2- 172.21.15.86:0/481875811 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdbc00f5410 0x7fdbc01067e0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0x7fdbb0005010 tx=0x7fdbb000b040).stop 2024-02-19T22:05:29.845 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:29.846+0000 7fdbc7f03700 1 -- 172.21.15.86:0/481875811 shutdown_connections 2024-02-19T22:05:29.845 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:29.846+0000 7fdbc7f03700 1 --2- 172.21.15.86:0/481875811 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fdbac02ff10 0x7fdbac0323d0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:05:29.845 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:29.846+0000 7fdbc7f03700 1 --2- 172.21.15.86:0/481875811 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdbc00f5410 0x7fdbc01067e0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:05:29.845 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:29.846+0000 7fdbc7f03700 1 -- 172.21.15.86:0/481875811 >> 172.21.15.86:0/481875811 conn(0x7fdbc00f0b70 msgr2=0x7fdbc005fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:05:29.845 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:29.846+0000 7fdbc7f03700 1 -- 172.21.15.86:0/481875811 shutdown_connections 2024-02-19T22:05:29.845 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:29.846+0000 7fdbc7f03700 1 -- 172.21.15.86:0/481875811 wait complete. 2024-02-19T22:05:29.845 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T22:05:30.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:05:30 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:05:29.519265+0000 mgr.smithi053.fyhcdi (mgr.14162) 275 : cluster [DBG] pgmap v261: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:05:30.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:05:30 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:05:29.844307+0000 mon.smithi053 (mon.0) 273 : audit [DBG] from='client.? 172.21.15.86:0/481875811' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:05:31.591 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:05:31.591 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:05:33.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:05:32 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:05:31.519538+0000 mgr.smithi053.fyhcdi (mgr.14162) 276 : cluster [DBG] pgmap v262: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:05:33.483 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:33.482+0000 7fd6fcce1700 1 -- 172.21.15.86:0/1046031533 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd6f80f5410 msgr2=0x7fd6f80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:05:33.483 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:33.482+0000 7fd6fcce1700 1 --2- 172.21.15.86:0/1046031533 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd6f80f5410 0x7fd6f80f5830 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 rx=0x7fd6e0009fb0 tx=0x7fd6e0009b30).stop 2024-02-19T22:05:33.483 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:33.482+0000 7fd6fcce1700 1 -- 172.21.15.86:0/1046031533 shutdown_connections 2024-02-19T22:05:33.483 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:33.482+0000 7fd6fcce1700 1 --2- 172.21.15.86:0/1046031533 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd6f80f5410 0x7fd6f80f5830 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:05:33.483 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:33.482+0000 7fd6fcce1700 1 -- 172.21.15.86:0/1046031533 >> 172.21.15.86:0/1046031533 conn(0x7fd6f80f0b70 msgr2=0x7fd6f80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:05:33.484 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:33.482+0000 7fd6fcce1700 1 -- 172.21.15.86:0/1046031533 shutdown_connections 2024-02-19T22:05:33.484 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:33.482+0000 7fd6fcce1700 1 -- 172.21.15.86:0/1046031533 wait complete. 2024-02-19T22:05:33.484 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:33.482+0000 7fd6fcce1700 1 Processor -- start 2024-02-19T22:05:33.484 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:33.482+0000 7fd6fcce1700 1 -- start start 2024-02-19T22:05:33.484 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:33.482+0000 7fd6fcce1700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd6f80f5410 0x7fd6f81067b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:05:33.484 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:33.482+0000 7fd6fcce1700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6f8106ca0 con 0x7fd6f80f5410 2024-02-19T22:05:33.484 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:33.483+0000 7fd6f659c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd6f80f5410 0x7fd6f81067b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:05:33.484 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:33.483+0000 7fd6f659c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd6f80f5410 0x7fd6f81067b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:43294/0 (socket says 172.21.15.86:43294) 2024-02-19T22:05:33.484 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:33.483+0000 7fd6f659c700 1 -- 172.21.15.86:0/2163415600 learned_addr learned my addr 172.21.15.86:0/2163415600 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T22:05:33.484 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:33.483+0000 7fd6f659c700 1 -- 172.21.15.86:0/2163415600 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6e0005040 con 0x7fd6f80f5410 2024-02-19T22:05:33.484 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:33.483+0000 7fd6f659c700 1 --2- 172.21.15.86:0/2163415600 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd6f80f5410 0x7fd6f81067b0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7fd6e00089f0 tx=0x7fd6e000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:05:33.484 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:33.483+0000 7fd6eeffd700 1 -- 172.21.15.86:0/2163415600 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd6e0009320 con 0x7fd6f80f5410 2024-02-19T22:05:33.484 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:33.483+0000 7fd6fcce1700 1 -- 172.21.15.86:0/2163415600 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6f8106ea0 con 0x7fd6f80f5410 2024-02-19T22:05:33.485 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:33.483+0000 7fd6eeffd700 1 -- 172.21.15.86:0/2163415600 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd6e0013380 con 0x7fd6f80f5410 2024-02-19T22:05:33.485 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:33.483+0000 7fd6eeffd700 1 -- 172.21.15.86:0/2163415600 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd6e001ed80 con 0x7fd6f80f5410 2024-02-19T22:05:33.486 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:33.483+0000 7fd6fcce1700 1 -- 172.21.15.86:0/2163415600 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6f8107330 con 0x7fd6f80f5410 2024-02-19T22:05:33.486 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:33.484+0000 7fd6eeffd700 1 -- 172.21.15.86:0/2163415600 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd6e001e420 con 0x7fd6f80f5410 2024-02-19T22:05:33.486 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:33.484+0000 7fd6fcce1700 1 -- 172.21.15.86:0/2163415600 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6f8045ad0 con 0x7fd6f80f5410 2024-02-19T22:05:33.486 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:33.484+0000 7fd6eeffd700 1 --2- 172.21.15.86:0/2163415600 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fd6e402ff60 0x7fd6e4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:05:33.486 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:33.484+0000 7fd6eeffd700 1 -- 172.21.15.86:0/2163415600 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd6e00410c0 con 0x7fd6f80f5410 2024-02-19T22:05:33.486 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:33.484+0000 7fd6f5d9b700 1 --2- 172.21.15.86:0/2163415600 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fd6e402ff60 0x7fd6e4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:05:33.486 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:33.485+0000 7fd6f5d9b700 1 --2- 172.21.15.86:0/2163415600 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fd6e402ff60 0x7fd6e4032420 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7fd6e80096a0 tx=0x7fd6e8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:05:33.488 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:33.489+0000 7fd6eeffd700 1 -- 172.21.15.86:0/2163415600 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd6e00095f0 con 0x7fd6f80f5410 2024-02-19T22:05:33.692 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:33.692+0000 7fd6fcce1700 1 -- 172.21.15.86:0/2163415600 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd6f80fafa0 con 0x7fd6f80f5410 2024-02-19T22:05:33.692 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:33.693+0000 7fd6eeffd700 1 -- 172.21.15.86:0/2163415600 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd6e001cb10 con 0x7fd6f80f5410 2024-02-19T22:05:33.693 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T22:05:33.693 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:05:33.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:33.694+0000 7fd6fcce1700 1 -- 172.21.15.86:0/2163415600 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fd6e402ff60 msgr2=0x7fd6e4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:05:33.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:33.694+0000 7fd6fcce1700 1 --2- 172.21.15.86:0/2163415600 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fd6e402ff60 0x7fd6e4032420 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7fd6e80096a0 tx=0x7fd6e8006b40).stop 2024-02-19T22:05:33.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:33.694+0000 7fd6fcce1700 1 -- 172.21.15.86:0/2163415600 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd6f80f5410 msgr2=0x7fd6f81067b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:05:33.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:33.694+0000 7fd6fcce1700 1 --2- 172.21.15.86:0/2163415600 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd6f80f5410 0x7fd6f81067b0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7fd6e00089f0 tx=0x7fd6e000b040).stop 2024-02-19T22:05:33.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:33.694+0000 7fd6fcce1700 1 -- 172.21.15.86:0/2163415600 shutdown_connections 2024-02-19T22:05:33.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:33.694+0000 7fd6fcce1700 1 --2- 172.21.15.86:0/2163415600 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fd6e402ff60 0x7fd6e4032420 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:05:33.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:33.694+0000 7fd6fcce1700 1 --2- 172.21.15.86:0/2163415600 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd6f80f5410 0x7fd6f81067b0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:05:33.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:33.694+0000 7fd6fcce1700 1 -- 172.21.15.86:0/2163415600 >> 172.21.15.86:0/2163415600 conn(0x7fd6f80f0b70 msgr2=0x7fd6f805fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:05:33.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:33.694+0000 7fd6fcce1700 1 -- 172.21.15.86:0/2163415600 shutdown_connections 2024-02-19T22:05:33.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:33.694+0000 7fd6fcce1700 1 -- 172.21.15.86:0/2163415600 wait complete. 2024-02-19T22:05:33.695 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T22:05:35.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:05:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:05:33.519816+0000 mgr.smithi053.fyhcdi (mgr.14162) 277 : cluster [DBG] pgmap v263: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:05:35.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:05:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:05:33.693894+0000 mon.smithi053 (mon.0) 274 : audit [DBG] from='client.? 172.21.15.86:0/2163415600' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:05:35.372 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:05:35.373 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:05:37.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:05:36 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:05:35.520123+0000 mgr.smithi053.fyhcdi (mgr.14162) 278 : cluster [DBG] pgmap v264: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:05:37.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:05:36 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:05:35.605481+0000 mon.smithi053 (mon.0) 275 : audit [INF] from='mgr.14162 172.21.15.53:0/2860358250' entity='mgr.smithi053.fyhcdi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi053.fyhcdi/mirror_snapshot_schedule"}]: dispatch 2024-02-19T22:05:37.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:05:36 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:05:35.608326+0000 mon.smithi053 (mon.0) 276 : audit [INF] from='mgr.14162 172.21.15.53:0/2860358250' entity='mgr.smithi053.fyhcdi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi053.fyhcdi/trash_purge_schedule"}]: dispatch 2024-02-19T22:05:37.482 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:37.478+0000 7fb3e7c2f700 1 -- 172.21.15.86:0/354703908 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb3e00f5410 msgr2=0x7fb3e00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:05:37.482 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:37.478+0000 7fb3e7c2f700 1 --2- 172.21.15.86:0/354703908 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb3e00f5410 0x7fb3e00f5830 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 rx=0x7fb3d0009fb0 tx=0x7fb3d0009b30).stop 2024-02-19T22:05:37.482 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:37.478+0000 7fb3e7c2f700 1 -- 172.21.15.86:0/354703908 shutdown_connections 2024-02-19T22:05:37.482 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:37.478+0000 7fb3e7c2f700 1 --2- 172.21.15.86:0/354703908 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb3e00f5410 0x7fb3e00f5830 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:05:37.482 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:37.478+0000 7fb3e7c2f700 1 -- 172.21.15.86:0/354703908 >> 172.21.15.86:0/354703908 conn(0x7fb3e00f0b70 msgr2=0x7fb3e00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:05:37.482 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:37.478+0000 7fb3e7c2f700 1 -- 172.21.15.86:0/354703908 shutdown_connections 2024-02-19T22:05:37.482 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:37.478+0000 7fb3e7c2f700 1 -- 172.21.15.86:0/354703908 wait complete. 2024-02-19T22:05:37.482 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:37.479+0000 7fb3e7c2f700 1 Processor -- start 2024-02-19T22:05:37.482 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:37.479+0000 7fb3e7c2f700 1 -- start start 2024-02-19T22:05:37.482 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:37.479+0000 7fb3e7c2f700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb3e00f5410 0x7fb3e01056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:05:37.483 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:37.479+0000 7fb3e7c2f700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3e0105bd0 con 0x7fb3e00f5410 2024-02-19T22:05:37.483 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:37.479+0000 7fb3e59cb700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb3e00f5410 0x7fb3e01056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:05:37.483 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:37.479+0000 7fb3e59cb700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb3e00f5410 0x7fb3e01056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:43308/0 (socket says 172.21.15.86:43308) 2024-02-19T22:05:37.483 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:37.479+0000 7fb3e59cb700 1 -- 172.21.15.86:0/235585222 learned_addr learned my addr 172.21.15.86:0/235585222 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T22:05:37.483 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:37.479+0000 7fb3e59cb700 1 -- 172.21.15.86:0/235585222 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3d0005040 con 0x7fb3e00f5410 2024-02-19T22:05:37.483 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:37.480+0000 7fb3e59cb700 1 --2- 172.21.15.86:0/235585222 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb3e00f5410 0x7fb3e01056e0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7fb3d0005010 tx=0x7fb3d000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:05:37.483 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:37.480+0000 7fb3d67fc700 1 -- 172.21.15.86:0/235585222 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb3d0009320 con 0x7fb3e00f5410 2024-02-19T22:05:37.483 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:37.480+0000 7fb3e7c2f700 1 -- 172.21.15.86:0/235585222 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3e0105dd0 con 0x7fb3e00f5410 2024-02-19T22:05:37.483 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:37.480+0000 7fb3d67fc700 1 -- 172.21.15.86:0/235585222 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb3d0013380 con 0x7fb3e00f5410 2024-02-19T22:05:37.483 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:37.480+0000 7fb3d67fc700 1 -- 172.21.15.86:0/235585222 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb3d0014e50 con 0x7fb3e00f5410 2024-02-19T22:05:37.483 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:37.480+0000 7fb3e7c2f700 1 -- 172.21.15.86:0/235585222 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3e0106260 con 0x7fb3e00f5410 2024-02-19T22:05:37.483 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:37.481+0000 7fb3d67fc700 1 -- 172.21.15.86:0/235585222 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb3d00095f0 con 0x7fb3e00f5410 2024-02-19T22:05:37.483 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:37.481+0000 7fb3e7c2f700 1 -- 172.21.15.86:0/235585222 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3e0045ad0 con 0x7fb3e00f5410 2024-02-19T22:05:37.483 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:37.481+0000 7fb3d67fc700 1 --2- 172.21.15.86:0/235585222 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fb3cc02ff10 0x7fb3cc0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:05:37.483 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:37.481+0000 7fb3d67fc700 1 -- 172.21.15.86:0/235585222 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb3d0040fa0 con 0x7fb3e00f5410 2024-02-19T22:05:37.484 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:37.481+0000 7fb3e51ca700 1 --2- 172.21.15.86:0/235585222 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fb3cc02ff10 0x7fb3cc0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:05:37.484 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:37.481+0000 7fb3e51ca700 1 --2- 172.21.15.86:0/235585222 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fb3cc02ff10 0x7fb3cc0323d0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7fb3dc0096a0 tx=0x7fb3dc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:05:37.485 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:37.486+0000 7fb3d67fc700 1 -- 172.21.15.86:0/235585222 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb3d001d990 con 0x7fb3e00f5410 2024-02-19T22:05:37.696 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:37.696+0000 7fb3e7c2f700 1 -- 172.21.15.86:0/235585222 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb3e0107110 con 0x7fb3e00f5410 2024-02-19T22:05:37.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:37.697+0000 7fb3d67fc700 1 -- 172.21.15.86:0/235585222 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb3d0010230 con 0x7fb3e00f5410 2024-02-19T22:05:37.697 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T22:05:37.697 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:05:37.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:37.699+0000 7fb3e7c2f700 1 -- 172.21.15.86:0/235585222 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fb3cc02ff10 msgr2=0x7fb3cc0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:05:37.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:37.699+0000 7fb3e7c2f700 1 --2- 172.21.15.86:0/235585222 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fb3cc02ff10 0x7fb3cc0323d0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7fb3dc0096a0 tx=0x7fb3dc006b40).stop 2024-02-19T22:05:37.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:37.699+0000 7fb3e7c2f700 1 -- 172.21.15.86:0/235585222 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb3e00f5410 msgr2=0x7fb3e01056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:05:37.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:37.699+0000 7fb3e7c2f700 1 --2- 172.21.15.86:0/235585222 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb3e00f5410 0x7fb3e01056e0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7fb3d0005010 tx=0x7fb3d000b040).stop 2024-02-19T22:05:37.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:37.699+0000 7fb3e7c2f700 1 -- 172.21.15.86:0/235585222 shutdown_connections 2024-02-19T22:05:37.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:37.699+0000 7fb3e7c2f700 1 --2- 172.21.15.86:0/235585222 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fb3cc02ff10 0x7fb3cc0323d0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:05:37.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:37.699+0000 7fb3e7c2f700 1 --2- 172.21.15.86:0/235585222 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb3e00f5410 0x7fb3e01056e0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:05:37.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:37.699+0000 7fb3e7c2f700 1 -- 172.21.15.86:0/235585222 >> 172.21.15.86:0/235585222 conn(0x7fb3e00f0b70 msgr2=0x7fb3e00f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:05:37.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:37.699+0000 7fb3e7c2f700 1 -- 172.21.15.86:0/235585222 shutdown_connections 2024-02-19T22:05:37.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:37.699+0000 7fb3e7c2f700 1 -- 172.21.15.86:0/235585222 wait complete. 2024-02-19T22:05:37.699 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T22:05:39.055 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:05:38 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:05:37.520479+0000 mgr.smithi053.fyhcdi (mgr.14162) 279 : cluster [DBG] pgmap v265: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:05:39.055 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:05:38 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:05:37.698388+0000 mon.smithi053 (mon.0) 277 : audit [DBG] from='client.? 172.21.15.86:0/235585222' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:05:39.495 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:05:39.495 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:05:40.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:05:40 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:05:39.520869+0000 mgr.smithi053.fyhcdi (mgr.14162) 280 : cluster [DBG] pgmap v266: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:05:41.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:41.555+0000 7f161f9b4700 1 -- 172.21.15.86:0/554824082 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f16180f6510 msgr2=0x7f16180f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:05:41.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:41.555+0000 7f161f9b4700 1 --2- 172.21.15.86:0/554824082 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f16180f6510 0x7f16180f6930 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 rx=0x7f1608009fb0 tx=0x7f1608009b30).stop 2024-02-19T22:05:41.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:41.555+0000 7f161f9b4700 1 -- 172.21.15.86:0/554824082 shutdown_connections 2024-02-19T22:05:41.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:41.555+0000 7f161f9b4700 1 --2- 172.21.15.86:0/554824082 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f16180f6510 0x7f16180f6930 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:05:41.557 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:41.555+0000 7f161f9b4700 1 -- 172.21.15.86:0/554824082 >> 172.21.15.86:0/554824082 conn(0x7f16180f1c70 msgr2=0x7f16180f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:05:41.557 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:41.555+0000 7f161f9b4700 1 -- 172.21.15.86:0/554824082 shutdown_connections 2024-02-19T22:05:41.557 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:41.555+0000 7f161f9b4700 1 -- 172.21.15.86:0/554824082 wait complete. 2024-02-19T22:05:41.557 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:41.555+0000 7f161f9b4700 1 Processor -- start 2024-02-19T22:05:41.557 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:41.555+0000 7f161f9b4700 1 -- start start 2024-02-19T22:05:41.557 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:41.555+0000 7f161f9b4700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f16180f6510 0x7f16181067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:05:41.557 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:41.555+0000 7f161f9b4700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1618106cd0 con 0x7f16180f6510 2024-02-19T22:05:41.557 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:41.556+0000 7f161d750700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f16180f6510 0x7f16181067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:05:41.557 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:41.556+0000 7f161d750700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f16180f6510 0x7f16181067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:45186/0 (socket says 172.21.15.86:45186) 2024-02-19T22:05:41.557 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:41.556+0000 7f161d750700 1 -- 172.21.15.86:0/1042749804 learned_addr learned my addr 172.21.15.86:0/1042749804 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T22:05:41.557 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:41.556+0000 7f161d750700 1 -- 172.21.15.86:0/1042749804 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1608005040 con 0x7f16180f6510 2024-02-19T22:05:41.557 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:41.556+0000 7f161d750700 1 --2- 172.21.15.86:0/1042749804 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f16180f6510 0x7f16181067e0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 rx=0x7f16080089f0 tx=0x7f160800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:05:41.557 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:41.556+0000 7f160dffb700 1 -- 172.21.15.86:0/1042749804 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1608009320 con 0x7f16180f6510 2024-02-19T22:05:41.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:41.556+0000 7f160dffb700 1 -- 172.21.15.86:0/1042749804 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1608013380 con 0x7f16180f6510 2024-02-19T22:05:41.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:41.556+0000 7f161f9b4700 1 -- 172.21.15.86:0/1042749804 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1618106ed0 con 0x7f16180f6510 2024-02-19T22:05:41.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:41.556+0000 7f160dffb700 1 -- 172.21.15.86:0/1042749804 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f160801ed80 con 0x7f16180f6510 2024-02-19T22:05:41.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:41.556+0000 7f161f9b4700 1 -- 172.21.15.86:0/1042749804 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1618107360 con 0x7f16180f6510 2024-02-19T22:05:41.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:41.557+0000 7f160dffb700 1 -- 172.21.15.86:0/1042749804 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f160801e420 con 0x7f16180f6510 2024-02-19T22:05:41.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:41.557+0000 7f161f9b4700 1 -- 172.21.15.86:0/1042749804 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1618045ad0 con 0x7f16180f6510 2024-02-19T22:05:41.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:41.557+0000 7f160dffb700 1 --2- 172.21.15.86:0/1042749804 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f160402ff60 0x7f1604032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:05:41.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:41.557+0000 7f160dffb700 1 -- 172.21.15.86:0/1042749804 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1608041080 con 0x7f16180f6510 2024-02-19T22:05:41.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:41.558+0000 7f161cf4f700 1 --2- 172.21.15.86:0/1042749804 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f160402ff60 0x7f1604032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:05:41.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:41.558+0000 7f161cf4f700 1 --2- 172.21.15.86:0/1042749804 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f160402ff60 0x7f1604032420 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f16140096a0 tx=0x7f1614006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:05:41.562 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:41.562+0000 7f160dffb700 1 -- 172.21.15.86:0/1042749804 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f16080095f0 con 0x7f16180f6510 2024-02-19T22:05:41.770 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:41.770+0000 7f161f9b4700 1 -- 172.21.15.86:0/1042749804 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f16181081b0 con 0x7f16180f6510 2024-02-19T22:05:41.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:41.771+0000 7f160dffb700 1 -- 172.21.15.86:0/1042749804 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f160801cb10 con 0x7f16180f6510 2024-02-19T22:05:41.771 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T22:05:41.771 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:05:41.772 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:41.773+0000 7f161f9b4700 1 -- 172.21.15.86:0/1042749804 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f160402ff60 msgr2=0x7f1604032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:05:41.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:41.773+0000 7f161f9b4700 1 --2- 172.21.15.86:0/1042749804 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f160402ff60 0x7f1604032420 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f16140096a0 tx=0x7f1614006b40).stop 2024-02-19T22:05:41.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:41.773+0000 7f161f9b4700 1 -- 172.21.15.86:0/1042749804 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f16180f6510 msgr2=0x7f16181067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:05:41.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:41.773+0000 7f161f9b4700 1 --2- 172.21.15.86:0/1042749804 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f16180f6510 0x7f16181067e0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 rx=0x7f16080089f0 tx=0x7f160800b040).stop 2024-02-19T22:05:41.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:41.773+0000 7f161f9b4700 1 -- 172.21.15.86:0/1042749804 shutdown_connections 2024-02-19T22:05:41.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:41.773+0000 7f161f9b4700 1 --2- 172.21.15.86:0/1042749804 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f160402ff60 0x7f1604032420 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:05:41.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:41.773+0000 7f161f9b4700 1 --2- 172.21.15.86:0/1042749804 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f16180f6510 0x7f16181067e0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:05:41.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:41.773+0000 7f161f9b4700 1 -- 172.21.15.86:0/1042749804 >> 172.21.15.86:0/1042749804 conn(0x7f16180f1c70 msgr2=0x7f16180f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:05:41.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:41.773+0000 7f161f9b4700 1 -- 172.21.15.86:0/1042749804 shutdown_connections 2024-02-19T22:05:41.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:41.773+0000 7f161f9b4700 1 -- 172.21.15.86:0/1042749804 wait complete. 2024-02-19T22:05:41.773 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T22:05:43.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:05:42 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:05:41.521155+0000 mgr.smithi053.fyhcdi (mgr.14162) 281 : cluster [DBG] pgmap v267: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:05:43.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:05:42 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:05:41.772248+0000 mon.smithi053 (mon.0) 278 : audit [DBG] from='client.? 172.21.15.86:0/1042749804' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:05:43.569 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:05:43.569 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:05:45.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:05:44 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:05:43.521371+0000 mgr.smithi053.fyhcdi (mgr.14162) 282 : cluster [DBG] pgmap v268: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:05:45.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:45.382+0000 7f6c5262e700 1 -- 172.21.15.86:0/803409319 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6c4c0f5410 msgr2=0x7f6c4c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:05:45.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:45.382+0000 7f6c5262e700 1 --2- 172.21.15.86:0/803409319 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6c4c0f5410 0x7f6c4c0f5830 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 rx=0x7f6c3c009fb0 tx=0x7f6c3c009b30).stop 2024-02-19T22:05:45.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:45.382+0000 7f6c5262e700 1 -- 172.21.15.86:0/803409319 shutdown_connections 2024-02-19T22:05:45.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:45.382+0000 7f6c5262e700 1 --2- 172.21.15.86:0/803409319 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6c4c0f5410 0x7f6c4c0f5830 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:05:45.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:45.382+0000 7f6c5262e700 1 -- 172.21.15.86:0/803409319 >> 172.21.15.86:0/803409319 conn(0x7f6c4c0f0b70 msgr2=0x7f6c4c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:05:45.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:45.382+0000 7f6c5262e700 1 -- 172.21.15.86:0/803409319 shutdown_connections 2024-02-19T22:05:45.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:45.382+0000 7f6c5262e700 1 -- 172.21.15.86:0/803409319 wait complete. 2024-02-19T22:05:45.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:45.383+0000 7f6c5262e700 1 Processor -- start 2024-02-19T22:05:45.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:45.383+0000 7f6c5262e700 1 -- start start 2024-02-19T22:05:45.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:45.383+0000 7f6c5262e700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6c4c0f5410 0x7f6c4c1094e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:05:45.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:45.383+0000 7f6c5262e700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c4c060240 con 0x7f6c4c0f5410 2024-02-19T22:05:45.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:45.383+0000 7f6c4bfff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6c4c0f5410 0x7f6c4c1094e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:05:45.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:45.383+0000 7f6c4bfff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6c4c0f5410 0x7f6c4c1094e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:45196/0 (socket says 172.21.15.86:45196) 2024-02-19T22:05:45.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:45.383+0000 7f6c4bfff700 1 -- 172.21.15.86:0/1629967405 learned_addr learned my addr 172.21.15.86:0/1629967405 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T22:05:45.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:45.383+0000 7f6c4bfff700 1 -- 172.21.15.86:0/1629967405 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c3c005040 con 0x7f6c4c0f5410 2024-02-19T22:05:45.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:45.384+0000 7f6c4bfff700 1 --2- 172.21.15.86:0/1629967405 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6c4c0f5410 0x7f6c4c1094e0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 rx=0x7f6c3c005010 tx=0x7f6c3c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:05:45.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:45.384+0000 7f6c48ff9700 1 -- 172.21.15.86:0/1629967405 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6c3c009320 con 0x7f6c4c0f5410 2024-02-19T22:05:45.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:45.384+0000 7f6c48ff9700 1 -- 172.21.15.86:0/1629967405 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6c3c013380 con 0x7f6c4c0f5410 2024-02-19T22:05:45.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:45.384+0000 7f6c5262e700 1 -- 172.21.15.86:0/1629967405 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c4c060440 con 0x7f6c4c0f5410 2024-02-19T22:05:45.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:45.384+0000 7f6c48ff9700 1 -- 172.21.15.86:0/1629967405 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6c3c014e50 con 0x7f6c4c0f5410 2024-02-19T22:05:45.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:45.384+0000 7f6c5262e700 1 -- 172.21.15.86:0/1629967405 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c4c109b90 con 0x7f6c4c0f5410 2024-02-19T22:05:45.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:45.385+0000 7f6c48ff9700 1 -- 172.21.15.86:0/1629967405 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6c3c0263a0 con 0x7f6c4c0f5410 2024-02-19T22:05:45.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:45.385+0000 7f6c5262e700 1 -- 172.21.15.86:0/1629967405 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c4c045ad0 con 0x7f6c4c0f5410 2024-02-19T22:05:45.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:45.385+0000 7f6c48ff9700 1 --2- 172.21.15.86:0/1629967405 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f6c3402ff10 0x7f6c340323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:05:45.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:45.385+0000 7f6c48ff9700 1 -- 172.21.15.86:0/1629967405 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6c3c041290 con 0x7f6c4c0f5410 2024-02-19T22:05:45.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:45.385+0000 7f6c4b7fe700 1 --2- 172.21.15.86:0/1629967405 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f6c3402ff10 0x7f6c340323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:05:45.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:45.385+0000 7f6c4b7fe700 1 --2- 172.21.15.86:0/1629967405 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f6c3402ff10 0x7f6c340323d0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f6c400096a0 tx=0x7f6c40006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:05:45.389 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:45.390+0000 7f6c48ff9700 1 -- 172.21.15.86:0/1629967405 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6c3c0126f0 con 0x7f6c4c0f5410 2024-02-19T22:05:45.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:45.594+0000 7f6c5262e700 1 -- 172.21.15.86:0/1629967405 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6c4c10a3b0 con 0x7f6c4c0f5410 2024-02-19T22:05:45.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:45.594+0000 7f6c48ff9700 1 -- 172.21.15.86:0/1629967405 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6c3c0126f0 con 0x7f6c4c0f5410 2024-02-19T22:05:45.595 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T22:05:45.595 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:05:45.596 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:45.596+0000 7f6c5262e700 1 -- 172.21.15.86:0/1629967405 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f6c3402ff10 msgr2=0x7f6c340323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:05:45.596 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:45.596+0000 7f6c5262e700 1 --2- 172.21.15.86:0/1629967405 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f6c3402ff10 0x7f6c340323d0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f6c400096a0 tx=0x7f6c40006b40).stop 2024-02-19T22:05:45.596 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:45.596+0000 7f6c5262e700 1 -- 172.21.15.86:0/1629967405 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6c4c0f5410 msgr2=0x7f6c4c1094e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:05:45.596 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:45.596+0000 7f6c5262e700 1 --2- 172.21.15.86:0/1629967405 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6c4c0f5410 0x7f6c4c1094e0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 rx=0x7f6c3c005010 tx=0x7f6c3c00b040).stop 2024-02-19T22:05:45.597 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:45.596+0000 7f6c5262e700 1 -- 172.21.15.86:0/1629967405 shutdown_connections 2024-02-19T22:05:45.597 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:45.596+0000 7f6c5262e700 1 --2- 172.21.15.86:0/1629967405 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f6c3402ff10 0x7f6c340323d0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:05:45.597 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:45.596+0000 7f6c5262e700 1 --2- 172.21.15.86:0/1629967405 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6c4c0f5410 0x7f6c4c1094e0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:05:45.597 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:45.596+0000 7f6c5262e700 1 -- 172.21.15.86:0/1629967405 >> 172.21.15.86:0/1629967405 conn(0x7f6c4c0f0b70 msgr2=0x7f6c4c0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:05:45.597 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:45.596+0000 7f6c5262e700 1 -- 172.21.15.86:0/1629967405 shutdown_connections 2024-02-19T22:05:45.597 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:45.596+0000 7f6c5262e700 1 -- 172.21.15.86:0/1629967405 wait complete. 2024-02-19T22:05:45.597 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T22:05:47.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:05:46 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:05:45.521716+0000 mgr.smithi053.fyhcdi (mgr.14162) 283 : cluster [DBG] pgmap v269: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:05:47.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:05:46 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:05:45.595686+0000 mon.smithi053 (mon.0) 279 : audit [DBG] from='client.? 172.21.15.86:0/1629967405' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:05:47.329 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:05:47.330 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:05:49.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:05:48 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:05:47.522166+0000 mgr.smithi053.fyhcdi (mgr.14162) 284 : cluster [DBG] pgmap v270: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:05:49.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:49.164+0000 7f47ffcc9700 1 -- 172.21.15.86:0/4109421692 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f47f80f6510 msgr2=0x7f47f80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:05:49.166 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:49.164+0000 7f47ffcc9700 1 --2- 172.21.15.86:0/4109421692 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f47f80f6510 0x7f47f80f6930 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 rx=0x7f47e8009fb0 tx=0x7f47e8009b30).stop 2024-02-19T22:05:49.166 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:49.164+0000 7f47ffcc9700 1 -- 172.21.15.86:0/4109421692 shutdown_connections 2024-02-19T22:05:49.166 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:49.164+0000 7f47ffcc9700 1 --2- 172.21.15.86:0/4109421692 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f47f80f6510 0x7f47f80f6930 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:05:49.166 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:49.164+0000 7f47ffcc9700 1 -- 172.21.15.86:0/4109421692 >> 172.21.15.86:0/4109421692 conn(0x7f47f80f1c70 msgr2=0x7f47f80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:05:49.166 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:49.164+0000 7f47ffcc9700 1 -- 172.21.15.86:0/4109421692 shutdown_connections 2024-02-19T22:05:49.166 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:49.164+0000 7f47ffcc9700 1 -- 172.21.15.86:0/4109421692 wait complete. 2024-02-19T22:05:49.166 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:49.164+0000 7f47ffcc9700 1 Processor -- start 2024-02-19T22:05:49.166 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:49.165+0000 7f47ffcc9700 1 -- start start 2024-02-19T22:05:49.166 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:49.165+0000 7f47ffcc9700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f47f80f6510 0x7f47f81067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:05:49.166 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:49.165+0000 7f47ffcc9700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47f8106cd0 con 0x7f47f80f6510 2024-02-19T22:05:49.166 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:49.165+0000 7f47fda65700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f47f80f6510 0x7f47f81067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:05:49.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:49.165+0000 7f47fda65700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f47f80f6510 0x7f47f81067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:45218/0 (socket says 172.21.15.86:45218) 2024-02-19T22:05:49.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:49.165+0000 7f47fda65700 1 -- 172.21.15.86:0/231301083 learned_addr learned my addr 172.21.15.86:0/231301083 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T22:05:49.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:49.165+0000 7f47fda65700 1 -- 172.21.15.86:0/231301083 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47e8005040 con 0x7f47f80f6510 2024-02-19T22:05:49.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:49.165+0000 7f47fda65700 1 --2- 172.21.15.86:0/231301083 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f47f80f6510 0x7f47f81067e0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0x7f47e80089f0 tx=0x7f47e800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:05:49.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:49.166+0000 7f47ee7fc700 1 -- 172.21.15.86:0/231301083 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f47e8009320 con 0x7f47f80f6510 2024-02-19T22:05:49.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:49.166+0000 7f47ee7fc700 1 -- 172.21.15.86:0/231301083 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f47e8013380 con 0x7f47f80f6510 2024-02-19T22:05:49.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:49.166+0000 7f47ffcc9700 1 -- 172.21.15.86:0/231301083 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47f8106ed0 con 0x7f47f80f6510 2024-02-19T22:05:49.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:49.166+0000 7f47ee7fc700 1 -- 172.21.15.86:0/231301083 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f47e801ed80 con 0x7f47f80f6510 2024-02-19T22:05:49.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:49.166+0000 7f47ffcc9700 1 -- 172.21.15.86:0/231301083 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47f8107360 con 0x7f47f80f6510 2024-02-19T22:05:49.168 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:49.166+0000 7f47ee7fc700 1 -- 172.21.15.86:0/231301083 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f47e801e420 con 0x7f47f80f6510 2024-02-19T22:05:49.168 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:49.167+0000 7f47ffcc9700 1 -- 172.21.15.86:0/231301083 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47f8045ad0 con 0x7f47f80f6510 2024-02-19T22:05:49.168 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:49.167+0000 7f47ee7fc700 1 --2- 172.21.15.86:0/231301083 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f47e402ff60 0x7f47e4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:05:49.168 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:49.167+0000 7f47ee7fc700 1 -- 172.21.15.86:0/231301083 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f47e8041150 con 0x7f47f80f6510 2024-02-19T22:05:49.168 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:49.167+0000 7f47fd264700 1 --2- 172.21.15.86:0/231301083 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f47e402ff60 0x7f47e4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:05:49.168 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:49.167+0000 7f47fd264700 1 --2- 172.21.15.86:0/231301083 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f47e402ff60 0x7f47e4032420 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f47f40096a0 tx=0x7f47f4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:05:49.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:49.171+0000 7f47ee7fc700 1 -- 172.21.15.86:0/231301083 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f47e80095f0 con 0x7f47f80f6510 2024-02-19T22:05:49.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:49.375+0000 7f47ffcc9700 1 -- 172.21.15.86:0/231301083 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f47f8108210 con 0x7f47f80f6510 2024-02-19T22:05:49.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:49.376+0000 7f47ee7fc700 1 -- 172.21.15.86:0/231301083 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f47e8004490 con 0x7f47f80f6510 2024-02-19T22:05:49.376 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T22:05:49.376 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:05:49.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:49.377+0000 7f47ffcc9700 1 -- 172.21.15.86:0/231301083 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f47e402ff60 msgr2=0x7f47e4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:05:49.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:49.377+0000 7f47ffcc9700 1 --2- 172.21.15.86:0/231301083 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f47e402ff60 0x7f47e4032420 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f47f40096a0 tx=0x7f47f4006b40).stop 2024-02-19T22:05:49.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:49.378+0000 7f47ffcc9700 1 -- 172.21.15.86:0/231301083 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f47f80f6510 msgr2=0x7f47f81067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:05:49.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:49.378+0000 7f47ffcc9700 1 --2- 172.21.15.86:0/231301083 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f47f80f6510 0x7f47f81067e0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0x7f47e80089f0 tx=0x7f47e800b040).stop 2024-02-19T22:05:49.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:49.378+0000 7f47ffcc9700 1 -- 172.21.15.86:0/231301083 shutdown_connections 2024-02-19T22:05:49.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:49.378+0000 7f47ffcc9700 1 --2- 172.21.15.86:0/231301083 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f47e402ff60 0x7f47e4032420 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:05:49.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:49.378+0000 7f47ffcc9700 1 --2- 172.21.15.86:0/231301083 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f47f80f6510 0x7f47f81067e0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:05:49.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:49.378+0000 7f47ffcc9700 1 -- 172.21.15.86:0/231301083 >> 172.21.15.86:0/231301083 conn(0x7f47f80f1c70 msgr2=0x7f47f80f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:05:49.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:49.378+0000 7f47ffcc9700 1 -- 172.21.15.86:0/231301083 shutdown_connections 2024-02-19T22:05:49.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:49.378+0000 7f47ffcc9700 1 -- 172.21.15.86:0/231301083 wait complete. 2024-02-19T22:05:49.379 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T22:05:50.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:05:49 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:05:49.377157+0000 mon.smithi053 (mon.0) 280 : audit [DBG] from='client.? 172.21.15.86:0/231301083' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:05:51.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:05:50 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:05:49.522452+0000 mgr.smithi053.fyhcdi (mgr.14162) 285 : cluster [DBG] pgmap v271: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:05:51.135 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:05:51.135 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:05:52.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:52.907+0000 7f1ce32f8700 1 -- 172.21.15.86:0/1616456164 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1cdc0f5410 msgr2=0x7f1cdc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:05:52.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:52.907+0000 7f1ce32f8700 1 --2- 172.21.15.86:0/1616456164 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1cdc0f5410 0x7f1cdc0f5830 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 rx=0x7f1ccc009fb0 tx=0x7f1ccc009b30).stop 2024-02-19T22:05:52.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:52.907+0000 7f1ce32f8700 1 -- 172.21.15.86:0/1616456164 shutdown_connections 2024-02-19T22:05:52.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:52.907+0000 7f1ce32f8700 1 --2- 172.21.15.86:0/1616456164 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1cdc0f5410 0x7f1cdc0f5830 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:05:52.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:52.907+0000 7f1ce32f8700 1 -- 172.21.15.86:0/1616456164 >> 172.21.15.86:0/1616456164 conn(0x7f1cdc0f0b70 msgr2=0x7f1cdc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:05:52.911 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:52.908+0000 7f1ce32f8700 1 -- 172.21.15.86:0/1616456164 shutdown_connections 2024-02-19T22:05:52.911 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:52.908+0000 7f1ce32f8700 1 -- 172.21.15.86:0/1616456164 wait complete. 2024-02-19T22:05:52.911 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:52.908+0000 7f1ce32f8700 1 Processor -- start 2024-02-19T22:05:52.911 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:52.908+0000 7f1ce32f8700 1 -- start start 2024-02-19T22:05:52.911 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:52.908+0000 7f1ce32f8700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1cdc0f5410 0x7f1cdc1056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:05:52.911 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:52.908+0000 7f1ce32f8700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cdc105bd0 con 0x7f1cdc0f5410 2024-02-19T22:05:52.911 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:52.908+0000 7f1ce1094700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1cdc0f5410 0x7f1cdc1056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:05:52.911 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:52.908+0000 7f1ce1094700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1cdc0f5410 0x7f1cdc1056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:36814/0 (socket says 172.21.15.86:36814) 2024-02-19T22:05:52.911 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:52.908+0000 7f1ce1094700 1 -- 172.21.15.86:0/2318714425 learned_addr learned my addr 172.21.15.86:0/2318714425 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T22:05:52.911 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:52.909+0000 7f1ce1094700 1 -- 172.21.15.86:0/2318714425 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ccc005040 con 0x7f1cdc0f5410 2024-02-19T22:05:52.911 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:52.909+0000 7f1ce1094700 1 --2- 172.21.15.86:0/2318714425 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1cdc0f5410 0x7f1cdc1056e0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 rx=0x7f1ccc005010 tx=0x7f1ccc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:05:52.911 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:52.909+0000 7f1cd1ffb700 1 -- 172.21.15.86:0/2318714425 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1ccc009320 con 0x7f1cdc0f5410 2024-02-19T22:05:52.911 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:52.909+0000 7f1ce32f8700 1 -- 172.21.15.86:0/2318714425 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cdc105dd0 con 0x7f1cdc0f5410 2024-02-19T22:05:52.911 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:52.909+0000 7f1cd1ffb700 1 -- 172.21.15.86:0/2318714425 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1ccc013380 con 0x7f1cdc0f5410 2024-02-19T22:05:52.912 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:52.909+0000 7f1cd1ffb700 1 -- 172.21.15.86:0/2318714425 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1ccc014e50 con 0x7f1cdc0f5410 2024-02-19T22:05:52.912 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:52.909+0000 7f1ce32f8700 1 -- 172.21.15.86:0/2318714425 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cdc106260 con 0x7f1cdc0f5410 2024-02-19T22:05:52.912 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:52.910+0000 7f1cd1ffb700 1 -- 172.21.15.86:0/2318714425 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1ccc0095f0 con 0x7f1cdc0f5410 2024-02-19T22:05:52.912 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:52.910+0000 7f1ce32f8700 1 -- 172.21.15.86:0/2318714425 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1cdc045ad0 con 0x7f1cdc0f5410 2024-02-19T22:05:52.912 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:52.910+0000 7f1cd1ffb700 1 --2- 172.21.15.86:0/2318714425 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f1cc802ff10 0x7f1cc80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:05:52.912 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:52.910+0000 7f1cd1ffb700 1 -- 172.21.15.86:0/2318714425 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1ccc040f70 con 0x7f1cdc0f5410 2024-02-19T22:05:52.912 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:52.910+0000 7f1ce0893700 1 --2- 172.21.15.86:0/2318714425 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f1cc802ff10 0x7f1cc80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:05:52.912 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:52.911+0000 7f1ce0893700 1 --2- 172.21.15.86:0/2318714425 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f1cc802ff10 0x7f1cc80323d0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7f1cd80096a0 tx=0x7f1cd8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:05:52.915 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:52.915+0000 7f1cd1ffb700 1 -- 172.21.15.86:0/2318714425 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1ccc014420 con 0x7f1cdc0f5410 2024-02-19T22:05:53.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:05:52 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:05:51.522787+0000 mgr.smithi053.fyhcdi (mgr.14162) 286 : cluster [DBG] pgmap v272: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:05:53.117 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:53.117+0000 7f1ce32f8700 1 -- 172.21.15.86:0/2318714425 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1cdc107110 con 0x7f1cdc0f5410 2024-02-19T22:05:53.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:53.119+0000 7f1cd1ffb700 1 -- 172.21.15.86:0/2318714425 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1ccc01d980 con 0x7f1cdc0f5410 2024-02-19T22:05:53.118 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T22:05:53.119 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:05:53.119 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:53.120+0000 7f1ce32f8700 1 -- 172.21.15.86:0/2318714425 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f1cc802ff10 msgr2=0x7f1cc80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:05:53.120 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:53.120+0000 7f1ce32f8700 1 --2- 172.21.15.86:0/2318714425 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f1cc802ff10 0x7f1cc80323d0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7f1cd80096a0 tx=0x7f1cd8006b40).stop 2024-02-19T22:05:53.120 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:53.120+0000 7f1ce32f8700 1 -- 172.21.15.86:0/2318714425 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1cdc0f5410 msgr2=0x7f1cdc1056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:05:53.120 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:53.120+0000 7f1ce32f8700 1 --2- 172.21.15.86:0/2318714425 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1cdc0f5410 0x7f1cdc1056e0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 rx=0x7f1ccc005010 tx=0x7f1ccc00b040).stop 2024-02-19T22:05:53.120 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:53.120+0000 7f1ce32f8700 1 -- 172.21.15.86:0/2318714425 shutdown_connections 2024-02-19T22:05:53.120 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:53.120+0000 7f1ce32f8700 1 --2- 172.21.15.86:0/2318714425 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f1cc802ff10 0x7f1cc80323d0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:05:53.120 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:53.120+0000 7f1ce32f8700 1 --2- 172.21.15.86:0/2318714425 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1cdc0f5410 0x7f1cdc1056e0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:05:53.120 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:53.120+0000 7f1ce32f8700 1 -- 172.21.15.86:0/2318714425 >> 172.21.15.86:0/2318714425 conn(0x7f1cdc0f0b70 msgr2=0x7f1cdc0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:05:53.120 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:53.120+0000 7f1ce32f8700 1 -- 172.21.15.86:0/2318714425 shutdown_connections 2024-02-19T22:05:53.120 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:53.120+0000 7f1ce32f8700 1 -- 172.21.15.86:0/2318714425 wait complete. 2024-02-19T22:05:53.121 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T22:05:54.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:05:53 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:05:53.119310+0000 mon.smithi053 (mon.0) 281 : audit [DBG] from='client.? 172.21.15.86:0/2318714425' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:05:54.651 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:05:54.652 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:05:55.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:05:54 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:05:53.523143+0000 mgr.smithi053.fyhcdi (mgr.14162) 287 : cluster [DBG] pgmap v273: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:05:56.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:56.516+0000 7fa127a83700 1 -- 172.21.15.86:0/1936432181 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa1200f5410 msgr2=0x7fa1200f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:05:56.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:56.516+0000 7fa127a83700 1 --2- 172.21.15.86:0/1936432181 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa1200f5410 0x7fa1200f5830 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 rx=0x7fa110009fb0 tx=0x7fa110009b30).stop 2024-02-19T22:05:56.547 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:56.516+0000 7fa127a83700 1 -- 172.21.15.86:0/1936432181 shutdown_connections 2024-02-19T22:05:56.547 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:56.517+0000 7fa127a83700 1 --2- 172.21.15.86:0/1936432181 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa1200f5410 0x7fa1200f5830 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:05:56.547 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:56.517+0000 7fa127a83700 1 -- 172.21.15.86:0/1936432181 >> 172.21.15.86:0/1936432181 conn(0x7fa1200f0b70 msgr2=0x7fa1200f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:05:56.547 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:56.517+0000 7fa127a83700 1 -- 172.21.15.86:0/1936432181 shutdown_connections 2024-02-19T22:05:56.547 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:56.517+0000 7fa127a83700 1 -- 172.21.15.86:0/1936432181 wait complete. 2024-02-19T22:05:56.547 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:56.517+0000 7fa127a83700 1 Processor -- start 2024-02-19T22:05:56.547 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:56.517+0000 7fa127a83700 1 -- start start 2024-02-19T22:05:56.547 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:56.517+0000 7fa127a83700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa1200f5410 0x7fa1201094e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:05:56.547 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:56.517+0000 7fa127a83700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa120060240 con 0x7fa1200f5410 2024-02-19T22:05:56.547 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:56.517+0000 7fa12581f700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa1200f5410 0x7fa1201094e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:05:56.547 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:56.517+0000 7fa12581f700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa1200f5410 0x7fa1201094e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:36836/0 (socket says 172.21.15.86:36836) 2024-02-19T22:05:56.547 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:56.517+0000 7fa12581f700 1 -- 172.21.15.86:0/3745423320 learned_addr learned my addr 172.21.15.86:0/3745423320 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T22:05:56.547 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:56.518+0000 7fa12581f700 1 -- 172.21.15.86:0/3745423320 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa110005040 con 0x7fa1200f5410 2024-02-19T22:05:56.547 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:56.518+0000 7fa12581f700 1 --2- 172.21.15.86:0/3745423320 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa1200f5410 0x7fa1201094e0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0x7fa110005010 tx=0x7fa11000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:05:56.548 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:56.518+0000 7fa1167fc700 1 -- 172.21.15.86:0/3745423320 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa110009320 con 0x7fa1200f5410 2024-02-19T22:05:56.548 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:56.518+0000 7fa127a83700 1 -- 172.21.15.86:0/3745423320 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa120060440 con 0x7fa1200f5410 2024-02-19T22:05:56.548 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:56.518+0000 7fa1167fc700 1 -- 172.21.15.86:0/3745423320 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa110013380 con 0x7fa1200f5410 2024-02-19T22:05:56.548 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:56.518+0000 7fa1167fc700 1 -- 172.21.15.86:0/3745423320 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa110014e50 con 0x7fa1200f5410 2024-02-19T22:05:56.548 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:56.518+0000 7fa127a83700 1 -- 172.21.15.86:0/3745423320 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa120109b90 con 0x7fa1200f5410 2024-02-19T22:05:56.548 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:56.519+0000 7fa127a83700 1 -- 172.21.15.86:0/3745423320 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa120045ad0 con 0x7fa1200f5410 2024-02-19T22:05:56.548 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:56.519+0000 7fa1167fc700 1 -- 172.21.15.86:0/3745423320 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa1100095f0 con 0x7fa1200f5410 2024-02-19T22:05:56.548 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:56.519+0000 7fa1167fc700 1 --2- 172.21.15.86:0/3745423320 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fa10c02ff10 0x7fa10c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:05:56.548 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:56.519+0000 7fa1167fc700 1 -- 172.21.15.86:0/3745423320 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa110040e90 con 0x7fa1200f5410 2024-02-19T22:05:56.548 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:56.519+0000 7fa12501e700 1 --2- 172.21.15.86:0/3745423320 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fa10c02ff10 0x7fa10c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:05:56.548 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:56.520+0000 7fa12501e700 1 --2- 172.21.15.86:0/3745423320 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fa10c02ff10 0x7fa10c0323d0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7fa11c0096a0 tx=0x7fa11c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:05:56.548 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:56.524+0000 7fa1167fc700 1 -- 172.21.15.86:0/3745423320 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa110014420 con 0x7fa1200f5410 2024-02-19T22:05:56.729 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:56.729+0000 7fa127a83700 1 -- 172.21.15.86:0/3745423320 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa12010a3b0 con 0x7fa1200f5410 2024-02-19T22:05:56.729 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:56.730+0000 7fa1167fc700 1 -- 172.21.15.86:0/3745423320 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa110010230 con 0x7fa1200f5410 2024-02-19T22:05:56.730 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T22:05:56.730 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:05:56.732 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:56.731+0000 7fa127a83700 1 -- 172.21.15.86:0/3745423320 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fa10c02ff10 msgr2=0x7fa10c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:05:56.732 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:56.731+0000 7fa127a83700 1 --2- 172.21.15.86:0/3745423320 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fa10c02ff10 0x7fa10c0323d0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7fa11c0096a0 tx=0x7fa11c006b40).stop 2024-02-19T22:05:56.732 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:56.732+0000 7fa127a83700 1 -- 172.21.15.86:0/3745423320 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa1200f5410 msgr2=0x7fa1201094e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:05:56.732 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:56.732+0000 7fa127a83700 1 --2- 172.21.15.86:0/3745423320 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa1200f5410 0x7fa1201094e0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0x7fa110005010 tx=0x7fa11000b040).stop 2024-02-19T22:05:56.732 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:56.732+0000 7fa127a83700 1 -- 172.21.15.86:0/3745423320 shutdown_connections 2024-02-19T22:05:56.732 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:56.732+0000 7fa127a83700 1 --2- 172.21.15.86:0/3745423320 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fa10c02ff10 0x7fa10c0323d0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:05:56.733 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:56.732+0000 7fa127a83700 1 --2- 172.21.15.86:0/3745423320 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa1200f5410 0x7fa1201094e0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:05:56.733 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:56.732+0000 7fa127a83700 1 -- 172.21.15.86:0/3745423320 >> 172.21.15.86:0/3745423320 conn(0x7fa1200f0b70 msgr2=0x7fa1200f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:05:56.733 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:56.732+0000 7fa127a83700 1 -- 172.21.15.86:0/3745423320 shutdown_connections 2024-02-19T22:05:56.733 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:05:56.732+0000 7fa127a83700 1 -- 172.21.15.86:0/3745423320 wait complete. 2024-02-19T22:05:56.733 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T22:05:57.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:05:56 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:05:55.523450+0000 mgr.smithi053.fyhcdi (mgr.14162) 288 : cluster [DBG] pgmap v274: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:05:58.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:05:57 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:05:56.730789+0000 mon.smithi053 (mon.0) 282 : audit [DBG] from='client.? 172.21.15.86:0/3745423320' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:05:58.452 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:05:58.452 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:05:59.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:05:58 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:05:57.523808+0000 mgr.smithi053.fyhcdi (mgr.14162) 289 : cluster [DBG] pgmap v275: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:06:00.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:00.424+0000 7f74f69a1700 1 -- 172.21.15.86:0/76902401 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f74f00f51a0 msgr2=0x7f74f00f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:06:00.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:00.424+0000 7f74f69a1700 1 --2- 172.21.15.86:0/76902401 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f74f00f51a0 0x7f74f00f55c0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 rx=0x7f74e0009fb0 tx=0x7f74e0009b30).stop 2024-02-19T22:06:00.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:00.424+0000 7f74f69a1700 1 -- 172.21.15.86:0/76902401 shutdown_connections 2024-02-19T22:06:00.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:00.424+0000 7f74f69a1700 1 --2- 172.21.15.86:0/76902401 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f74f00f51a0 0x7f74f00f55c0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:06:00.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:00.424+0000 7f74f69a1700 1 -- 172.21.15.86:0/76902401 >> 172.21.15.86:0/76902401 conn(0x7f74f00f09a0 msgr2=0x7f74f00f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:06:00.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:00.424+0000 7f74f69a1700 1 -- 172.21.15.86:0/76902401 shutdown_connections 2024-02-19T22:06:00.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:00.424+0000 7f74f69a1700 1 -- 172.21.15.86:0/76902401 wait complete. 2024-02-19T22:06:00.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:00.425+0000 7f74f69a1700 1 Processor -- start 2024-02-19T22:06:00.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:00.425+0000 7f74f69a1700 1 -- start start 2024-02-19T22:06:00.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:00.425+0000 7f74f69a1700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f74f00f51a0 0x7f74f00f2f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:06:00.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:00.425+0000 7f74f69a1700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74f00f13a0 con 0x7f74f00f51a0 2024-02-19T22:06:00.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:00.425+0000 7f74f599f700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f74f00f51a0 0x7f74f00f2f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:06:00.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:00.425+0000 7f74f599f700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f74f00f51a0 0x7f74f00f2f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:45276/0 (socket says 172.21.15.86:45276) 2024-02-19T22:06:00.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:00.425+0000 7f74f599f700 1 -- 172.21.15.86:0/1947128008 learned_addr learned my addr 172.21.15.86:0/1947128008 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T22:06:00.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:00.425+0000 7f74f599f700 1 -- 172.21.15.86:0/1947128008 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74e0005040 con 0x7f74f00f51a0 2024-02-19T22:06:00.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:00.425+0000 7f74f599f700 1 --2- 172.21.15.86:0/1947128008 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f74f00f51a0 0x7f74f00f2f20 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0x7f74e00089f0 tx=0x7f74e000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:06:00.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:00.426+0000 7f74e67fc700 1 -- 172.21.15.86:0/1947128008 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f74e0009320 con 0x7f74f00f51a0 2024-02-19T22:06:00.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:00.426+0000 7f74f69a1700 1 -- 172.21.15.86:0/1947128008 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74f00f1540 con 0x7f74f00f51a0 2024-02-19T22:06:00.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:00.426+0000 7f74e67fc700 1 -- 172.21.15.86:0/1947128008 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f74e0013380 con 0x7f74f00f51a0 2024-02-19T22:06:00.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:00.426+0000 7f74e67fc700 1 -- 172.21.15.86:0/1947128008 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f74e0014e20 con 0x7f74f00f51a0 2024-02-19T22:06:00.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:00.426+0000 7f74f69a1700 1 -- 172.21.15.86:0/1947128008 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74f00f19d0 con 0x7f74f00f51a0 2024-02-19T22:06:00.428 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:00.426+0000 7f74e67fc700 1 -- 172.21.15.86:0/1947128008 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f74e00095f0 con 0x7f74f00f51a0 2024-02-19T22:06:00.428 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:00.427+0000 7f74e67fc700 1 --2- 172.21.15.86:0/1947128008 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f74dc02ff10 0x7f74dc0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:06:00.428 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:00.427+0000 7f74f69a1700 1 -- 172.21.15.86:0/1947128008 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74f0059100 con 0x7f74f00f51a0 2024-02-19T22:06:00.428 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:00.427+0000 7f74e67fc700 1 -- 172.21.15.86:0/1947128008 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f74e0040f50 con 0x7f74f00f51a0 2024-02-19T22:06:00.428 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:00.427+0000 7f74f519e700 1 --2- 172.21.15.86:0/1947128008 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f74dc02ff10 0x7f74dc0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:06:00.429 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:00.427+0000 7f74f519e700 1 --2- 172.21.15.86:0/1947128008 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f74dc02ff10 0x7f74dc0323d0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7f74ec0096a0 tx=0x7f74ec006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:06:00.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:00.431+0000 7f74e67fc700 1 -- 172.21.15.86:0/1947128008 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f74e0014420 con 0x7f74f00f51a0 2024-02-19T22:06:00.636 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:00.636+0000 7f74f69a1700 1 -- 172.21.15.86:0/1947128008 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f74f0045ad0 con 0x7f74f00f51a0 2024-02-19T22:06:00.637 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:00.637+0000 7f74e67fc700 1 -- 172.21.15.86:0/1947128008 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f74e001d980 con 0x7f74f00f51a0 2024-02-19T22:06:00.637 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T22:06:00.637 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:06:00.638 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:00.638+0000 7f74f69a1700 1 -- 172.21.15.86:0/1947128008 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f74dc02ff10 msgr2=0x7f74dc0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:06:00.639 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:00.638+0000 7f74f69a1700 1 --2- 172.21.15.86:0/1947128008 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f74dc02ff10 0x7f74dc0323d0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7f74ec0096a0 tx=0x7f74ec006b40).stop 2024-02-19T22:06:00.639 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:00.638+0000 7f74f69a1700 1 -- 172.21.15.86:0/1947128008 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f74f00f51a0 msgr2=0x7f74f00f2f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:06:00.639 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:00.638+0000 7f74f69a1700 1 --2- 172.21.15.86:0/1947128008 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f74f00f51a0 0x7f74f00f2f20 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0x7f74e00089f0 tx=0x7f74e000b040).stop 2024-02-19T22:06:00.639 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:00.639+0000 7f74f69a1700 1 -- 172.21.15.86:0/1947128008 shutdown_connections 2024-02-19T22:06:00.639 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:00.639+0000 7f74f69a1700 1 --2- 172.21.15.86:0/1947128008 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f74dc02ff10 0x7f74dc0323d0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:06:00.639 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:00.639+0000 7f74f69a1700 1 --2- 172.21.15.86:0/1947128008 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f74f00f51a0 0x7f74f00f2f20 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:06:00.639 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:00.639+0000 7f74f69a1700 1 -- 172.21.15.86:0/1947128008 >> 172.21.15.86:0/1947128008 conn(0x7f74f00f09a0 msgr2=0x7f74f005fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:06:00.639 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:00.639+0000 7f74f69a1700 1 -- 172.21.15.86:0/1947128008 shutdown_connections 2024-02-19T22:06:00.639 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:00.639+0000 7f74f69a1700 1 -- 172.21.15.86:0/1947128008 wait complete. 2024-02-19T22:06:00.639 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T22:06:00.794 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:06:00 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:05:59.524167+0000 mgr.smithi053.fyhcdi (mgr.14162) 290 : cluster [DBG] pgmap v276: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:06:01.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:06:01 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:06:00.638297+0000 mon.smithi053 (mon.0) 283 : audit [DBG] from='client.? 172.21.15.86:0/1947128008' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:06:02.345 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:06:02.346 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:06:02.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:06:02 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:06:01.524399+0000 mgr.smithi053.fyhcdi (mgr.14162) 291 : cluster [DBG] pgmap v277: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:06:04.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:04.144+0000 7f0405570700 1 -- 172.21.15.86:0/3933627609 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f04000f51a0 msgr2=0x7f04000f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:06:04.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:04.144+0000 7f0405570700 1 --2- 172.21.15.86:0/3933627609 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f04000f51a0 0x7f04000f55c0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 rx=0x7f03f0009fb0 tx=0x7f03f0009b30).stop 2024-02-19T22:06:04.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:04.144+0000 7f0405570700 1 -- 172.21.15.86:0/3933627609 shutdown_connections 2024-02-19T22:06:04.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:04.144+0000 7f0405570700 1 --2- 172.21.15.86:0/3933627609 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f04000f51a0 0x7f04000f55c0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:06:04.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:04.144+0000 7f0405570700 1 -- 172.21.15.86:0/3933627609 >> 172.21.15.86:0/3933627609 conn(0x7f04000f09a0 msgr2=0x7f04000f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:06:04.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:04.144+0000 7f0405570700 1 -- 172.21.15.86:0/3933627609 shutdown_connections 2024-02-19T22:06:04.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:04.144+0000 7f0405570700 1 -- 172.21.15.86:0/3933627609 wait complete. 2024-02-19T22:06:04.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:04.144+0000 7f0405570700 1 Processor -- start 2024-02-19T22:06:04.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:04.144+0000 7f0405570700 1 -- start start 2024-02-19T22:06:04.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:04.144+0000 7f0405570700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f04000f51a0 0x7f04001092f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:06:04.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:04.144+0000 7f0405570700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0400060220 con 0x7f04000f51a0 2024-02-19T22:06:04.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:04.145+0000 7f03fffff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f04000f51a0 0x7f04001092f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:06:04.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:04.145+0000 7f03fffff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f04000f51a0 0x7f04001092f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:45298/0 (socket says 172.21.15.86:45298) 2024-02-19T22:06:04.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:04.145+0000 7f03fffff700 1 -- 172.21.15.86:0/1249974308 learned_addr learned my addr 172.21.15.86:0/1249974308 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T22:06:04.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:04.145+0000 7f03fffff700 1 -- 172.21.15.86:0/1249974308 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03f0005040 con 0x7f04000f51a0 2024-02-19T22:06:04.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:04.145+0000 7f03fffff700 1 --2- 172.21.15.86:0/1249974308 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f04000f51a0 0x7f04001092f0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7f03f00089f0 tx=0x7f03f000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:06:04.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:04.145+0000 7f03fcff9700 1 -- 172.21.15.86:0/1249974308 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f03f0009320 con 0x7f04000f51a0 2024-02-19T22:06:04.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:04.145+0000 7f0405570700 1 -- 172.21.15.86:0/1249974308 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0400060420 con 0x7f04000f51a0 2024-02-19T22:06:04.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:04.145+0000 7f03fcff9700 1 -- 172.21.15.86:0/1249974308 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f03f0013380 con 0x7f04000f51a0 2024-02-19T22:06:04.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:04.145+0000 7f03fcff9700 1 -- 172.21.15.86:0/1249974308 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f03f0014e20 con 0x7f04000f51a0 2024-02-19T22:06:04.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:04.145+0000 7f0405570700 1 -- 172.21.15.86:0/1249974308 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04001099a0 con 0x7f04000f51a0 2024-02-19T22:06:04.147 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:04.146+0000 7f03fcff9700 1 -- 172.21.15.86:0/1249974308 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f03f00095f0 con 0x7f04000f51a0 2024-02-19T22:06:04.147 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:04.146+0000 7f0405570700 1 -- 172.21.15.86:0/1249974308 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0400059100 con 0x7f04000f51a0 2024-02-19T22:06:04.147 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:04.146+0000 7f03fcff9700 1 --2- 172.21.15.86:0/1249974308 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f03e802ff10 0x7f03e80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:06:04.148 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:04.146+0000 7f03fcff9700 1 -- 172.21.15.86:0/1249974308 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f03f0040ef0 con 0x7f04000f51a0 2024-02-19T22:06:04.148 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:04.147+0000 7f03ff7fe700 1 --2- 172.21.15.86:0/1249974308 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f03e802ff10 0x7f03e80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:06:04.148 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:04.147+0000 7f03ff7fe700 1 --2- 172.21.15.86:0/1249974308 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f03e802ff10 0x7f03e80323d0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f03f40096a0 tx=0x7f03f4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:06:04.151 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:04.151+0000 7f03fcff9700 1 -- 172.21.15.86:0/1249974308 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f03f0014420 con 0x7f04000f51a0 2024-02-19T22:06:04.354 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:04.354+0000 7f0405570700 1 -- 172.21.15.86:0/1249974308 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f04000fad30 con 0x7f04000f51a0 2024-02-19T22:06:04.355 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:04.355+0000 7f03fcff9700 1 -- 172.21.15.86:0/1249974308 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f03f0010230 con 0x7f04000f51a0 2024-02-19T22:06:04.355 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T22:06:04.355 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:06:04.356 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:04.356+0000 7f0405570700 1 -- 172.21.15.86:0/1249974308 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f03e802ff10 msgr2=0x7f03e80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:06:04.356 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:04.356+0000 7f0405570700 1 --2- 172.21.15.86:0/1249974308 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f03e802ff10 0x7f03e80323d0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f03f40096a0 tx=0x7f03f4006b40).stop 2024-02-19T22:06:04.357 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:04.356+0000 7f0405570700 1 -- 172.21.15.86:0/1249974308 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f04000f51a0 msgr2=0x7f04001092f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:06:04.357 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:04.356+0000 7f0405570700 1 --2- 172.21.15.86:0/1249974308 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f04000f51a0 0x7f04001092f0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7f03f00089f0 tx=0x7f03f000b040).stop 2024-02-19T22:06:04.357 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:04.356+0000 7f0405570700 1 -- 172.21.15.86:0/1249974308 shutdown_connections 2024-02-19T22:06:04.357 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:04.356+0000 7f0405570700 1 --2- 172.21.15.86:0/1249974308 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f03e802ff10 0x7f03e80323d0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:06:04.357 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:04.356+0000 7f0405570700 1 --2- 172.21.15.86:0/1249974308 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f04000f51a0 0x7f04001092f0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:06:04.357 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:04.356+0000 7f0405570700 1 -- 172.21.15.86:0/1249974308 >> 172.21.15.86:0/1249974308 conn(0x7f04000f09a0 msgr2=0x7f04000f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:06:04.357 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:04.357+0000 7f0405570700 1 -- 172.21.15.86:0/1249974308 shutdown_connections 2024-02-19T22:06:04.357 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:04.357+0000 7f0405570700 1 -- 172.21.15.86:0/1249974308 wait complete. 2024-02-19T22:06:04.357 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T22:06:05.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:06:04 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:06:03.524714+0000 mgr.smithi053.fyhcdi (mgr.14162) 292 : cluster [DBG] pgmap v278: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:06:05.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:06:04 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:06:04.356392+0000 mon.smithi053 (mon.0) 284 : audit [DBG] from='client.? 172.21.15.86:0/1249974308' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:06:06.199 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:06:06.200 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:06:07.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:06:06 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:06:05.524998+0000 mgr.smithi053.fyhcdi (mgr.14162) 293 : cluster [DBG] pgmap v279: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:06:08.079 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:08.077+0000 7f67206dd700 1 -- 172.21.15.86:0/2413031650 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f67180f5410 msgr2=0x7f67180f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:06:08.079 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:08.077+0000 7f67206dd700 1 --2- 172.21.15.86:0/2413031650 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f67180f5410 0x7f67180f5830 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 rx=0x7f6708009960 tx=0x7f6708004d10).stop 2024-02-19T22:06:08.079 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:08.077+0000 7f67206dd700 1 -- 172.21.15.86:0/2413031650 shutdown_connections 2024-02-19T22:06:08.079 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:08.077+0000 7f67206dd700 1 --2- 172.21.15.86:0/2413031650 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f67180f5410 0x7f67180f5830 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:06:08.079 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:08.077+0000 7f67206dd700 1 -- 172.21.15.86:0/2413031650 >> 172.21.15.86:0/2413031650 conn(0x7f67180f0b70 msgr2=0x7f67180f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:06:08.079 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:08.077+0000 7f67206dd700 1 -- 172.21.15.86:0/2413031650 shutdown_connections 2024-02-19T22:06:08.079 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:08.077+0000 7f67206dd700 1 -- 172.21.15.86:0/2413031650 wait complete. 2024-02-19T22:06:08.079 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:08.078+0000 7f67206dd700 1 Processor -- start 2024-02-19T22:06:08.079 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:08.078+0000 7f67206dd700 1 -- start start 2024-02-19T22:06:08.079 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:08.078+0000 7f67206dd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f67180f5410 0x7f6718105720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:06:08.079 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:08.078+0000 7f67206dd700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6718105c10 con 0x7f67180f5410 2024-02-19T22:06:08.080 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:08.078+0000 7f671e479700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f67180f5410 0x7f6718105720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:06:08.080 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:08.078+0000 7f671e479700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f67180f5410 0x7f6718105720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:45320/0 (socket says 172.21.15.86:45320) 2024-02-19T22:06:08.080 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:08.078+0000 7f671e479700 1 -- 172.21.15.86:0/2262168097 learned_addr learned my addr 172.21.15.86:0/2262168097 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T22:06:08.080 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:08.078+0000 7f671e479700 1 -- 172.21.15.86:0/2262168097 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6708009640 con 0x7f67180f5410 2024-02-19T22:06:08.080 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:08.078+0000 7f671e479700 1 --2- 172.21.15.86:0/2262168097 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f67180f5410 0x7f6718105720 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0x7f6708004a10 tx=0x7f6708004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:06:08.080 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:08.079+0000 7f6716ffd700 1 -- 172.21.15.86:0/2262168097 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6708018070 con 0x7f67180f5410 2024-02-19T22:06:08.080 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:08.079+0000 7f6716ffd700 1 -- 172.21.15.86:0/2262168097 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6708007e70 con 0x7f67180f5410 2024-02-19T22:06:08.080 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:08.079+0000 7f67206dd700 1 -- 172.21.15.86:0/2262168097 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6718105e10 con 0x7f67180f5410 2024-02-19T22:06:08.080 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:08.079+0000 7f6716ffd700 1 -- 172.21.15.86:0/2262168097 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f670801d420 con 0x7f67180f5410 2024-02-19T22:06:08.080 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:08.079+0000 7f67206dd700 1 -- 172.21.15.86:0/2262168097 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67181062a0 con 0x7f67180f5410 2024-02-19T22:06:08.081 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:08.079+0000 7f6716ffd700 1 -- 172.21.15.86:0/2262168097 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f670801d6e0 con 0x7f67180f5410 2024-02-19T22:06:08.081 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:08.080+0000 7f6716ffd700 1 --2- 172.21.15.86:0/2262168097 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f670002ff60 0x7f6700032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:06:08.081 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:08.080+0000 7f6716ffd700 1 -- 172.21.15.86:0/2262168097 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6708041680 con 0x7f67180f5410 2024-02-19T22:06:08.081 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:08.080+0000 7f67206dd700 1 -- 172.21.15.86:0/2262168097 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6718045ad0 con 0x7f67180f5410 2024-02-19T22:06:08.081 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:08.080+0000 7f671dc78700 1 --2- 172.21.15.86:0/2262168097 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f670002ff60 0x7f6700032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:06:08.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:08.080+0000 7f671dc78700 1 --2- 172.21.15.86:0/2262168097 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f670002ff60 0x7f6700032420 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f670c0096a0 tx=0x7f670c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:06:08.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:08.085+0000 7f6716ffd700 1 -- 172.21.15.86:0/2262168097 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f670801dae0 con 0x7f67180f5410 2024-02-19T22:06:08.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:08.289+0000 7f67206dd700 1 -- 172.21.15.86:0/2262168097 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6718106f20 con 0x7f67180f5410 2024-02-19T22:06:08.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:08.289+0000 7f6716ffd700 1 -- 172.21.15.86:0/2262168097 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6708011320 con 0x7f67180f5410 2024-02-19T22:06:08.290 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T22:06:08.290 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:06:08.291 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:08.291+0000 7f67206dd700 1 -- 172.21.15.86:0/2262168097 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f670002ff60 msgr2=0x7f6700032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:06:08.291 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:08.291+0000 7f67206dd700 1 --2- 172.21.15.86:0/2262168097 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f670002ff60 0x7f6700032420 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f670c0096a0 tx=0x7f670c006b40).stop 2024-02-19T22:06:08.291 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:08.291+0000 7f67206dd700 1 -- 172.21.15.86:0/2262168097 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f67180f5410 msgr2=0x7f6718105720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:06:08.291 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:08.291+0000 7f67206dd700 1 --2- 172.21.15.86:0/2262168097 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f67180f5410 0x7f6718105720 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0x7f6708004a10 tx=0x7f6708004d10).stop 2024-02-19T22:06:08.291 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:08.291+0000 7f67206dd700 1 -- 172.21.15.86:0/2262168097 shutdown_connections 2024-02-19T22:06:08.292 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:08.291+0000 7f67206dd700 1 --2- 172.21.15.86:0/2262168097 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f670002ff60 0x7f6700032420 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:06:08.292 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:08.291+0000 7f67206dd700 1 --2- 172.21.15.86:0/2262168097 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f67180f5410 0x7f6718105720 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:06:08.292 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:08.291+0000 7f67206dd700 1 -- 172.21.15.86:0/2262168097 >> 172.21.15.86:0/2262168097 conn(0x7f67180f0b70 msgr2=0x7f67180f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:06:08.292 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:08.291+0000 7f67206dd700 1 -- 172.21.15.86:0/2262168097 shutdown_connections 2024-02-19T22:06:08.292 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:08.291+0000 7f67206dd700 1 -- 172.21.15.86:0/2262168097 wait complete. 2024-02-19T22:06:08.292 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T22:06:09.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:06:08 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:06:07.525372+0000 mgr.smithi053.fyhcdi (mgr.14162) 294 : cluster [DBG] pgmap v280: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:06:09.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:06:08 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:06:08.290621+0000 mon.smithi053 (mon.0) 285 : audit [DBG] from='client.? 172.21.15.86:0/2262168097' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:06:09.876 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:06:09.877 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:06:10.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:06:10 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:06:09.525763+0000 mgr.smithi053.fyhcdi (mgr.14162) 295 : cluster [DBG] pgmap v281: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:06:11.906 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:11.906+0000 7f52a659d700 1 -- 172.21.15.86:0/2886641081 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f52a0060180 msgr2=0x7f52a00f8f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:06:11.906 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:11.906+0000 7f52a659d700 1 --2- 172.21.15.86:0/2886641081 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f52a0060180 0x7f52a00f8f80 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 rx=0x7f5290009fb0 tx=0x7f5290009b30).stop 2024-02-19T22:06:11.906 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:11.906+0000 7f52a659d700 1 -- 172.21.15.86:0/2886641081 shutdown_connections 2024-02-19T22:06:11.906 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:11.906+0000 7f52a659d700 1 --2- 172.21.15.86:0/2886641081 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f52a0060180 0x7f52a00f8f80 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:06:11.906 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:11.906+0000 7f52a659d700 1 -- 172.21.15.86:0/2886641081 >> 172.21.15.86:0/2886641081 conn(0x7f52a00f09a0 msgr2=0x7f52a00f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:06:11.906 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:11.906+0000 7f52a659d700 1 -- 172.21.15.86:0/2886641081 shutdown_connections 2024-02-19T22:06:11.906 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:11.906+0000 7f52a659d700 1 -- 172.21.15.86:0/2886641081 wait complete. 2024-02-19T22:06:11.906 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:11.907+0000 7f52a659d700 1 Processor -- start 2024-02-19T22:06:11.906 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:11.907+0000 7f52a659d700 1 -- start start 2024-02-19T22:06:11.907 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:11.907+0000 7f52a659d700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f52a01066e0 0x7f52a0106b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:06:11.907 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:11.907+0000 7f52a659d700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52a0106ff0 con 0x7f52a01066e0 2024-02-19T22:06:11.908 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:11.907+0000 7f52a559b700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f52a01066e0 0x7f52a0106b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:06:11.908 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:11.907+0000 7f52a559b700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f52a01066e0 0x7f52a0106b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:55562/0 (socket says 172.21.15.86:55562) 2024-02-19T22:06:11.908 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:11.907+0000 7f52a559b700 1 -- 172.21.15.86:0/3859404849 learned_addr learned my addr 172.21.15.86:0/3859404849 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T22:06:11.908 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:11.908+0000 7f52a559b700 1 -- 172.21.15.86:0/3859404849 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5290005040 con 0x7f52a01066e0 2024-02-19T22:06:11.908 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:11.908+0000 7f52a559b700 1 --2- 172.21.15.86:0/3859404849 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f52a01066e0 0x7f52a0106b00 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7f52a005ff50 tx=0x7f5290009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:06:11.909 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:11.908+0000 7f5295ffb700 1 -- 172.21.15.86:0/3859404849 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5290013380 con 0x7f52a01066e0 2024-02-19T22:06:11.909 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:11.908+0000 7f52a659d700 1 -- 172.21.15.86:0/3859404849 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52a01071f0 con 0x7f52a01066e0 2024-02-19T22:06:11.909 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:11.908+0000 7f5295ffb700 1 -- 172.21.15.86:0/3859404849 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5290013790 con 0x7f52a01066e0 2024-02-19T22:06:11.909 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:11.908+0000 7f5295ffb700 1 -- 172.21.15.86:0/3859404849 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f529001ee70 con 0x7f52a01066e0 2024-02-19T22:06:11.909 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:11.908+0000 7f52a659d700 1 -- 172.21.15.86:0/3859404849 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52a0107d40 con 0x7f52a01066e0 2024-02-19T22:06:11.909 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:11.909+0000 7f5295ffb700 1 -- 172.21.15.86:0/3859404849 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5290025060 con 0x7f52a01066e0 2024-02-19T22:06:11.909 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:11.909+0000 7f52a659d700 1 -- 172.21.15.86:0/3859404849 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52a0059100 con 0x7f52a01066e0 2024-02-19T22:06:11.909 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:11.909+0000 7f5295ffb700 1 --2- 172.21.15.86:0/3859404849 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f528c02ff60 0x7f528c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:06:11.909 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:11.909+0000 7f5295ffb700 1 -- 172.21.15.86:0/3859404849 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5290040e80 con 0x7f52a01066e0 2024-02-19T22:06:11.909 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:11.909+0000 7f52a4d9a700 1 --2- 172.21.15.86:0/3859404849 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f528c02ff60 0x7f528c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:06:11.909 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:11.910+0000 7f52a4d9a700 1 --2- 172.21.15.86:0/3859404849 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f528c02ff60 0x7f528c032420 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f529c0096a0 tx=0x7f529c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:06:11.912 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:11.914+0000 7f5295ffb700 1 -- 172.21.15.86:0/3859404849 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f529001c950 con 0x7f52a01066e0 2024-02-19T22:06:12.120 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:12.121+0000 7f52a659d700 1 -- 172.21.15.86:0/3859404849 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f52a0108500 con 0x7f52a01066e0 2024-02-19T22:06:12.120 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:12.122+0000 7f5295ffb700 1 -- 172.21.15.86:0/3859404849 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f529001cb60 con 0x7f52a01066e0 2024-02-19T22:06:12.121 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T22:06:12.121 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:06:12.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:12.123+0000 7f52a659d700 1 -- 172.21.15.86:0/3859404849 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f528c02ff60 msgr2=0x7f528c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:06:12.123 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:12.123+0000 7f52a659d700 1 --2- 172.21.15.86:0/3859404849 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f528c02ff60 0x7f528c032420 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f529c0096a0 tx=0x7f529c006b40).stop 2024-02-19T22:06:12.123 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:12.123+0000 7f52a659d700 1 -- 172.21.15.86:0/3859404849 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f52a01066e0 msgr2=0x7f52a0106b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:06:12.123 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:12.123+0000 7f52a659d700 1 --2- 172.21.15.86:0/3859404849 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f52a01066e0 0x7f52a0106b00 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7f52a005ff50 tx=0x7f5290009b30).stop 2024-02-19T22:06:12.123 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:12.123+0000 7f52a659d700 1 -- 172.21.15.86:0/3859404849 shutdown_connections 2024-02-19T22:06:12.123 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:12.123+0000 7f52a659d700 1 --2- 172.21.15.86:0/3859404849 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f528c02ff60 0x7f528c032420 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:06:12.123 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:12.123+0000 7f52a659d700 1 --2- 172.21.15.86:0/3859404849 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f52a01066e0 0x7f52a0106b00 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:06:12.123 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:12.123+0000 7f52a659d700 1 -- 172.21.15.86:0/3859404849 >> 172.21.15.86:0/3859404849 conn(0x7f52a00f09a0 msgr2=0x7f52a00f1420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:06:12.123 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:12.123+0000 7f52a659d700 1 -- 172.21.15.86:0/3859404849 shutdown_connections 2024-02-19T22:06:12.123 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:12.124+0000 7f52a659d700 1 -- 172.21.15.86:0/3859404849 wait complete. 2024-02-19T22:06:12.123 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T22:06:13.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:06:12 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:06:11.526069+0000 mgr.smithi053.fyhcdi (mgr.14162) 296 : cluster [DBG] pgmap v282: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:06:13.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:06:12 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:06:12.121951+0000 mon.smithi053 (mon.0) 286 : audit [DBG] from='client.? 172.21.15.86:0/3859404849' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:06:13.804 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:06:13.804 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:06:15.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:06:14 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:06:13.526424+0000 mgr.smithi053.fyhcdi (mgr.14162) 297 : cluster [DBG] pgmap v283: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:06:15.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:15.681+0000 7f81459a0700 1 -- 172.21.15.86:0/1793227922 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f81400f4b80 msgr2=0x7f81400f4fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:06:15.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:15.681+0000 7f81459a0700 1 --2- 172.21.15.86:0/1793227922 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f81400f4b80 0x7f81400f4fa0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 rx=0x7f8134004770 tx=0x7f813400ac40).stop 2024-02-19T22:06:15.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:15.681+0000 7f81459a0700 1 -- 172.21.15.86:0/1793227922 shutdown_connections 2024-02-19T22:06:15.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:15.681+0000 7f81459a0700 1 --2- 172.21.15.86:0/1793227922 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f81400f4b80 0x7f81400f4fa0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:06:15.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:15.681+0000 7f81459a0700 1 -- 172.21.15.86:0/1793227922 >> 172.21.15.86:0/1793227922 conn(0x7f81400f02e0 msgr2=0x7f81400f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:06:15.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:15.681+0000 7f81459a0700 1 -- 172.21.15.86:0/1793227922 shutdown_connections 2024-02-19T22:06:15.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:15.681+0000 7f81459a0700 1 -- 172.21.15.86:0/1793227922 wait complete. 2024-02-19T22:06:15.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:15.682+0000 7f81459a0700 1 Processor -- start 2024-02-19T22:06:15.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:15.682+0000 7f81459a0700 1 -- start start 2024-02-19T22:06:15.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:15.682+0000 7f81459a0700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f81400f4b80 0x7f8140104580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:06:15.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:15.682+0000 7f81459a0700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8140104a70 con 0x7f81400f4b80 2024-02-19T22:06:15.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:15.682+0000 7f813effd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f81400f4b80 0x7f8140104580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:06:15.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:15.682+0000 7f813effd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f81400f4b80 0x7f8140104580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:55584/0 (socket says 172.21.15.86:55584) 2024-02-19T22:06:15.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:15.682+0000 7f813effd700 1 -- 172.21.15.86:0/2659246142 learned_addr learned my addr 172.21.15.86:0/2659246142 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T22:06:15.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:15.682+0000 7f813effd700 1 -- 172.21.15.86:0/2659246142 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8134005040 con 0x7f81400f4b80 2024-02-19T22:06:15.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:15.683+0000 7f813effd700 1 --2- 172.21.15.86:0/2659246142 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f81400f4b80 0x7f8140104580 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0x7f813400aaa0 tx=0x7f813400ac40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:06:15.684 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:15.683+0000 7f81277fe700 1 -- 172.21.15.86:0/2659246142 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f81340148d0 con 0x7f81400f4b80 2024-02-19T22:06:15.684 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:15.683+0000 7f81459a0700 1 -- 172.21.15.86:0/2659246142 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8140104c70 con 0x7f81400f4b80 2024-02-19T22:06:15.684 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:15.683+0000 7f81277fe700 1 -- 172.21.15.86:0/2659246142 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8134013aa0 con 0x7f81400f4b80 2024-02-19T22:06:15.684 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:15.683+0000 7f81277fe700 1 -- 172.21.15.86:0/2659246142 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8134020b20 con 0x7f81400f4b80 2024-02-19T22:06:15.684 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:15.683+0000 7f81459a0700 1 -- 172.21.15.86:0/2659246142 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8140105100 con 0x7f81400f4b80 2024-02-19T22:06:15.684 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:15.684+0000 7f81277fe700 1 -- 172.21.15.86:0/2659246142 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8134013c10 con 0x7f81400f4b80 2024-02-19T22:06:15.684 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:15.684+0000 7f81459a0700 1 -- 172.21.15.86:0/2659246142 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8140045ad0 con 0x7f81400f4b80 2024-02-19T22:06:15.684 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:15.684+0000 7f81277fe700 1 --2- 172.21.15.86:0/2659246142 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f812802ff60 0x7f8128032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:06:15.684 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:15.684+0000 7f81277fe700 1 -- 172.21.15.86:0/2659246142 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8134043b10 con 0x7f81400f4b80 2024-02-19T22:06:15.684 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:15.684+0000 7f813e7fc700 1 --2- 172.21.15.86:0/2659246142 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f812802ff60 0x7f8128032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:06:15.684 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:15.684+0000 7f813e7fc700 1 --2- 172.21.15.86:0/2659246142 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f812802ff60 0x7f8128032420 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f81300096a0 tx=0x7f8130006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:06:15.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:15.689+0000 7f81277fe700 1 -- 172.21.15.86:0/2659246142 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f813401e9e0 con 0x7f81400f4b80 2024-02-19T22:06:15.893 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:15.894+0000 7f81459a0700 1 -- 172.21.15.86:0/2659246142 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8140105fb0 con 0x7f81400f4b80 2024-02-19T22:06:15.893 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:15.894+0000 7f81277fe700 1 -- 172.21.15.86:0/2659246142 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8134027300 con 0x7f81400f4b80 2024-02-19T22:06:15.894 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T22:06:15.894 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:06:15.895 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:15.896+0000 7f81459a0700 1 -- 172.21.15.86:0/2659246142 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f812802ff60 msgr2=0x7f8128032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:06:15.895 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:15.896+0000 7f81459a0700 1 --2- 172.21.15.86:0/2659246142 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f812802ff60 0x7f8128032420 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f81300096a0 tx=0x7f8130006b40).stop 2024-02-19T22:06:15.895 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:15.896+0000 7f81459a0700 1 -- 172.21.15.86:0/2659246142 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f81400f4b80 msgr2=0x7f8140104580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:06:15.895 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:15.896+0000 7f81459a0700 1 --2- 172.21.15.86:0/2659246142 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f81400f4b80 0x7f8140104580 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0x7f813400aaa0 tx=0x7f813400ac40).stop 2024-02-19T22:06:15.896 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:15.896+0000 7f81459a0700 1 -- 172.21.15.86:0/2659246142 shutdown_connections 2024-02-19T22:06:15.896 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:15.896+0000 7f81459a0700 1 --2- 172.21.15.86:0/2659246142 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f812802ff60 0x7f8128032420 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:06:15.896 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:15.896+0000 7f81459a0700 1 --2- 172.21.15.86:0/2659246142 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f81400f4b80 0x7f8140104580 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:06:15.896 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:15.896+0000 7f81459a0700 1 -- 172.21.15.86:0/2659246142 >> 172.21.15.86:0/2659246142 conn(0x7f81400f02e0 msgr2=0x7f81400f0d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:06:15.896 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:15.896+0000 7f81459a0700 1 -- 172.21.15.86:0/2659246142 shutdown_connections 2024-02-19T22:06:15.896 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:15.896+0000 7f81459a0700 1 -- 172.21.15.86:0/2659246142 wait complete. 2024-02-19T22:06:15.896 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T22:06:17.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:06:16 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:06:15.526718+0000 mgr.smithi053.fyhcdi (mgr.14162) 298 : cluster [DBG] pgmap v284: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:06:17.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:06:16 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:06:15.894657+0000 mon.smithi053 (mon.0) 287 : audit [DBG] from='client.? 172.21.15.86:0/2659246142' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:06:17.624 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:06:17.624 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:06:19.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:06:18 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:06:17.526977+0000 mgr.smithi053.fyhcdi (mgr.14162) 299 : cluster [DBG] pgmap v285: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:06:19.337 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:19.336+0000 7fd141dab700 1 -- 172.21.15.86:0/1906279001 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd13c0f5410 msgr2=0x7fd13c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:06:19.338 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:19.336+0000 7fd141dab700 1 --2- 172.21.15.86:0/1906279001 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd13c0f5410 0x7fd13c0f5830 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 rx=0x7fd12c009fb0 tx=0x7fd12c009b30).stop 2024-02-19T22:06:19.338 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:19.336+0000 7fd141dab700 1 -- 172.21.15.86:0/1906279001 shutdown_connections 2024-02-19T22:06:19.338 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:19.336+0000 7fd141dab700 1 --2- 172.21.15.86:0/1906279001 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd13c0f5410 0x7fd13c0f5830 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:06:19.338 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:19.336+0000 7fd141dab700 1 -- 172.21.15.86:0/1906279001 >> 172.21.15.86:0/1906279001 conn(0x7fd13c0f0b70 msgr2=0x7fd13c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:06:19.338 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:19.336+0000 7fd141dab700 1 -- 172.21.15.86:0/1906279001 shutdown_connections 2024-02-19T22:06:19.338 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:19.336+0000 7fd141dab700 1 -- 172.21.15.86:0/1906279001 wait complete. 2024-02-19T22:06:19.338 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:19.336+0000 7fd141dab700 1 Processor -- start 2024-02-19T22:06:19.338 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:19.336+0000 7fd141dab700 1 -- start start 2024-02-19T22:06:19.338 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:19.337+0000 7fd141dab700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd13c0f5410 0x7fd13c1056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:06:19.338 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:19.337+0000 7fd141dab700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd13c105bd0 con 0x7fd13c0f5410 2024-02-19T22:06:19.338 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:19.337+0000 7fd13b7fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd13c0f5410 0x7fd13c1056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:06:19.338 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:19.337+0000 7fd13b7fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd13c0f5410 0x7fd13c1056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:48936/0 (socket says 172.21.15.86:48936) 2024-02-19T22:06:19.339 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:19.337+0000 7fd13b7fe700 1 -- 172.21.15.86:0/3172724776 learned_addr learned my addr 172.21.15.86:0/3172724776 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T22:06:19.339 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:19.337+0000 7fd13b7fe700 1 -- 172.21.15.86:0/3172724776 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd12c005040 con 0x7fd13c0f5410 2024-02-19T22:06:19.339 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:19.337+0000 7fd13b7fe700 1 --2- 172.21.15.86:0/3172724776 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd13c0f5410 0x7fd13c1056e0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0x7fd12c005010 tx=0x7fd12c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:06:19.339 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:19.337+0000 7fd123fff700 1 -- 172.21.15.86:0/3172724776 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd12c009320 con 0x7fd13c0f5410 2024-02-19T22:06:19.339 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:19.337+0000 7fd141dab700 1 -- 172.21.15.86:0/3172724776 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd13c105dd0 con 0x7fd13c0f5410 2024-02-19T22:06:19.339 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:19.337+0000 7fd141dab700 1 -- 172.21.15.86:0/3172724776 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd13c106260 con 0x7fd13c0f5410 2024-02-19T22:06:19.339 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:19.337+0000 7fd123fff700 1 -- 172.21.15.86:0/3172724776 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd12c013380 con 0x7fd13c0f5410 2024-02-19T22:06:19.339 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:19.337+0000 7fd123fff700 1 -- 172.21.15.86:0/3172724776 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd12c014e50 con 0x7fd13c0f5410 2024-02-19T22:06:19.340 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:19.338+0000 7fd123fff700 1 -- 172.21.15.86:0/3172724776 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd12c0263a0 con 0x7fd13c0f5410 2024-02-19T22:06:19.340 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:19.338+0000 7fd141dab700 1 -- 172.21.15.86:0/3172724776 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd13c045ad0 con 0x7fd13c0f5410 2024-02-19T22:06:19.340 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:19.338+0000 7fd123fff700 1 --2- 172.21.15.86:0/3172724776 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fd12402ff10 0x7fd1240323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:06:19.340 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:19.339+0000 7fd123fff700 1 -- 172.21.15.86:0/3172724776 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd12c040fb0 con 0x7fd13c0f5410 2024-02-19T22:06:19.341 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:19.339+0000 7fd13affd700 1 --2- 172.21.15.86:0/3172724776 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fd12402ff10 0x7fd1240323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:06:19.341 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:19.339+0000 7fd13affd700 1 --2- 172.21.15.86:0/3172724776 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fd12402ff10 0x7fd1240323d0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7fd1300096a0 tx=0x7fd130006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:06:19.343 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:19.343+0000 7fd123fff700 1 -- 172.21.15.86:0/3172724776 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd12c01d9d0 con 0x7fd13c0f5410 2024-02-19T22:06:19.548 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:19.548+0000 7fd141dab700 1 -- 172.21.15.86:0/3172724776 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd13c0fafa0 con 0x7fd13c0f5410 2024-02-19T22:06:19.548 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:19.548+0000 7fd123fff700 1 -- 172.21.15.86:0/3172724776 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd12c01d380 con 0x7fd13c0f5410 2024-02-19T22:06:19.549 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T22:06:19.549 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:06:19.550 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:19.550+0000 7fd141dab700 1 -- 172.21.15.86:0/3172724776 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fd12402ff10 msgr2=0x7fd1240323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:06:19.550 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:19.550+0000 7fd141dab700 1 --2- 172.21.15.86:0/3172724776 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fd12402ff10 0x7fd1240323d0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7fd1300096a0 tx=0x7fd130006b40).stop 2024-02-19T22:06:19.550 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:19.550+0000 7fd141dab700 1 -- 172.21.15.86:0/3172724776 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd13c0f5410 msgr2=0x7fd13c1056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:06:19.550 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:19.550+0000 7fd141dab700 1 --2- 172.21.15.86:0/3172724776 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd13c0f5410 0x7fd13c1056e0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0x7fd12c005010 tx=0x7fd12c00b040).stop 2024-02-19T22:06:19.550 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:19.550+0000 7fd141dab700 1 -- 172.21.15.86:0/3172724776 shutdown_connections 2024-02-19T22:06:19.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:19.550+0000 7fd141dab700 1 --2- 172.21.15.86:0/3172724776 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fd12402ff10 0x7fd1240323d0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:06:19.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:19.550+0000 7fd141dab700 1 --2- 172.21.15.86:0/3172724776 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd13c0f5410 0x7fd13c1056e0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:06:19.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:19.550+0000 7fd141dab700 1 -- 172.21.15.86:0/3172724776 >> 172.21.15.86:0/3172724776 conn(0x7fd13c0f0b70 msgr2=0x7fd13c0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:06:19.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:19.550+0000 7fd141dab700 1 -- 172.21.15.86:0/3172724776 shutdown_connections 2024-02-19T22:06:19.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:19.550+0000 7fd141dab700 1 -- 172.21.15.86:0/3172724776 wait complete. 2024-02-19T22:06:19.551 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T22:06:20.794 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:06:20 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:06:19.527358+0000 mgr.smithi053.fyhcdi (mgr.14162) 300 : cluster [DBG] pgmap v286: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:06:20.794 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:06:20 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:06:19.549600+0000 mon.smithi053 (mon.0) 288 : audit [DBG] from='client.? 172.21.15.86:0/3172724776' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:06:21.300 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:06:21.300 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:06:23.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:06:22 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:06:21.527693+0000 mgr.smithi053.fyhcdi (mgr.14162) 301 : cluster [DBG] pgmap v287: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:06:23.210 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:23.210+0000 7f5d91e21700 1 -- 172.21.15.86:0/2356290398 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5d8c0f51a0 msgr2=0x7f5d8c0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:06:23.210 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:23.210+0000 7f5d91e21700 1 --2- 172.21.15.86:0/2356290398 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5d8c0f51a0 0x7f5d8c0f55c0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 rx=0x7f5d80009fb0 tx=0x7f5d80009b30).stop 2024-02-19T22:06:23.210 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:23.210+0000 7f5d91e21700 1 -- 172.21.15.86:0/2356290398 shutdown_connections 2024-02-19T22:06:23.210 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:23.210+0000 7f5d91e21700 1 --2- 172.21.15.86:0/2356290398 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5d8c0f51a0 0x7f5d8c0f55c0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:06:23.211 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:23.210+0000 7f5d91e21700 1 -- 172.21.15.86:0/2356290398 >> 172.21.15.86:0/2356290398 conn(0x7f5d8c0f09a0 msgr2=0x7f5d8c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:06:23.211 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:23.210+0000 7f5d91e21700 1 -- 172.21.15.86:0/2356290398 shutdown_connections 2024-02-19T22:06:23.211 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:23.210+0000 7f5d91e21700 1 -- 172.21.15.86:0/2356290398 wait complete. 2024-02-19T22:06:23.211 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:23.210+0000 7f5d91e21700 1 Processor -- start 2024-02-19T22:06:23.211 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:23.210+0000 7f5d91e21700 1 -- start start 2024-02-19T22:06:23.213 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:23.211+0000 7f5d91e21700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5d8c0f51a0 0x7f5d8c109290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:06:23.213 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:23.211+0000 7f5d91e21700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d8c060280 con 0x7f5d8c0f51a0 2024-02-19T22:06:23.213 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:23.211+0000 7f5d90e1f700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5d8c0f51a0 0x7f5d8c109290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:06:23.213 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:23.211+0000 7f5d90e1f700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5d8c0f51a0 0x7f5d8c109290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:48964/0 (socket says 172.21.15.86:48964) 2024-02-19T22:06:23.213 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:23.211+0000 7f5d90e1f700 1 -- 172.21.15.86:0/1739031948 learned_addr learned my addr 172.21.15.86:0/1739031948 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T22:06:23.213 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:23.211+0000 7f5d90e1f700 1 -- 172.21.15.86:0/1739031948 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d80005040 con 0x7f5d8c0f51a0 2024-02-19T22:06:23.213 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:23.211+0000 7f5d90e1f700 1 --2- 172.21.15.86:0/1739031948 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5d8c0f51a0 0x7f5d8c109290 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0x7f5d800089f0 tx=0x7f5d8000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:06:23.214 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:23.211+0000 7f5d897fa700 1 -- 172.21.15.86:0/1739031948 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5d80009320 con 0x7f5d8c0f51a0 2024-02-19T22:06:23.214 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:23.212+0000 7f5d91e21700 1 -- 172.21.15.86:0/1739031948 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d8c060480 con 0x7f5d8c0f51a0 2024-02-19T22:06:23.214 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:23.212+0000 7f5d897fa700 1 -- 172.21.15.86:0/1739031948 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5d80013380 con 0x7f5d8c0f51a0 2024-02-19T22:06:23.214 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:23.212+0000 7f5d897fa700 1 -- 172.21.15.86:0/1739031948 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5d80014e20 con 0x7f5d8c0f51a0 2024-02-19T22:06:23.214 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:23.212+0000 7f5d91e21700 1 -- 172.21.15.86:0/1739031948 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d8c1099c0 con 0x7f5d8c0f51a0 2024-02-19T22:06:23.214 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:23.212+0000 7f5d897fa700 1 -- 172.21.15.86:0/1739031948 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5d800095f0 con 0x7f5d8c0f51a0 2024-02-19T22:06:23.214 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:23.212+0000 7f5d91e21700 1 -- 172.21.15.86:0/1739031948 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d8c045ad0 con 0x7f5d8c0f51a0 2024-02-19T22:06:23.214 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:23.212+0000 7f5d897fa700 1 --2- 172.21.15.86:0/1739031948 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f5d7c02ff10 0x7f5d7c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:06:23.214 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:23.213+0000 7f5d897fa700 1 -- 172.21.15.86:0/1739031948 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5d80040ef0 con 0x7f5d8c0f51a0 2024-02-19T22:06:23.214 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:23.213+0000 7f5d8bfff700 1 --2- 172.21.15.86:0/1739031948 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f5d7c02ff10 0x7f5d7c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:06:23.214 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:23.213+0000 7f5d8bfff700 1 --2- 172.21.15.86:0/1739031948 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f5d7c02ff10 0x7f5d7c0323d0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7f5d780096a0 tx=0x7f5d78006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:06:23.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:23.217+0000 7f5d897fa700 1 -- 172.21.15.86:0/1739031948 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5d80014420 con 0x7f5d8c0f51a0 2024-02-19T22:06:23.420 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:23.420+0000 7f5d91e21700 1 -- 172.21.15.86:0/1739031948 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5d8c0fad30 con 0x7f5d8c0f51a0 2024-02-19T22:06:23.421 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:23.421+0000 7f5d897fa700 1 -- 172.21.15.86:0/1739031948 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5d80010230 con 0x7f5d8c0f51a0 2024-02-19T22:06:23.421 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T22:06:23.421 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:06:23.423 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:23.423+0000 7f5d91e21700 1 -- 172.21.15.86:0/1739031948 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f5d7c02ff10 msgr2=0x7f5d7c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:06:23.423 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:23.423+0000 7f5d91e21700 1 --2- 172.21.15.86:0/1739031948 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f5d7c02ff10 0x7f5d7c0323d0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7f5d780096a0 tx=0x7f5d78006b40).stop 2024-02-19T22:06:23.423 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:23.423+0000 7f5d91e21700 1 -- 172.21.15.86:0/1739031948 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5d8c0f51a0 msgr2=0x7f5d8c109290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:06:23.423 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:23.423+0000 7f5d91e21700 1 --2- 172.21.15.86:0/1739031948 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5d8c0f51a0 0x7f5d8c109290 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0x7f5d800089f0 tx=0x7f5d8000b040).stop 2024-02-19T22:06:23.423 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:23.423+0000 7f5d91e21700 1 -- 172.21.15.86:0/1739031948 shutdown_connections 2024-02-19T22:06:23.423 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:23.423+0000 7f5d91e21700 1 --2- 172.21.15.86:0/1739031948 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f5d7c02ff10 0x7f5d7c0323d0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:06:23.423 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:23.423+0000 7f5d91e21700 1 --2- 172.21.15.86:0/1739031948 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5d8c0f51a0 0x7f5d8c109290 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:06:23.423 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:23.423+0000 7f5d91e21700 1 -- 172.21.15.86:0/1739031948 >> 172.21.15.86:0/1739031948 conn(0x7f5d8c0f09a0 msgr2=0x7f5d8c0f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:06:23.424 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:23.423+0000 7f5d91e21700 1 -- 172.21.15.86:0/1739031948 shutdown_connections 2024-02-19T22:06:23.424 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:23.423+0000 7f5d91e21700 1 -- 172.21.15.86:0/1739031948 wait complete. 2024-02-19T22:06:23.424 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T22:06:24.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:06:23 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:06:23.422349+0000 mon.smithi053 (mon.0) 289 : audit [DBG] from='client.? 172.21.15.86:0/1739031948' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:06:25.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:06:24 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:06:23.527971+0000 mgr.smithi053.fyhcdi (mgr.14162) 302 : cluster [DBG] pgmap v288: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:06:25.088 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:06:25.088 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:06:27.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:06:26 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:06:25.528264+0000 mgr.smithi053.fyhcdi (mgr.14162) 303 : cluster [DBG] pgmap v289: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:06:27.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:27.141+0000 7fc437830700 1 -- 172.21.15.86:0/3254327193 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc4300f5410 msgr2=0x7fc4300f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:06:27.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:27.141+0000 7fc437830700 1 --2- 172.21.15.86:0/3254327193 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc4300f5410 0x7fc4300f5830 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 rx=0x7fc420009fb0 tx=0x7fc420009b30).stop 2024-02-19T22:06:27.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:27.141+0000 7fc437830700 1 -- 172.21.15.86:0/3254327193 shutdown_connections 2024-02-19T22:06:27.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:27.141+0000 7fc437830700 1 --2- 172.21.15.86:0/3254327193 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc4300f5410 0x7fc4300f5830 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:06:27.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:27.141+0000 7fc437830700 1 -- 172.21.15.86:0/3254327193 >> 172.21.15.86:0/3254327193 conn(0x7fc4300f0b70 msgr2=0x7fc4300f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:06:27.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:27.142+0000 7fc437830700 1 -- 172.21.15.86:0/3254327193 shutdown_connections 2024-02-19T22:06:27.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:27.142+0000 7fc437830700 1 -- 172.21.15.86:0/3254327193 wait complete. 2024-02-19T22:06:27.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:27.142+0000 7fc437830700 1 Processor -- start 2024-02-19T22:06:27.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:27.142+0000 7fc437830700 1 -- start start 2024-02-19T22:06:27.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:27.142+0000 7fc437830700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc4300f5410 0x7fc4301067b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:06:27.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:27.142+0000 7fc437830700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc430106ca0 con 0x7fc4300f5410 2024-02-19T22:06:27.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:27.143+0000 7fc4355cc700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc4300f5410 0x7fc4301067b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:06:27.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:27.143+0000 7fc4355cc700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc4300f5410 0x7fc4301067b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:48990/0 (socket says 172.21.15.86:48990) 2024-02-19T22:06:27.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:27.143+0000 7fc4355cc700 1 -- 172.21.15.86:0/4092214406 learned_addr learned my addr 172.21.15.86:0/4092214406 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T22:06:27.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:27.143+0000 7fc4355cc700 1 -- 172.21.15.86:0/4092214406 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc420005040 con 0x7fc4300f5410 2024-02-19T22:06:27.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:27.143+0000 7fc4355cc700 1 --2- 172.21.15.86:0/4092214406 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc4300f5410 0x7fc4301067b0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 rx=0x7fc420005010 tx=0x7fc42000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:06:27.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:27.143+0000 7fc425ffb700 1 -- 172.21.15.86:0/4092214406 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc420009320 con 0x7fc4300f5410 2024-02-19T22:06:27.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:27.143+0000 7fc437830700 1 -- 172.21.15.86:0/4092214406 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc430106ea0 con 0x7fc4300f5410 2024-02-19T22:06:27.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:27.143+0000 7fc425ffb700 1 -- 172.21.15.86:0/4092214406 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc420013380 con 0x7fc4300f5410 2024-02-19T22:06:27.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:27.143+0000 7fc425ffb700 1 -- 172.21.15.86:0/4092214406 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc420014e50 con 0x7fc4300f5410 2024-02-19T22:06:27.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:27.143+0000 7fc437830700 1 -- 172.21.15.86:0/4092214406 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc430107330 con 0x7fc4300f5410 2024-02-19T22:06:27.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:27.144+0000 7fc425ffb700 1 -- 172.21.15.86:0/4092214406 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc420014420 con 0x7fc4300f5410 2024-02-19T22:06:27.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:27.144+0000 7fc437830700 1 -- 172.21.15.86:0/4092214406 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc430045ad0 con 0x7fc4300f5410 2024-02-19T22:06:27.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:27.144+0000 7fc425ffb700 1 --2- 172.21.15.86:0/4092214406 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fc41c02ff10 0x7fc41c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:06:27.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:27.144+0000 7fc425ffb700 1 -- 172.21.15.86:0/4092214406 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc420040f70 con 0x7fc4300f5410 2024-02-19T22:06:27.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:27.145+0000 7fc434dcb700 1 --2- 172.21.15.86:0/4092214406 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fc41c02ff10 0x7fc41c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:06:27.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:27.145+0000 7fc434dcb700 1 --2- 172.21.15.86:0/4092214406 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fc41c02ff10 0x7fc41c0323d0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7fc42c0096a0 tx=0x7fc42c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:06:27.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:27.149+0000 7fc425ffb700 1 -- 172.21.15.86:0/4092214406 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc4200095f0 con 0x7fc4300f5410 2024-02-19T22:06:27.355 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:27.355+0000 7fc437830700 1 -- 172.21.15.86:0/4092214406 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc4301081e0 con 0x7fc4300f5410 2024-02-19T22:06:27.356 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:27.356+0000 7fc425ffb700 1 -- 172.21.15.86:0/4092214406 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc42001db70 con 0x7fc4300f5410 2024-02-19T22:06:27.356 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T22:06:27.356 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:06:27.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:27.357+0000 7fc437830700 1 -- 172.21.15.86:0/4092214406 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fc41c02ff10 msgr2=0x7fc41c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:06:27.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:27.358+0000 7fc437830700 1 --2- 172.21.15.86:0/4092214406 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fc41c02ff10 0x7fc41c0323d0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7fc42c0096a0 tx=0x7fc42c006b40).stop 2024-02-19T22:06:27.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:27.358+0000 7fc437830700 1 -- 172.21.15.86:0/4092214406 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc4300f5410 msgr2=0x7fc4301067b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:06:27.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:27.358+0000 7fc437830700 1 --2- 172.21.15.86:0/4092214406 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc4300f5410 0x7fc4301067b0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 rx=0x7fc420005010 tx=0x7fc42000b040).stop 2024-02-19T22:06:27.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:27.358+0000 7fc437830700 1 -- 172.21.15.86:0/4092214406 shutdown_connections 2024-02-19T22:06:27.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:27.358+0000 7fc437830700 1 --2- 172.21.15.86:0/4092214406 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fc41c02ff10 0x7fc41c0323d0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:06:27.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:27.358+0000 7fc437830700 1 --2- 172.21.15.86:0/4092214406 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc4300f5410 0x7fc4301067b0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:06:27.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:27.358+0000 7fc437830700 1 -- 172.21.15.86:0/4092214406 >> 172.21.15.86:0/4092214406 conn(0x7fc4300f0b70 msgr2=0x7fc43005fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:06:27.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:27.358+0000 7fc437830700 1 -- 172.21.15.86:0/4092214406 shutdown_connections 2024-02-19T22:06:27.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:27.358+0000 7fc437830700 1 -- 172.21.15.86:0/4092214406 wait complete. 2024-02-19T22:06:27.359 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T22:06:28.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:06:27 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:06:27.357177+0000 mon.smithi053 (mon.0) 290 : audit [DBG] from='client.? 172.21.15.86:0/4092214406' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:06:29.012 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:06:29.012 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:06:29.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:06:28 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:06:27.528614+0000 mgr.smithi053.fyhcdi (mgr.14162) 304 : cluster [DBG] pgmap v290: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:06:30.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:06:30 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:06:29.528915+0000 mgr.smithi053.fyhcdi (mgr.14162) 305 : cluster [DBG] pgmap v291: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:06:31.126 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:31.123+0000 7f83d3fff700 1 -- 172.21.15.86:0/2673567821 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f83d40f62a0 msgr2=0x7f83d40f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:06:31.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:31.123+0000 7f83d3fff700 1 --2- 172.21.15.86:0/2673567821 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f83d40f62a0 0x7f83d40f66c0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 rx=0x7f83c4009fb0 tx=0x7f83c4009b30).stop 2024-02-19T22:06:31.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:31.123+0000 7f83d3fff700 1 -- 172.21.15.86:0/2673567821 shutdown_connections 2024-02-19T22:06:31.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:31.123+0000 7f83d3fff700 1 --2- 172.21.15.86:0/2673567821 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f83d40f62a0 0x7f83d40f66c0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:06:31.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:31.123+0000 7f83d3fff700 1 -- 172.21.15.86:0/2673567821 >> 172.21.15.86:0/2673567821 conn(0x7f83d40f1aa0 msgr2=0x7f83d40f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:06:31.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:31.123+0000 7f83d3fff700 1 -- 172.21.15.86:0/2673567821 shutdown_connections 2024-02-19T22:06:31.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:31.123+0000 7f83d3fff700 1 -- 172.21.15.86:0/2673567821 wait complete. 2024-02-19T22:06:31.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:31.123+0000 7f83d3fff700 1 Processor -- start 2024-02-19T22:06:31.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:31.123+0000 7f83d3fff700 1 -- start start 2024-02-19T22:06:31.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:31.124+0000 7f83d3fff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f83d40f62a0 0x7f83d41065d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:06:31.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:31.124+0000 7f83d3fff700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83d4106ac0 con 0x7f83d40f62a0 2024-02-19T22:06:31.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:31.124+0000 7f83d2ffd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f83d40f62a0 0x7f83d41065d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:06:31.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:31.124+0000 7f83d2ffd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f83d40f62a0 0x7f83d41065d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:43540/0 (socket says 172.21.15.86:43540) 2024-02-19T22:06:31.128 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:31.124+0000 7f83d2ffd700 1 -- 172.21.15.86:0/2226370143 learned_addr learned my addr 172.21.15.86:0/2226370143 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T22:06:31.128 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:31.124+0000 7f83d2ffd700 1 -- 172.21.15.86:0/2226370143 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83c4005040 con 0x7f83d40f62a0 2024-02-19T22:06:31.128 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:31.124+0000 7f83d2ffd700 1 --2- 172.21.15.86:0/2226370143 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f83d40f62a0 0x7f83d41065d0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0x7f83c40089f0 tx=0x7f83c400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:06:31.128 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:31.124+0000 7f83bb7fe700 1 -- 172.21.15.86:0/2226370143 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f83c4009320 con 0x7f83d40f62a0 2024-02-19T22:06:31.128 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:31.124+0000 7f83d3fff700 1 -- 172.21.15.86:0/2226370143 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83d4106cc0 con 0x7f83d40f62a0 2024-02-19T22:06:31.128 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:31.124+0000 7f83bb7fe700 1 -- 172.21.15.86:0/2226370143 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f83c4013380 con 0x7f83d40f62a0 2024-02-19T22:06:31.128 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:31.124+0000 7f83bb7fe700 1 -- 172.21.15.86:0/2226370143 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f83c4014e20 con 0x7f83d40f62a0 2024-02-19T22:06:31.128 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:31.124+0000 7f83d3fff700 1 -- 172.21.15.86:0/2226370143 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83d4107150 con 0x7f83d40f62a0 2024-02-19T22:06:31.128 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:31.125+0000 7f83d3fff700 1 -- 172.21.15.86:0/2226370143 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83d4045ad0 con 0x7f83d40f62a0 2024-02-19T22:06:31.128 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:31.125+0000 7f83bb7fe700 1 -- 172.21.15.86:0/2226370143 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f83c40095f0 con 0x7f83d40f62a0 2024-02-19T22:06:31.128 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:31.125+0000 7f83bb7fe700 1 --2- 172.21.15.86:0/2226370143 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f83bc02ff10 0x7f83bc0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:06:31.128 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:31.125+0000 7f83bb7fe700 1 -- 172.21.15.86:0/2226370143 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f83c4040ec0 con 0x7f83d40f62a0 2024-02-19T22:06:31.128 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:31.126+0000 7f83d27fc700 1 --2- 172.21.15.86:0/2226370143 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f83bc02ff10 0x7f83bc0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:06:31.129 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:31.126+0000 7f83d27fc700 1 --2- 172.21.15.86:0/2226370143 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f83bc02ff10 0x7f83bc0323d0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f83c80096a0 tx=0x7f83c8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:06:31.130 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:31.130+0000 7f83bb7fe700 1 -- 172.21.15.86:0/2226370143 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f83c4014420 con 0x7f83d40f62a0 2024-02-19T22:06:31.334 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:31.334+0000 7f83d3fff700 1 -- 172.21.15.86:0/2226370143 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f83d4107fa0 con 0x7f83d40f62a0 2024-02-19T22:06:31.335 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:31.335+0000 7f83bb7fe700 1 -- 172.21.15.86:0/2226370143 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f83c4010230 con 0x7f83d40f62a0 2024-02-19T22:06:31.335 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T22:06:31.335 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:06:31.337 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:31.336+0000 7f83d3fff700 1 -- 172.21.15.86:0/2226370143 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f83bc02ff10 msgr2=0x7f83bc0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:06:31.337 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:31.336+0000 7f83d3fff700 1 --2- 172.21.15.86:0/2226370143 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f83bc02ff10 0x7f83bc0323d0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f83c80096a0 tx=0x7f83c8006b40).stop 2024-02-19T22:06:31.337 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:31.336+0000 7f83d3fff700 1 -- 172.21.15.86:0/2226370143 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f83d40f62a0 msgr2=0x7f83d41065d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:06:31.337 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:31.336+0000 7f83d3fff700 1 --2- 172.21.15.86:0/2226370143 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f83d40f62a0 0x7f83d41065d0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0x7f83c40089f0 tx=0x7f83c400b040).stop 2024-02-19T22:06:31.337 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:31.336+0000 7f83d3fff700 1 -- 172.21.15.86:0/2226370143 shutdown_connections 2024-02-19T22:06:31.338 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:31.336+0000 7f83d3fff700 1 --2- 172.21.15.86:0/2226370143 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f83bc02ff10 0x7f83bc0323d0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:06:31.338 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:31.336+0000 7f83d3fff700 1 --2- 172.21.15.86:0/2226370143 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f83d40f62a0 0x7f83d41065d0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:06:31.338 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:31.336+0000 7f83d3fff700 1 -- 172.21.15.86:0/2226370143 >> 172.21.15.86:0/2226370143 conn(0x7f83d40f1aa0 msgr2=0x7f83d40f2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:06:31.338 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:31.336+0000 7f83d3fff700 1 -- 172.21.15.86:0/2226370143 shutdown_connections 2024-02-19T22:06:31.338 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:31.336+0000 7f83d3fff700 1 -- 172.21.15.86:0/2226370143 wait complete. 2024-02-19T22:06:31.338 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T22:06:31.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:06:31 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:06:31.335996+0000 mon.smithi053 (mon.0) 291 : audit [DBG] from='client.? 172.21.15.86:0/2226370143' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:06:32.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:06:32 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:06:31.529153+0000 mgr.smithi053.fyhcdi (mgr.14162) 306 : cluster [DBG] pgmap v292: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:06:33.073 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:06:33.073 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:06:34.973 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:34.971+0000 7fce271aa700 1 -- 172.21.15.86:0/2768500596 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fce200f5410 msgr2=0x7fce200f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:06:34.973 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:34.971+0000 7fce271aa700 1 --2- 172.21.15.86:0/2768500596 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fce200f5410 0x7fce200f5830 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 rx=0x7fce14009fb0 tx=0x7fce14009b30).stop 2024-02-19T22:06:34.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:34.972+0000 7fce271aa700 1 -- 172.21.15.86:0/2768500596 shutdown_connections 2024-02-19T22:06:34.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:34.972+0000 7fce271aa700 1 --2- 172.21.15.86:0/2768500596 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fce200f5410 0x7fce200f5830 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:06:34.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:34.972+0000 7fce271aa700 1 -- 172.21.15.86:0/2768500596 >> 172.21.15.86:0/2768500596 conn(0x7fce200f0b70 msgr2=0x7fce200f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:06:34.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:34.972+0000 7fce271aa700 1 -- 172.21.15.86:0/2768500596 shutdown_connections 2024-02-19T22:06:34.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:34.972+0000 7fce271aa700 1 -- 172.21.15.86:0/2768500596 wait complete. 2024-02-19T22:06:34.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:34.972+0000 7fce271aa700 1 Processor -- start 2024-02-19T22:06:34.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:34.972+0000 7fce271aa700 1 -- start start 2024-02-19T22:06:34.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:34.972+0000 7fce271aa700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fce200f5410 0x7fce20106800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:06:34.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:34.972+0000 7fce271aa700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce20106cf0 con 0x7fce200f5410 2024-02-19T22:06:34.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:34.973+0000 7fce24f46700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fce200f5410 0x7fce20106800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:06:34.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:34.973+0000 7fce24f46700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fce200f5410 0x7fce20106800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:43554/0 (socket says 172.21.15.86:43554) 2024-02-19T22:06:34.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:34.973+0000 7fce24f46700 1 -- 172.21.15.86:0/3373493988 learned_addr learned my addr 172.21.15.86:0/3373493988 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T22:06:34.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:34.973+0000 7fce24f46700 1 -- 172.21.15.86:0/3373493988 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce14005040 con 0x7fce200f5410 2024-02-19T22:06:34.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:34.973+0000 7fce24f46700 1 --2- 172.21.15.86:0/3373493988 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fce200f5410 0x7fce20106800 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 rx=0x7fce140089f0 tx=0x7fce1400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:06:34.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:34.973+0000 7fce1d7fa700 1 -- 172.21.15.86:0/3373493988 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fce14009320 con 0x7fce200f5410 2024-02-19T22:06:34.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:34.973+0000 7fce271aa700 1 -- 172.21.15.86:0/3373493988 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce20106ef0 con 0x7fce200f5410 2024-02-19T22:06:34.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:34.973+0000 7fce1d7fa700 1 -- 172.21.15.86:0/3373493988 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fce14013380 con 0x7fce200f5410 2024-02-19T22:06:34.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:34.973+0000 7fce271aa700 1 -- 172.21.15.86:0/3373493988 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce20107380 con 0x7fce200f5410 2024-02-19T22:06:34.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:34.973+0000 7fce1d7fa700 1 -- 172.21.15.86:0/3373493988 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fce1401ed80 con 0x7fce200f5410 2024-02-19T22:06:34.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:34.974+0000 7fce271aa700 1 -- 172.21.15.86:0/3373493988 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce20045ad0 con 0x7fce200f5410 2024-02-19T22:06:34.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:34.974+0000 7fce1d7fa700 1 -- 172.21.15.86:0/3373493988 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fce1401e420 con 0x7fce200f5410 2024-02-19T22:06:34.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:34.974+0000 7fce1d7fa700 1 --2- 172.21.15.86:0/3373493988 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fce10034370 0x7fce10036830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:06:34.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:34.974+0000 7fce1d7fa700 1 -- 172.21.15.86:0/3373493988 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fce14041060 con 0x7fce200f5410 2024-02-19T22:06:34.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:34.975+0000 7fce1ffff700 1 --2- 172.21.15.86:0/3373493988 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fce10034370 0x7fce10036830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:06:34.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:34.975+0000 7fce1ffff700 1 --2- 172.21.15.86:0/3373493988 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fce10034370 0x7fce10036830 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7fce0c0096a0 tx=0x7fce0c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:06:34.979 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:34.979+0000 7fce1d7fa700 1 -- 172.21.15.86:0/3373493988 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fce140095f0 con 0x7fce200f5410 2024-02-19T22:06:35.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:06:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:06:33.529419+0000 mgr.smithi053.fyhcdi (mgr.14162) 307 : cluster [DBG] pgmap v293: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:06:35.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:35.184+0000 7fce271aa700 1 -- 172.21.15.86:0/3373493988 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fce201081d0 con 0x7fce200f5410 2024-02-19T22:06:35.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:35.185+0000 7fce1d7fa700 1 -- 172.21.15.86:0/3373493988 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fce1401cb10 con 0x7fce200f5410 2024-02-19T22:06:35.186 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T22:06:35.186 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:06:35.187 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:35.187+0000 7fce271aa700 1 -- 172.21.15.86:0/3373493988 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fce10034370 msgr2=0x7fce10036830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:06:35.187 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:35.187+0000 7fce271aa700 1 --2- 172.21.15.86:0/3373493988 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fce10034370 0x7fce10036830 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7fce0c0096a0 tx=0x7fce0c006b40).stop 2024-02-19T22:06:35.187 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:35.187+0000 7fce271aa700 1 -- 172.21.15.86:0/3373493988 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fce200f5410 msgr2=0x7fce20106800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:06:35.187 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:35.187+0000 7fce271aa700 1 --2- 172.21.15.86:0/3373493988 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fce200f5410 0x7fce20106800 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 rx=0x7fce140089f0 tx=0x7fce1400b040).stop 2024-02-19T22:06:35.187 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:35.187+0000 7fce271aa700 1 -- 172.21.15.86:0/3373493988 shutdown_connections 2024-02-19T22:06:35.187 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:35.187+0000 7fce271aa700 1 --2- 172.21.15.86:0/3373493988 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fce10034370 0x7fce10036830 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:06:35.187 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:35.187+0000 7fce271aa700 1 --2- 172.21.15.86:0/3373493988 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fce200f5410 0x7fce20106800 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:06:35.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:35.187+0000 7fce271aa700 1 -- 172.21.15.86:0/3373493988 >> 172.21.15.86:0/3373493988 conn(0x7fce200f0b70 msgr2=0x7fce200f15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:06:35.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:35.187+0000 7fce271aa700 1 -- 172.21.15.86:0/3373493988 shutdown_connections 2024-02-19T22:06:35.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:35.187+0000 7fce271aa700 1 -- 172.21.15.86:0/3373493988 wait complete. 2024-02-19T22:06:35.188 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T22:06:35.914 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:06:35 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:06:35.186437+0000 mon.smithi053 (mon.0) 292 : audit [DBG] from='client.? 172.21.15.86:0/3373493988' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:06:36.899 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:06:36.900 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:06:37.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:06:36 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:06:35.529712+0000 mgr.smithi053.fyhcdi (mgr.14162) 308 : cluster [DBG] pgmap v294: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:06:37.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:06:36 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:06:35.606601+0000 mon.smithi053 (mon.0) 293 : audit [INF] from='mgr.14162 172.21.15.53:0/2860358250' entity='mgr.smithi053.fyhcdi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi053.fyhcdi/mirror_snapshot_schedule"}]: dispatch 2024-02-19T22:06:37.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:06:36 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:06:35.609186+0000 mon.smithi053 (mon.0) 294 : audit [INF] from='mgr.14162 172.21.15.53:0/2860358250' entity='mgr.smithi053.fyhcdi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi053.fyhcdi/trash_purge_schedule"}]: dispatch 2024-02-19T22:06:37.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:06:36 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:06:35.915860+0000 mon.smithi053 (mon.0) 295 : audit [DBG] from='mgr.14162 172.21.15.53:0/2860358250' entity='mgr.smithi053.fyhcdi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T22:06:37.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:06:36 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:06:36.143516+0000 mon.smithi053 (mon.0) 296 : audit [INF] from='mgr.14162 172.21.15.53:0/2860358250' entity='mgr.smithi053.fyhcdi' 2024-02-19T22:06:37.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:06:36 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:06:36.152724+0000 mon.smithi053 (mon.0) 297 : audit [INF] from='mgr.14162 172.21.15.53:0/2860358250' entity='mgr.smithi053.fyhcdi' 2024-02-19T22:06:37.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:06:36 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:06:36.374648+0000 mon.smithi053 (mon.0) 298 : audit [INF] from='mgr.14162 172.21.15.53:0/2860358250' entity='mgr.smithi053.fyhcdi' 2024-02-19T22:06:38.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:06:37 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cephadm 2024-02-19T22:06:35.915253+0000 mgr.smithi053.fyhcdi (mgr.14162) 309 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Deploy daemon prometheus.smithi053 ... 2024-02-19T22:06:38.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:06:37 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/prometheus:v2.18.1 -e NODE_NAME=smithi053 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/prometheus:v2.18.1 -c %u %g /etc/prometheus 2024-02-19T22:06:38.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:06:37 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: stat: stderr Trying to pull docker.io/prom/prometheus:v2.18.1... 2024-02-19T22:06:38.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:06:37 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: stat: stderr time="2024-02-19T22:04:43Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-19T22:06:38.045 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:06:37 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: stat: stderr time="2024-02-19T22:05:20Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-19T22:06:38.045 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:06:37 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: stat: stderr time="2024-02-19T22:05:58Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-19T22:06:38.045 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:06:37 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: stat: stderr Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-19T22:06:38.045 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:06:37 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: Traceback (most recent call last): 2024-02-19T22:06:38.045 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:06:37 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: File "/var/lib/ceph/867b6b52-cf71-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-02-19T22:06:38.045 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:06:37 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: main() 2024-02-19T22:06:38.045 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:06:37 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: File "/var/lib/ceph/867b6b52-cf71-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-02-19T22:06:38.045 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:06:37 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: r = ctx.func(ctx) 2024-02-19T22:06:38.045 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:06:37 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: File "/var/lib/ceph/867b6b52-cf71-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-02-19T22:06:38.045 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:06:37 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: return func(ctx) 2024-02-19T22:06:38.045 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:06:37 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: File "/var/lib/ceph/867b6b52-cf71-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy 2024-02-19T22:06:38.045 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:06:37 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2024-02-19T22:06:38.046 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:06:37 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: File "/var/lib/ceph/867b6b52-cf71-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4185, in extract_uid_gid_monitoring 2024-02-19T22:06:38.046 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:06:37 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: uid, gid = extract_uid_gid(ctx, file_path='/etc/prometheus') 2024-02-19T22:06:38.046 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:06:37 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: File "/var/lib/ceph/867b6b52-cf71-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid 2024-02-19T22:06:38.046 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:06:37 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: raise RuntimeError('uid/gid not found') 2024-02-19T22:06:38.046 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:06:37 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: RuntimeError: uid/gid not found 2024-02-19T22:06:38.046 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:06:37 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: Traceback (most recent call last): 2024-02-19T22:06:38.046 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:06:37 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1144, in _remote_connection 2024-02-19T22:06:38.046 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:06:37 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: yield (conn, connr) 2024-02-19T22:06:38.046 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:06:37 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1059, in _run_cephadm 2024-02-19T22:06:38.046 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:06:37 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: code, '\n'.join(err))) 2024-02-19T22:06:38.046 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:06:37 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Deploy daemon prometheus.smithi053 ... 2024-02-19T22:06:38.046 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:06:37 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/prometheus:v2.18.1 -e NODE_NAME=smithi053 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/prometheus:v2.18.1 -c %u %g /etc/prometheus 2024-02-19T22:06:38.046 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:06:37 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: stat: stderr Trying to pull docker.io/prom/prometheus:v2.18.1... 2024-02-19T22:06:38.047 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:06:37 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: stat: stderr time="2024-02-19T22:04:43Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-19T22:06:38.047 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:06:37 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: stat: stderr time="2024-02-19T22:05:20Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-19T22:06:38.047 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:06:37 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: stat: stderr time="2024-02-19T22:05:58Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-19T22:06:38.047 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:06:37 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: stat: stderr Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-19T22:06:38.047 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:06:37 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: Traceback (most recent call last): 2024-02-19T22:06:38.047 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:06:37 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: File "/var/lib/ceph/867b6b52-cf71-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-02-19T22:06:38.047 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:06:37 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: main() 2024-02-19T22:06:38.047 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:06:37 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: File "/var/lib/ceph/867b6b52-cf71-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-02-19T22:06:38.047 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:06:37 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: r = ctx.func(ctx) 2024-02-19T22:06:38.047 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:06:37 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: File "/var/lib/ceph/867b6b52-cf71-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-02-19T22:06:38.047 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:06:37 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: return func(ctx) 2024-02-19T22:06:38.047 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:06:37 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: File "/var/lib/ceph/867b6b52-cf71-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy 2024-02-19T22:06:38.047 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:06:37 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2024-02-19T22:06:38.047 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:06:37 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: File "/var/lib/ceph/867b6b52-cf71-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4185, in extract_uid_gid_monitoring 2024-02-19T22:06:38.048 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:06:37 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: uid, gid = extract_uid_gid(ctx, file_path='/etc/prometheus') 2024-02-19T22:06:38.048 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:06:37 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: File "/var/lib/ceph/867b6b52-cf71-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid 2024-02-19T22:06:38.048 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:06:37 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: raise RuntimeError('uid/gid not found') 2024-02-19T22:06:38.048 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:06:37 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: RuntimeError: uid/gid not found 2024-02-19T22:06:39.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:06:38 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:06:37.529968+0000 mgr.smithi053.fyhcdi (mgr.14162) 310 : cluster [DBG] pgmap v295: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:06:39.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:06:38 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:06:38.057777+0000 mon.smithi053 (mon.0) 299 : audit [INF] from='mgr.14162 172.21.15.53:0/2860358250' entity='mgr.smithi053.fyhcdi' 2024-02-19T22:06:40.470 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:40.264+0000 7f1d8eecf700 1 -- 172.21.15.86:0/3694732752 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1d880f5410 msgr2=0x7f1d880f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:06:40.470 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:40.264+0000 7f1d8eecf700 1 --2- 172.21.15.86:0/3694732752 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1d880f5410 0x7f1d880f5830 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 rx=0x7f1d7c009fb0 tx=0x7f1d7c009b30).stop 2024-02-19T22:06:40.471 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:40.264+0000 7f1d8eecf700 1 -- 172.21.15.86:0/3694732752 shutdown_connections 2024-02-19T22:06:40.471 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:40.264+0000 7f1d8eecf700 1 --2- 172.21.15.86:0/3694732752 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1d880f5410 0x7f1d880f5830 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:06:40.471 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:40.264+0000 7f1d8eecf700 1 -- 172.21.15.86:0/3694732752 >> 172.21.15.86:0/3694732752 conn(0x7f1d880f0b70 msgr2=0x7f1d880f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:06:40.471 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:40.264+0000 7f1d8eecf700 1 -- 172.21.15.86:0/3694732752 shutdown_connections 2024-02-19T22:06:40.471 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:40.264+0000 7f1d8eecf700 1 -- 172.21.15.86:0/3694732752 wait complete. 2024-02-19T22:06:40.471 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:40.265+0000 7f1d8eecf700 1 Processor -- start 2024-02-19T22:06:40.471 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:40.265+0000 7f1d8eecf700 1 -- start start 2024-02-19T22:06:40.471 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:40.265+0000 7f1d8eecf700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1d880f5410 0x7f1d880fd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:06:40.471 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:40.265+0000 7f1d8eecf700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d880fb960 con 0x7f1d880f5410 2024-02-19T22:06:40.471 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:40.265+0000 7f1d8cc6b700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1d880f5410 0x7f1d880fd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:06:40.471 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:40.265+0000 7f1d8cc6b700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1d880f5410 0x7f1d880fd4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:32986/0 (socket says 172.21.15.86:32986) 2024-02-19T22:06:40.471 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:40.265+0000 7f1d8cc6b700 1 -- 172.21.15.86:0/2599023322 learned_addr learned my addr 172.21.15.86:0/2599023322 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T22:06:40.471 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:40.266+0000 7f1d8cc6b700 1 -- 172.21.15.86:0/2599023322 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d7c005040 con 0x7f1d880f5410 2024-02-19T22:06:40.471 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:40.266+0000 7f1d8cc6b700 1 --2- 172.21.15.86:0/2599023322 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1d880f5410 0x7f1d880fd4e0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0x7f1d7c005010 tx=0x7f1d7c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:06:40.472 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:40.266+0000 7f1d857fa700 1 -- 172.21.15.86:0/2599023322 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1d7c009320 con 0x7f1d880f5410 2024-02-19T22:06:40.472 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:40.266+0000 7f1d8eecf700 1 -- 172.21.15.86:0/2599023322 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d880fbb00 con 0x7f1d880f5410 2024-02-19T22:06:40.472 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:40.266+0000 7f1d857fa700 1 -- 172.21.15.86:0/2599023322 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1d7c013380 con 0x7f1d880f5410 2024-02-19T22:06:40.472 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:40.266+0000 7f1d857fa700 1 -- 172.21.15.86:0/2599023322 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1d7c014e50 con 0x7f1d880f5410 2024-02-19T22:06:40.472 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:40.266+0000 7f1d8eecf700 1 -- 172.21.15.86:0/2599023322 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d880fbf90 con 0x7f1d880f5410 2024-02-19T22:06:40.472 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:40.267+0000 7f1d857fa700 1 -- 172.21.15.86:0/2599023322 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1d7c0263a0 con 0x7f1d880f5410 2024-02-19T22:06:40.472 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:40.267+0000 7f1d857fa700 1 --2- 172.21.15.86:0/2599023322 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f1d7802ff10 0x7f1d780323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:06:40.472 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:40.267+0000 7f1d857fa700 1 -- 172.21.15.86:0/2599023322 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1d7c040f10 con 0x7f1d880f5410 2024-02-19T22:06:40.472 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:40.267+0000 7f1d8eecf700 1 -- 172.21.15.86:0/2599023322 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d88059100 con 0x7f1d880f5410 2024-02-19T22:06:40.472 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:40.267+0000 7f1d87fff700 1 --2- 172.21.15.86:0/2599023322 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f1d7802ff10 0x7f1d780323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:06:40.472 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:40.268+0000 7f1d87fff700 1 --2- 172.21.15.86:0/2599023322 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f1d7802ff10 0x7f1d780323d0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7f1d740096a0 tx=0x7f1d74006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:06:40.472 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:40.272+0000 7f1d857fa700 1 -- 172.21.15.86:0/2599023322 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1d7c012920 con 0x7f1d880f5410 2024-02-19T22:06:40.482 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:40.482+0000 7f1d8eecf700 1 -- 172.21.15.86:0/2599023322 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1d880fafa0 con 0x7f1d880f5410 2024-02-19T22:06:40.483 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:40.482+0000 7f1d857fa700 1 -- 172.21.15.86:0/2599023322 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1d7c012330 con 0x7f1d880f5410 2024-02-19T22:06:40.483 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T22:06:40.484 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:06:40.485 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:40.484+0000 7f1d8eecf700 1 -- 172.21.15.86:0/2599023322 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f1d7802ff10 msgr2=0x7f1d780323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:06:40.485 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:40.484+0000 7f1d8eecf700 1 --2- 172.21.15.86:0/2599023322 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f1d7802ff10 0x7f1d780323d0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7f1d740096a0 tx=0x7f1d74006b40).stop 2024-02-19T22:06:40.485 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:40.484+0000 7f1d8eecf700 1 -- 172.21.15.86:0/2599023322 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1d880f5410 msgr2=0x7f1d880fd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:06:40.485 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:40.484+0000 7f1d8eecf700 1 --2- 172.21.15.86:0/2599023322 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1d880f5410 0x7f1d880fd4e0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0x7f1d7c005010 tx=0x7f1d7c00b040).stop 2024-02-19T22:06:40.485 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:40.484+0000 7f1d8eecf700 1 -- 172.21.15.86:0/2599023322 shutdown_connections 2024-02-19T22:06:40.485 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:40.484+0000 7f1d8eecf700 1 --2- 172.21.15.86:0/2599023322 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f1d7802ff10 0x7f1d780323d0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:06:40.485 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:40.484+0000 7f1d8eecf700 1 --2- 172.21.15.86:0/2599023322 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1d880f5410 0x7f1d880fd4e0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:06:40.485 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:40.484+0000 7f1d8eecf700 1 -- 172.21.15.86:0/2599023322 >> 172.21.15.86:0/2599023322 conn(0x7f1d880f0b70 msgr2=0x7f1d8805fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:06:40.486 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:40.484+0000 7f1d8eecf700 1 -- 172.21.15.86:0/2599023322 shutdown_connections 2024-02-19T22:06:40.486 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:40.484+0000 7f1d8eecf700 1 -- 172.21.15.86:0/2599023322 wait complete. 2024-02-19T22:06:40.486 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T22:06:40.794 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:06:40 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:06:39.530463+0000 mgr.smithi053.fyhcdi (mgr.14162) 311 : cluster [DBG] pgmap v296: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:06:40.794 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:06:40 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:06:40.483553+0000 mon.smithi053 (mon.0) 300 : audit [DBG] from='client.? 172.21.15.86:0/2599023322' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:06:42.206 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:06:42.206 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:06:42.294 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:06:41 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:06:40.928495+0000 mon.smithi053 (mon.0) 301 : audit [INF] from='mgr.14162 172.21.15.53:0/2860358250' entity='mgr.smithi053.fyhcdi' 2024-02-19T22:06:42.294 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:06:41 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:06:41.530739+0000 mgr.smithi053.fyhcdi (mgr.14162) 312 : cluster [DBG] pgmap v297: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:06:43.295 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:06:42 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:06:41.956584+0000 mon.smithi053 (mon.0) 302 : audit [INF] from='mgr.14162 172.21.15.53:0/2860358250' entity='mgr.smithi053.fyhcdi' 2024-02-19T22:06:43.296 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:06:42 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:06:41.957410+0000 mgr.smithi053.fyhcdi (mgr.14162) 313 : cluster [DBG] pgmap v298: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:06:43.296 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:06:42 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:06:41.959437+0000 mon.smithi053 (mon.0) 303 : audit [INF] from='mgr.14162 172.21.15.53:0/2860358250' entity='mgr.smithi053.fyhcdi' 2024-02-19T22:06:43.296 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:06:42 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cephadm 2024-02-19T22:06:41.961543+0000 mgr.smithi053.fyhcdi (mgr.14162) 314 : cephadm [INF] Deploying daemon alertmanager.smithi086 on smithi086 2024-02-19T22:06:44.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:44.098+0000 7f1f4e770700 1 -- 172.21.15.86:0/4201222736 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1f480f5410 msgr2=0x7f1f480f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:06:44.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:44.098+0000 7f1f4e770700 1 --2- 172.21.15.86:0/4201222736 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1f480f5410 0x7f1f480f5830 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 rx=0x7f1f3c009fb0 tx=0x7f1f3c009b30).stop 2024-02-19T22:06:44.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:44.098+0000 7f1f4e770700 1 -- 172.21.15.86:0/4201222736 shutdown_connections 2024-02-19T22:06:44.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:44.098+0000 7f1f4e770700 1 --2- 172.21.15.86:0/4201222736 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1f480f5410 0x7f1f480f5830 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:06:44.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:44.098+0000 7f1f4e770700 1 -- 172.21.15.86:0/4201222736 >> 172.21.15.86:0/4201222736 conn(0x7f1f480f0b70 msgr2=0x7f1f480f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:06:44.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:44.098+0000 7f1f4e770700 1 -- 172.21.15.86:0/4201222736 shutdown_connections 2024-02-19T22:06:44.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:44.098+0000 7f1f4e770700 1 -- 172.21.15.86:0/4201222736 wait complete. 2024-02-19T22:06:44.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:44.098+0000 7f1f4e770700 1 Processor -- start 2024-02-19T22:06:44.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:44.099+0000 7f1f4e770700 1 -- start start 2024-02-19T22:06:44.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:44.099+0000 7f1f4e770700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1f480f5410 0x7f1f481056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:06:44.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:44.099+0000 7f1f4e770700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f48105bd0 con 0x7f1f480f5410 2024-02-19T22:06:44.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:44.099+0000 7f1f47fff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1f480f5410 0x7f1f481056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:06:44.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:44.099+0000 7f1f47fff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1f480f5410 0x7f1f481056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:33012/0 (socket says 172.21.15.86:33012) 2024-02-19T22:06:44.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:44.099+0000 7f1f47fff700 1 -- 172.21.15.86:0/2765184424 learned_addr learned my addr 172.21.15.86:0/2765184424 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T22:06:44.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:44.100+0000 7f1f47fff700 1 -- 172.21.15.86:0/2765184424 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f3c005040 con 0x7f1f480f5410 2024-02-19T22:06:44.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:44.100+0000 7f1f47fff700 1 --2- 172.21.15.86:0/2765184424 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1f480f5410 0x7f1f481056e0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0x7f1f3c005010 tx=0x7f1f3c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:06:44.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:44.100+0000 7f1f44ff9700 1 -- 172.21.15.86:0/2765184424 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1f3c009320 con 0x7f1f480f5410 2024-02-19T22:06:44.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:44.100+0000 7f1f4e770700 1 -- 172.21.15.86:0/2765184424 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f48105dd0 con 0x7f1f480f5410 2024-02-19T22:06:44.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:44.100+0000 7f1f44ff9700 1 -- 172.21.15.86:0/2765184424 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1f3c013380 con 0x7f1f480f5410 2024-02-19T22:06:44.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:44.100+0000 7f1f44ff9700 1 -- 172.21.15.86:0/2765184424 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1f3c014e50 con 0x7f1f480f5410 2024-02-19T22:06:44.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:44.100+0000 7f1f4e770700 1 -- 172.21.15.86:0/2765184424 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f48106260 con 0x7f1f480f5410 2024-02-19T22:06:44.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:44.101+0000 7f1f44ff9700 1 -- 172.21.15.86:0/2765184424 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1f3c014420 con 0x7f1f480f5410 2024-02-19T22:06:44.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:44.101+0000 7f1f4e770700 1 -- 172.21.15.86:0/2765184424 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f48045ad0 con 0x7f1f480f5410 2024-02-19T22:06:44.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:44.101+0000 7f1f44ff9700 1 --2- 172.21.15.86:0/2765184424 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f1f3002ff10 0x7f1f300323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:06:44.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:44.101+0000 7f1f44ff9700 1 -- 172.21.15.86:0/2765184424 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1f3c042c20 con 0x7f1f480f5410 2024-02-19T22:06:44.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:44.101+0000 7f1f477fe700 1 --2- 172.21.15.86:0/2765184424 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f1f3002ff10 0x7f1f300323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:06:44.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:44.101+0000 7f1f477fe700 1 --2- 172.21.15.86:0/2765184424 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f1f3002ff10 0x7f1f300323d0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f1f380096a0 tx=0x7f1f38006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:06:44.106 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:44.106+0000 7f1f44ff9700 1 -- 172.21.15.86:0/2765184424 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1f3c0095f0 con 0x7f1f480f5410 2024-02-19T22:06:44.294 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:06:43 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:06:42.955404+0000 mon.smithi053 (mon.0) 304 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-02-19T22:06:44.309 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:44.309+0000 7f1f4e770700 1 -- 172.21.15.86:0/2765184424 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1f48107110 con 0x7f1f480f5410 2024-02-19T22:06:44.310 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:44.311+0000 7f1f44ff9700 1 -- 172.21.15.86:0/2765184424 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1f3c01fb70 con 0x7f1f480f5410 2024-02-19T22:06:44.310 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T22:06:44.310 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:06:44.312 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:44.311+0000 7f1f4e770700 1 -- 172.21.15.86:0/2765184424 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f1f3002ff10 msgr2=0x7f1f300323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:06:44.312 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:44.311+0000 7f1f4e770700 1 --2- 172.21.15.86:0/2765184424 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f1f3002ff10 0x7f1f300323d0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f1f380096a0 tx=0x7f1f38006b40).stop 2024-02-19T22:06:44.312 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:44.311+0000 7f1f4e770700 1 -- 172.21.15.86:0/2765184424 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1f480f5410 msgr2=0x7f1f481056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:06:44.312 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:44.311+0000 7f1f4e770700 1 --2- 172.21.15.86:0/2765184424 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1f480f5410 0x7f1f481056e0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0x7f1f3c005010 tx=0x7f1f3c00b040).stop 2024-02-19T22:06:44.312 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:44.312+0000 7f1f4e770700 1 -- 172.21.15.86:0/2765184424 shutdown_connections 2024-02-19T22:06:44.312 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:44.312+0000 7f1f4e770700 1 --2- 172.21.15.86:0/2765184424 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f1f3002ff10 0x7f1f300323d0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:06:44.312 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:44.312+0000 7f1f4e770700 1 --2- 172.21.15.86:0/2765184424 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1f480f5410 0x7f1f481056e0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:06:44.312 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:44.312+0000 7f1f4e770700 1 -- 172.21.15.86:0/2765184424 >> 172.21.15.86:0/2765184424 conn(0x7f1f480f0b70 msgr2=0x7f1f480f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:06:44.312 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:44.312+0000 7f1f4e770700 1 -- 172.21.15.86:0/2765184424 shutdown_connections 2024-02-19T22:06:44.312 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:44.312+0000 7f1f4e770700 1 -- 172.21.15.86:0/2765184424 wait complete. 2024-02-19T22:06:44.313 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T22:06:45.293 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:06:44 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:06:43.957715+0000 mgr.smithi053.fyhcdi (mgr.14162) 315 : cluster [DBG] pgmap v299: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:06:45.294 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:06:44 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:06:44.311282+0000 mon.smithi053 (mon.0) 305 : audit [DBG] from='client.? 172.21.15.86:0/2765184424' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:06:45.921 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:06:45.921 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:06:47.294 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:06:46 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:06:45.958014+0000 mgr.smithi053.fyhcdi (mgr.14162) 316 : cluster [DBG] pgmap v300: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:06:47.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:47.846+0000 7f5cf3879700 1 -- 172.21.15.86:0/3294661101 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5cec0f5410 msgr2=0x7f5cec0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:06:47.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:47.846+0000 7f5cf3879700 1 --2- 172.21.15.86:0/3294661101 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5cec0f5410 0x7f5cec0f5830 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 rx=0x7f5ce4009fb0 tx=0x7f5ce4009b30).stop 2024-02-19T22:06:47.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:47.846+0000 7f5cf3879700 1 -- 172.21.15.86:0/3294661101 shutdown_connections 2024-02-19T22:06:47.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:47.846+0000 7f5cf3879700 1 --2- 172.21.15.86:0/3294661101 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5cec0f5410 0x7f5cec0f5830 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:06:47.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:47.846+0000 7f5cf3879700 1 -- 172.21.15.86:0/3294661101 >> 172.21.15.86:0/3294661101 conn(0x7f5cec0f0b70 msgr2=0x7f5cec0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:06:47.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:47.846+0000 7f5cf3879700 1 -- 172.21.15.86:0/3294661101 shutdown_connections 2024-02-19T22:06:47.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:47.846+0000 7f5cf3879700 1 -- 172.21.15.86:0/3294661101 wait complete. 2024-02-19T22:06:47.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:47.847+0000 7f5cf3879700 1 Processor -- start 2024-02-19T22:06:47.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:47.847+0000 7f5cf3879700 1 -- start start 2024-02-19T22:06:47.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:47.847+0000 7f5cf3879700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5cec0f5410 0x7f5cec1056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:06:47.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:47.847+0000 7f5cf3879700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cec105bd0 con 0x7f5cec0f5410 2024-02-19T22:06:47.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:47.847+0000 7f5cf1615700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5cec0f5410 0x7f5cec1056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:06:47.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:47.847+0000 7f5cf1615700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5cec0f5410 0x7f5cec1056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:33038/0 (socket says 172.21.15.86:33038) 2024-02-19T22:06:47.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:47.847+0000 7f5cf1615700 1 -- 172.21.15.86:0/1406307056 learned_addr learned my addr 172.21.15.86:0/1406307056 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T22:06:47.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:47.847+0000 7f5cf1615700 1 -- 172.21.15.86:0/1406307056 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ce4005040 con 0x7f5cec0f5410 2024-02-19T22:06:47.850 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:47.848+0000 7f5cf1615700 1 --2- 172.21.15.86:0/1406307056 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5cec0f5410 0x7f5cec1056e0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 rx=0x7f5ce4005010 tx=0x7f5ce400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:06:47.850 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:47.848+0000 7f5ce9ffb700 1 -- 172.21.15.86:0/1406307056 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5ce4009320 con 0x7f5cec0f5410 2024-02-19T22:06:47.850 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:47.848+0000 7f5cf3879700 1 -- 172.21.15.86:0/1406307056 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5cec105dd0 con 0x7f5cec0f5410 2024-02-19T22:06:47.850 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:47.848+0000 7f5cf3879700 1 -- 172.21.15.86:0/1406307056 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5cec106260 con 0x7f5cec0f5410 2024-02-19T22:06:47.850 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:47.848+0000 7f5ce9ffb700 1 -- 172.21.15.86:0/1406307056 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5ce4013380 con 0x7f5cec0f5410 2024-02-19T22:06:47.850 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:47.848+0000 7f5ce9ffb700 1 -- 172.21.15.86:0/1406307056 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5ce4014df0 con 0x7f5cec0f5410 2024-02-19T22:06:47.850 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:47.849+0000 7f5ce9ffb700 1 -- 172.21.15.86:0/1406307056 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5ce4014420 con 0x7f5cec0f5410 2024-02-19T22:06:47.850 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:47.849+0000 7f5cf3879700 1 -- 172.21.15.86:0/1406307056 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5cec045ad0 con 0x7f5cec0f5410 2024-02-19T22:06:47.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:47.849+0000 7f5ce9ffb700 1 --2- 172.21.15.86:0/1406307056 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f5cd402ff60 0x7f5cd4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:06:47.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:47.849+0000 7f5ce9ffb700 1 -- 172.21.15.86:0/1406307056 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5ce4041100 con 0x7f5cec0f5410 2024-02-19T22:06:47.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:47.850+0000 7f5cf0e14700 1 --2- 172.21.15.86:0/1406307056 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f5cd402ff60 0x7f5cd4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:06:47.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:47.850+0000 7f5cf0e14700 1 --2- 172.21.15.86:0/1406307056 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f5cd402ff60 0x7f5cd4032420 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f5ce00096a0 tx=0x7f5ce0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:06:47.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:47.854+0000 7f5ce9ffb700 1 -- 172.21.15.86:0/1406307056 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5ce4014790 con 0x7f5cec0f5410 2024-02-19T22:06:48.060 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:48.060+0000 7f5cf3879700 1 -- 172.21.15.86:0/1406307056 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5cec107110 con 0x7f5cec0f5410 2024-02-19T22:06:48.061 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:48.061+0000 7f5ce9ffb700 1 -- 172.21.15.86:0/1406307056 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5ce400f990 con 0x7f5cec0f5410 2024-02-19T22:06:48.061 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T22:06:48.061 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:06:48.062 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:48.062+0000 7f5cf3879700 1 -- 172.21.15.86:0/1406307056 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f5cd402ff60 msgr2=0x7f5cd4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:06:48.062 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:48.062+0000 7f5cf3879700 1 --2- 172.21.15.86:0/1406307056 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f5cd402ff60 0x7f5cd4032420 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f5ce00096a0 tx=0x7f5ce0006b40).stop 2024-02-19T22:06:48.063 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:48.062+0000 7f5cf3879700 1 -- 172.21.15.86:0/1406307056 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5cec0f5410 msgr2=0x7f5cec1056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:06:48.063 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:48.062+0000 7f5cf3879700 1 --2- 172.21.15.86:0/1406307056 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5cec0f5410 0x7f5cec1056e0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 rx=0x7f5ce4005010 tx=0x7f5ce400b040).stop 2024-02-19T22:06:48.063 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:48.062+0000 7f5cf3879700 1 -- 172.21.15.86:0/1406307056 shutdown_connections 2024-02-19T22:06:48.063 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:48.062+0000 7f5cf3879700 1 --2- 172.21.15.86:0/1406307056 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f5cd402ff60 0x7f5cd4032420 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:06:48.063 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:48.062+0000 7f5cf3879700 1 --2- 172.21.15.86:0/1406307056 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5cec0f5410 0x7f5cec1056e0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:06:48.063 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:48.062+0000 7f5cf3879700 1 -- 172.21.15.86:0/1406307056 >> 172.21.15.86:0/1406307056 conn(0x7f5cec0f0b70 msgr2=0x7f5cec0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:06:48.063 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:48.062+0000 7f5cf3879700 1 -- 172.21.15.86:0/1406307056 shutdown_connections 2024-02-19T22:06:48.063 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:48.062+0000 7f5cf3879700 1 -- 172.21.15.86:0/1406307056 wait complete. 2024-02-19T22:06:48.063 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T22:06:49.294 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:06:48 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:06:47.958259+0000 mgr.smithi053.fyhcdi (mgr.14162) 317 : cluster [DBG] pgmap v301: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:06:49.294 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:06:48 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:06:48.062137+0000 mon.smithi053 (mon.0) 306 : audit [DBG] from='client.? 172.21.15.86:0/1406307056' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:06:49.725 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:06:49.725 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:06:51.574 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:51.574+0000 7f8571d5b700 1 -- 172.21.15.86:0/1410597457 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f856c0601e0 msgr2=0x7f856c0f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:06:51.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:51.574+0000 7f8571d5b700 1 --2- 172.21.15.86:0/1410597457 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f856c0601e0 0x7f856c0f8f20 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 rx=0x7f8560009fb0 tx=0x7f8560009b30).stop 2024-02-19T22:06:51.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:51.574+0000 7f8571d5b700 1 -- 172.21.15.86:0/1410597457 shutdown_connections 2024-02-19T22:06:51.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:51.574+0000 7f8571d5b700 1 --2- 172.21.15.86:0/1410597457 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f856c0601e0 0x7f856c0f8f20 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:06:51.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:51.574+0000 7f8571d5b700 1 -- 172.21.15.86:0/1410597457 >> 172.21.15.86:0/1410597457 conn(0x7f856c0f09a0 msgr2=0x7f856c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:06:51.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:51.574+0000 7f8571d5b700 1 -- 172.21.15.86:0/1410597457 shutdown_connections 2024-02-19T22:06:51.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:51.574+0000 7f8571d5b700 1 -- 172.21.15.86:0/1410597457 wait complete. 2024-02-19T22:06:51.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:51.574+0000 7f8571d5b700 1 Processor -- start 2024-02-19T22:06:51.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:51.574+0000 7f8571d5b700 1 -- start start 2024-02-19T22:06:51.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:51.575+0000 7f8571d5b700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f856c0601e0 0x7f856c1065e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:06:51.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:51.575+0000 7f8571d5b700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f856c106ad0 con 0x7f856c0601e0 2024-02-19T22:06:51.577 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:51.575+0000 7f8570d59700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f856c0601e0 0x7f856c1065e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:06:51.577 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:51.575+0000 7f8570d59700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f856c0601e0 0x7f856c1065e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:32840/0 (socket says 172.21.15.86:32840) 2024-02-19T22:06:51.577 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:51.575+0000 7f8570d59700 1 -- 172.21.15.86:0/3066448402 learned_addr learned my addr 172.21.15.86:0/3066448402 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T22:06:51.577 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:51.575+0000 7f8570d59700 1 -- 172.21.15.86:0/3066448402 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8560005040 con 0x7f856c0601e0 2024-02-19T22:06:51.577 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:51.575+0000 7f8570d59700 1 --2- 172.21.15.86:0/3066448402 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f856c0601e0 0x7f856c1065e0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0x7f85600089f0 tx=0x7f856000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:06:51.577 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:51.576+0000 7f85697fa700 1 -- 172.21.15.86:0/3066448402 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8560009320 con 0x7f856c0601e0 2024-02-19T22:06:51.577 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:51.576+0000 7f8571d5b700 1 -- 172.21.15.86:0/3066448402 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f856c106cd0 con 0x7f856c0601e0 2024-02-19T22:06:51.578 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:51.576+0000 7f85697fa700 1 -- 172.21.15.86:0/3066448402 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8560013380 con 0x7f856c0601e0 2024-02-19T22:06:51.578 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:51.576+0000 7f85697fa700 1 -- 172.21.15.86:0/3066448402 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f856001ed80 con 0x7f856c0601e0 2024-02-19T22:06:51.578 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:51.576+0000 7f8571d5b700 1 -- 172.21.15.86:0/3066448402 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f856c107160 con 0x7f856c0601e0 2024-02-19T22:06:51.578 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:51.576+0000 7f85697fa700 1 -- 172.21.15.86:0/3066448402 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f856001e420 con 0x7f856c0601e0 2024-02-19T22:06:51.578 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:51.577+0000 7f8571d5b700 1 -- 172.21.15.86:0/3066448402 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f856c045ad0 con 0x7f856c0601e0 2024-02-19T22:06:51.578 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:51.577+0000 7f85697fa700 1 --2- 172.21.15.86:0/3066448402 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f855c02ff60 0x7f855c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:06:51.578 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:51.577+0000 7f85697fa700 1 -- 172.21.15.86:0/3066448402 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8560041010 con 0x7f856c0601e0 2024-02-19T22:06:51.578 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:51.577+0000 7f856bfff700 1 --2- 172.21.15.86:0/3066448402 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f855c02ff60 0x7f855c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:06:51.578 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:51.577+0000 7f856bfff700 1 --2- 172.21.15.86:0/3066448402 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f855c02ff60 0x7f855c032420 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f85580096a0 tx=0x7f8558006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:06:51.581 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:51.581+0000 7f85697fa700 1 -- 172.21.15.86:0/3066448402 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f85600095f0 con 0x7f856c0601e0 2024-02-19T22:06:51.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:51.787+0000 7f8571d5b700 1 -- 172.21.15.86:0/3066448402 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f856c108010 con 0x7f856c0601e0 2024-02-19T22:06:51.788 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:51.788+0000 7f85697fa700 1 -- 172.21.15.86:0/3066448402 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f856002c2d0 con 0x7f856c0601e0 2024-02-19T22:06:51.788 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T22:06:51.788 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:06:51.790 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:51.789+0000 7f8571d5b700 1 -- 172.21.15.86:0/3066448402 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f855c02ff60 msgr2=0x7f855c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:06:51.790 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:51.789+0000 7f8571d5b700 1 --2- 172.21.15.86:0/3066448402 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f855c02ff60 0x7f855c032420 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f85580096a0 tx=0x7f8558006b40).stop 2024-02-19T22:06:51.790 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:51.789+0000 7f8571d5b700 1 -- 172.21.15.86:0/3066448402 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f856c0601e0 msgr2=0x7f856c1065e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:06:51.790 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:51.789+0000 7f8571d5b700 1 --2- 172.21.15.86:0/3066448402 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f856c0601e0 0x7f856c1065e0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0x7f85600089f0 tx=0x7f856000b040).stop 2024-02-19T22:06:51.790 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:51.789+0000 7f8571d5b700 1 -- 172.21.15.86:0/3066448402 shutdown_connections 2024-02-19T22:06:51.790 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:51.789+0000 7f8571d5b700 1 --2- 172.21.15.86:0/3066448402 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f855c02ff60 0x7f855c032420 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:06:51.790 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:51.789+0000 7f8571d5b700 1 --2- 172.21.15.86:0/3066448402 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f856c0601e0 0x7f856c1065e0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:06:51.790 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:51.789+0000 7f8571d5b700 1 -- 172.21.15.86:0/3066448402 >> 172.21.15.86:0/3066448402 conn(0x7f856c0f09a0 msgr2=0x7f856c0f13c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:06:51.790 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:51.790+0000 7f8571d5b700 1 -- 172.21.15.86:0/3066448402 shutdown_connections 2024-02-19T22:06:51.790 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:51.790+0000 7f8571d5b700 1 -- 172.21.15.86:0/3066448402 wait complete. 2024-02-19T22:06:51.791 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T22:06:51.793 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:06:51 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:06:49.958579+0000 mgr.smithi053.fyhcdi (mgr.14162) 318 : cluster [DBG] pgmap v302: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:06:52.794 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:06:52 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:06:51.789167+0000 mon.smithi053 (mon.0) 307 : audit [DBG] from='client.? 172.21.15.86:0/3066448402' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:06:53.404 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:06:53.404 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:06:53.794 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:06:53 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:06:51.958972+0000 mgr.smithi053.fyhcdi (mgr.14162) 319 : cluster [DBG] pgmap v303: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:06:55.344 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:55.342+0000 7fa1cfd33700 1 -- 172.21.15.86:0/3455270623 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa1c80f5410 msgr2=0x7fa1c80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:06:55.344 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:55.342+0000 7fa1cfd33700 1 --2- 172.21.15.86:0/3455270623 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa1c80f5410 0x7fa1c80f5830 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 rx=0x7fa1bc009fb0 tx=0x7fa1bc009b30).stop 2024-02-19T22:06:55.345 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:55.342+0000 7fa1cfd33700 1 -- 172.21.15.86:0/3455270623 shutdown_connections 2024-02-19T22:06:55.345 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:55.342+0000 7fa1cfd33700 1 --2- 172.21.15.86:0/3455270623 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa1c80f5410 0x7fa1c80f5830 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:06:55.345 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:55.342+0000 7fa1cfd33700 1 -- 172.21.15.86:0/3455270623 >> 172.21.15.86:0/3455270623 conn(0x7fa1c80f0b70 msgr2=0x7fa1c80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:06:55.345 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:55.342+0000 7fa1cfd33700 1 -- 172.21.15.86:0/3455270623 shutdown_connections 2024-02-19T22:06:55.345 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:55.342+0000 7fa1cfd33700 1 -- 172.21.15.86:0/3455270623 wait complete. 2024-02-19T22:06:55.345 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:55.343+0000 7fa1cfd33700 1 Processor -- start 2024-02-19T22:06:55.345 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:55.343+0000 7fa1cfd33700 1 -- start start 2024-02-19T22:06:55.345 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:55.343+0000 7fa1cfd33700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa1c80f5410 0x7fa1c81056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:06:55.345 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:55.343+0000 7fa1cfd33700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1c8105bd0 con 0x7fa1c80f5410 2024-02-19T22:06:55.345 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:55.343+0000 7fa1cdacf700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa1c80f5410 0x7fa1c81056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:06:55.345 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:55.343+0000 7fa1cdacf700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa1c80f5410 0x7fa1c81056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:32850/0 (socket says 172.21.15.86:32850) 2024-02-19T22:06:55.345 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:55.343+0000 7fa1cdacf700 1 -- 172.21.15.86:0/2209477971 learned_addr learned my addr 172.21.15.86:0/2209477971 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T22:06:55.345 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:55.343+0000 7fa1cdacf700 1 -- 172.21.15.86:0/2209477971 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1bc005040 con 0x7fa1c80f5410 2024-02-19T22:06:55.345 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:55.344+0000 7fa1cdacf700 1 --2- 172.21.15.86:0/2209477971 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa1c80f5410 0x7fa1c81056e0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 rx=0x7fa1bc005010 tx=0x7fa1bc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:06:55.346 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:55.344+0000 7fa1ba7fc700 1 -- 172.21.15.86:0/2209477971 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa1bc009320 con 0x7fa1c80f5410 2024-02-19T22:06:55.346 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:55.344+0000 7fa1cfd33700 1 -- 172.21.15.86:0/2209477971 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1c8105dd0 con 0x7fa1c80f5410 2024-02-19T22:06:55.346 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:55.344+0000 7fa1ba7fc700 1 -- 172.21.15.86:0/2209477971 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa1bc013380 con 0x7fa1c80f5410 2024-02-19T22:06:55.346 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:55.344+0000 7fa1ba7fc700 1 -- 172.21.15.86:0/2209477971 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa1bc014e50 con 0x7fa1c80f5410 2024-02-19T22:06:55.346 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:55.344+0000 7fa1cfd33700 1 -- 172.21.15.86:0/2209477971 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1c8106260 con 0x7fa1c80f5410 2024-02-19T22:06:55.347 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:55.345+0000 7fa1ba7fc700 1 -- 172.21.15.86:0/2209477971 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa1bc0095f0 con 0x7fa1c80f5410 2024-02-19T22:06:55.347 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:55.345+0000 7fa1cfd33700 1 -- 172.21.15.86:0/2209477971 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1c8045ad0 con 0x7fa1c80f5410 2024-02-19T22:06:55.347 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:55.345+0000 7fa1ba7fc700 1 --2- 172.21.15.86:0/2209477971 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fa1b402ff10 0x7fa1b40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:06:55.347 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:55.345+0000 7fa1ba7fc700 1 -- 172.21.15.86:0/2209477971 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa1bc042d80 con 0x7fa1c80f5410 2024-02-19T22:06:55.347 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:55.345+0000 7fa1cd2ce700 1 --2- 172.21.15.86:0/2209477971 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fa1b402ff10 0x7fa1b40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:06:55.347 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:55.345+0000 7fa1cd2ce700 1 --2- 172.21.15.86:0/2209477971 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fa1b402ff10 0x7fa1b40323d0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7fa1c40096a0 tx=0x7fa1c4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:06:55.350 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:55.350+0000 7fa1ba7fc700 1 -- 172.21.15.86:0/2209477971 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa1bc014420 con 0x7fa1c80f5410 2024-02-19T22:06:55.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:55.553+0000 7fa1cfd33700 1 -- 172.21.15.86:0/2209477971 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa1c8107110 con 0x7fa1c80f5410 2024-02-19T22:06:55.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:55.554+0000 7fa1ba7fc700 1 -- 172.21.15.86:0/2209477971 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa1bc011290 con 0x7fa1c80f5410 2024-02-19T22:06:55.554 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T22:06:55.555 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:06:55.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:55.555+0000 7fa1cfd33700 1 -- 172.21.15.86:0/2209477971 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fa1b402ff10 msgr2=0x7fa1b40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:06:55.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:55.555+0000 7fa1cfd33700 1 --2- 172.21.15.86:0/2209477971 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fa1b402ff10 0x7fa1b40323d0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7fa1c40096a0 tx=0x7fa1c4006b40).stop 2024-02-19T22:06:55.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:55.555+0000 7fa1cfd33700 1 -- 172.21.15.86:0/2209477971 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa1c80f5410 msgr2=0x7fa1c81056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:06:55.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:55.555+0000 7fa1cfd33700 1 --2- 172.21.15.86:0/2209477971 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa1c80f5410 0x7fa1c81056e0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 rx=0x7fa1bc005010 tx=0x7fa1bc00b040).stop 2024-02-19T22:06:55.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:55.556+0000 7fa1cfd33700 1 -- 172.21.15.86:0/2209477971 shutdown_connections 2024-02-19T22:06:55.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:55.556+0000 7fa1cfd33700 1 --2- 172.21.15.86:0/2209477971 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fa1b402ff10 0x7fa1b40323d0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:06:55.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:55.556+0000 7fa1cfd33700 1 --2- 172.21.15.86:0/2209477971 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa1c80f5410 0x7fa1c81056e0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:06:55.557 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:55.556+0000 7fa1cfd33700 1 -- 172.21.15.86:0/2209477971 >> 172.21.15.86:0/2209477971 conn(0x7fa1c80f0b70 msgr2=0x7fa1c80f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:06:55.557 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:55.556+0000 7fa1cfd33700 1 -- 172.21.15.86:0/2209477971 shutdown_connections 2024-02-19T22:06:55.557 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:55.556+0000 7fa1cfd33700 1 -- 172.21.15.86:0/2209477971 wait complete. 2024-02-19T22:06:55.557 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T22:06:55.794 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:06:55 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:06:53.959371+0000 mgr.smithi053.fyhcdi (mgr.14162) 320 : cluster [DBG] pgmap v304: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:06:56.794 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:06:56 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:06:55.555264+0000 mon.smithi053 (mon.0) 308 : audit [DBG] from='client.? 172.21.15.86:0/2209477971' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:06:57.319 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:06:57.319 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:06:57.794 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:06:57 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:06:55.959752+0000 mgr.smithi053.fyhcdi (mgr.14162) 321 : cluster [DBG] pgmap v305: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:06:59.169 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:59.167+0000 7f8fd0f0b700 1 -- 172.21.15.86:0/1904989151 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8fcc0f5410 msgr2=0x7f8fcc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:06:59.169 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:59.167+0000 7f8fd0f0b700 1 --2- 172.21.15.86:0/1904989151 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8fcc0f5410 0x7f8fcc0f5830 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 rx=0x7f8fc0009fb0 tx=0x7f8fc0009b30).stop 2024-02-19T22:06:59.169 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:59.167+0000 7f8fd0f0b700 1 -- 172.21.15.86:0/1904989151 shutdown_connections 2024-02-19T22:06:59.169 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:59.167+0000 7f8fd0f0b700 1 --2- 172.21.15.86:0/1904989151 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8fcc0f5410 0x7f8fcc0f5830 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:06:59.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:59.167+0000 7f8fd0f0b700 1 -- 172.21.15.86:0/1904989151 >> 172.21.15.86:0/1904989151 conn(0x7f8fcc0f0b70 msgr2=0x7f8fcc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:06:59.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:59.167+0000 7f8fd0f0b700 1 -- 172.21.15.86:0/1904989151 shutdown_connections 2024-02-19T22:06:59.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:59.167+0000 7f8fd0f0b700 1 -- 172.21.15.86:0/1904989151 wait complete. 2024-02-19T22:06:59.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:59.168+0000 7f8fd0f0b700 1 Processor -- start 2024-02-19T22:06:59.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:59.168+0000 7f8fd0f0b700 1 -- start start 2024-02-19T22:06:59.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:59.168+0000 7f8fd0f0b700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8fcc0f5410 0x7f8fcc1056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:06:59.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:59.168+0000 7f8fd0f0b700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fcc105bd0 con 0x7f8fcc0f5410 2024-02-19T22:06:59.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:59.168+0000 7f8fca59c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8fcc0f5410 0x7f8fcc1056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:06:59.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:59.168+0000 7f8fca59c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8fcc0f5410 0x7f8fcc1056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:32868/0 (socket says 172.21.15.86:32868) 2024-02-19T22:06:59.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:59.168+0000 7f8fca59c700 1 -- 172.21.15.86:0/1997504578 learned_addr learned my addr 172.21.15.86:0/1997504578 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T22:06:59.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:59.169+0000 7f8fca59c700 1 -- 172.21.15.86:0/1997504578 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fc0005040 con 0x7f8fcc0f5410 2024-02-19T22:06:59.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:59.169+0000 7f8fca59c700 1 --2- 172.21.15.86:0/1997504578 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8fcc0f5410 0x7f8fcc1056e0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0x7f8fc0005010 tx=0x7f8fc000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:06:59.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:59.169+0000 7f8fbaffd700 1 -- 172.21.15.86:0/1997504578 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8fc0009320 con 0x7f8fcc0f5410 2024-02-19T22:06:59.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:59.169+0000 7f8fbaffd700 1 -- 172.21.15.86:0/1997504578 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8fc0013380 con 0x7f8fcc0f5410 2024-02-19T22:06:59.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:59.169+0000 7f8fd0f0b700 1 -- 172.21.15.86:0/1997504578 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fcc105dd0 con 0x7f8fcc0f5410 2024-02-19T22:06:59.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:59.169+0000 7f8fbaffd700 1 -- 172.21.15.86:0/1997504578 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8fc0014e50 con 0x7f8fcc0f5410 2024-02-19T22:06:59.172 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:59.169+0000 7f8fd0f0b700 1 -- 172.21.15.86:0/1997504578 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fcc106260 con 0x7f8fcc0f5410 2024-02-19T22:06:59.172 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:59.170+0000 7f8fbaffd700 1 -- 172.21.15.86:0/1997504578 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8fc00095f0 con 0x7f8fcc0f5410 2024-02-19T22:06:59.172 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:59.170+0000 7f8fd0f0b700 1 -- 172.21.15.86:0/1997504578 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fcc045ad0 con 0x7f8fcc0f5410 2024-02-19T22:06:59.172 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:59.170+0000 7f8fbaffd700 1 --2- 172.21.15.86:0/1997504578 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f8fb402ff10 0x7f8fb40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:06:59.172 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:59.170+0000 7f8fbaffd700 1 -- 172.21.15.86:0/1997504578 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8fc0042d30 con 0x7f8fcc0f5410 2024-02-19T22:06:59.172 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:59.170+0000 7f8fc9d9b700 1 --2- 172.21.15.86:0/1997504578 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f8fb402ff10 0x7f8fb40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:06:59.172 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:59.170+0000 7f8fc9d9b700 1 --2- 172.21.15.86:0/1997504578 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f8fb402ff10 0x7f8fb40323d0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f8fbc0096a0 tx=0x7f8fbc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:06:59.173 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:59.174+0000 7f8fbaffd700 1 -- 172.21.15.86:0/1997504578 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8fc0014420 con 0x7f8fcc0f5410 2024-02-19T22:06:59.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:59.379+0000 7f8fd0f0b700 1 -- 172.21.15.86:0/1997504578 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8fcc107110 con 0x7f8fcc0f5410 2024-02-19T22:06:59.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:59.380+0000 7f8fbaffd700 1 -- 172.21.15.86:0/1997504578 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8fc001f7f0 con 0x7f8fcc0f5410 2024-02-19T22:06:59.379 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T22:06:59.380 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:06:59.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:59.381+0000 7f8fd0f0b700 1 -- 172.21.15.86:0/1997504578 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f8fb402ff10 msgr2=0x7f8fb40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:06:59.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:59.381+0000 7f8fd0f0b700 1 --2- 172.21.15.86:0/1997504578 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f8fb402ff10 0x7f8fb40323d0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f8fbc0096a0 tx=0x7f8fbc006b40).stop 2024-02-19T22:06:59.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:59.381+0000 7f8fd0f0b700 1 -- 172.21.15.86:0/1997504578 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8fcc0f5410 msgr2=0x7f8fcc1056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:06:59.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:59.381+0000 7f8fd0f0b700 1 --2- 172.21.15.86:0/1997504578 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8fcc0f5410 0x7f8fcc1056e0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0x7f8fc0005010 tx=0x7f8fc000b040).stop 2024-02-19T22:06:59.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:59.381+0000 7f8fd0f0b700 1 -- 172.21.15.86:0/1997504578 shutdown_connections 2024-02-19T22:06:59.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:59.381+0000 7f8fd0f0b700 1 --2- 172.21.15.86:0/1997504578 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f8fb402ff10 0x7f8fb40323d0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:06:59.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:59.382+0000 7f8fd0f0b700 1 --2- 172.21.15.86:0/1997504578 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8fcc0f5410 0x7f8fcc1056e0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:06:59.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:59.382+0000 7f8fd0f0b700 1 -- 172.21.15.86:0/1997504578 >> 172.21.15.86:0/1997504578 conn(0x7f8fcc0f0b70 msgr2=0x7f8fcc0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:06:59.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:59.382+0000 7f8fd0f0b700 1 -- 172.21.15.86:0/1997504578 shutdown_connections 2024-02-19T22:06:59.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:06:59.382+0000 7f8fd0f0b700 1 -- 172.21.15.86:0/1997504578 wait complete. 2024-02-19T22:06:59.382 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T22:06:59.794 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:06:59 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:06:57.960030+0000 mgr.smithi053.fyhcdi (mgr.14162) 322 : cluster [DBG] pgmap v306: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:06:59.794 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:06:59 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:06:59.380395+0000 mon.smithi053 (mon.0) 309 : audit [DBG] from='client.? 172.21.15.86:0/1997504578' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:07:01.004 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:07:01.004 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:07:01.794 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:07:01 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:06:59.960371+0000 mgr.smithi053.fyhcdi (mgr.14162) 323 : cluster [DBG] pgmap v307: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:07:03.218 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:03.217+0000 7f46dab47700 1 -- 172.21.15.86:0/1509015630 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f46d40f51a0 msgr2=0x7f46d40f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:07:03.218 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:03.217+0000 7f46dab47700 1 --2- 172.21.15.86:0/1509015630 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f46d40f51a0 0x7f46d40f55c0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 rx=0x7f46cc009960 tx=0x7f46cc004d10).stop 2024-02-19T22:07:03.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:03.218+0000 7f46dab47700 1 -- 172.21.15.86:0/1509015630 shutdown_connections 2024-02-19T22:07:03.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:03.218+0000 7f46dab47700 1 --2- 172.21.15.86:0/1509015630 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f46d40f51a0 0x7f46d40f55c0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:07:03.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:03.218+0000 7f46dab47700 1 -- 172.21.15.86:0/1509015630 >> 172.21.15.86:0/1509015630 conn(0x7f46d40f09a0 msgr2=0x7f46d40f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:07:03.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:03.218+0000 7f46dab47700 1 -- 172.21.15.86:0/1509015630 shutdown_connections 2024-02-19T22:07:03.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:03.218+0000 7f46dab47700 1 -- 172.21.15.86:0/1509015630 wait complete. 2024-02-19T22:07:03.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:03.218+0000 7f46dab47700 1 Processor -- start 2024-02-19T22:07:03.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:03.218+0000 7f46dab47700 1 -- start start 2024-02-19T22:07:03.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:03.219+0000 7f46dab47700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f46d40f51a0 0x7f46d41054d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:07:03.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:03.219+0000 7f46dab47700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46d41059c0 con 0x7f46d40f51a0 2024-02-19T22:07:03.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:03.219+0000 7f46d9b45700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f46d40f51a0 0x7f46d41054d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:07:03.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:03.219+0000 7f46d9b45700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f46d40f51a0 0x7f46d41054d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:52810/0 (socket says 172.21.15.86:52810) 2024-02-19T22:07:03.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:03.219+0000 7f46d9b45700 1 -- 172.21.15.86:0/2081794735 learned_addr learned my addr 172.21.15.86:0/2081794735 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T22:07:03.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:03.219+0000 7f46d9b45700 1 -- 172.21.15.86:0/2081794735 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46cc009640 con 0x7f46d40f51a0 2024-02-19T22:07:03.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:03.219+0000 7f46d9b45700 1 --2- 172.21.15.86:0/2081794735 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f46d40f51a0 0x7f46d41054d0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 rx=0x7f46cc007bd0 tx=0x7f46cc004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:07:03.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:03.219+0000 7f46ca7fc700 1 -- 172.21.15.86:0/2081794735 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f46cc018070 con 0x7f46d40f51a0 2024-02-19T22:07:03.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:03.219+0000 7f46dab47700 1 -- 172.21.15.86:0/2081794735 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46d4105bc0 con 0x7f46d40f51a0 2024-02-19T22:07:03.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:03.219+0000 7f46ca7fc700 1 -- 172.21.15.86:0/2081794735 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f46cc00b970 con 0x7f46d40f51a0 2024-02-19T22:07:03.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:03.219+0000 7f46ca7fc700 1 -- 172.21.15.86:0/2081794735 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f46cc01d600 con 0x7f46d40f51a0 2024-02-19T22:07:03.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:03.219+0000 7f46dab47700 1 -- 172.21.15.86:0/2081794735 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46d4106050 con 0x7f46d40f51a0 2024-02-19T22:07:03.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:03.220+0000 7f46dab47700 1 -- 172.21.15.86:0/2081794735 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46d4045ad0 con 0x7f46d40f51a0 2024-02-19T22:07:03.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:03.220+0000 7f46ca7fc700 1 -- 172.21.15.86:0/2081794735 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f46cc024060 con 0x7f46d40f51a0 2024-02-19T22:07:03.222 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:03.220+0000 7f46ca7fc700 1 --2- 172.21.15.86:0/2081794735 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f46c002ff60 0x7f46c0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:07:03.222 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:03.221+0000 7f46ca7fc700 1 -- 172.21.15.86:0/2081794735 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f46cc0411d0 con 0x7f46d40f51a0 2024-02-19T22:07:03.222 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:03.221+0000 7f46d9344700 1 --2- 172.21.15.86:0/2081794735 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f46c002ff60 0x7f46c0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:07:03.222 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:03.221+0000 7f46d9344700 1 --2- 172.21.15.86:0/2081794735 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f46c002ff60 0x7f46c0032420 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f46c40096a0 tx=0x7f46c4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:07:03.225 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:03.225+0000 7f46ca7fc700 1 -- 172.21.15.86:0/2081794735 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f46cc01bb10 con 0x7f46d40f51a0 2024-02-19T22:07:03.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:03.434+0000 7f46dab47700 1 -- 172.21.15.86:0/2081794735 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f46d4106ea0 con 0x7f46d40f51a0 2024-02-19T22:07:03.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:03.435+0000 7f46ca7fc700 1 -- 172.21.15.86:0/2081794735 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f46cc027e00 con 0x7f46d40f51a0 2024-02-19T22:07:03.435 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T22:07:03.436 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:07:03.437 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:03.437+0000 7f46dab47700 1 -- 172.21.15.86:0/2081794735 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f46c002ff60 msgr2=0x7f46c0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:07:03.437 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:03.437+0000 7f46dab47700 1 --2- 172.21.15.86:0/2081794735 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f46c002ff60 0x7f46c0032420 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f46c40096a0 tx=0x7f46c4006b40).stop 2024-02-19T22:07:03.437 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:03.437+0000 7f46dab47700 1 -- 172.21.15.86:0/2081794735 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f46d40f51a0 msgr2=0x7f46d41054d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:07:03.437 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:03.437+0000 7f46dab47700 1 --2- 172.21.15.86:0/2081794735 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f46d40f51a0 0x7f46d41054d0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 rx=0x7f46cc007bd0 tx=0x7f46cc004d10).stop 2024-02-19T22:07:03.437 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:03.437+0000 7f46dab47700 1 -- 172.21.15.86:0/2081794735 shutdown_connections 2024-02-19T22:07:03.437 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:03.437+0000 7f46dab47700 1 --2- 172.21.15.86:0/2081794735 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f46c002ff60 0x7f46c0032420 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:07:03.437 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:03.437+0000 7f46dab47700 1 --2- 172.21.15.86:0/2081794735 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f46d40f51a0 0x7f46d41054d0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:07:03.437 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:03.437+0000 7f46dab47700 1 -- 172.21.15.86:0/2081794735 >> 172.21.15.86:0/2081794735 conn(0x7f46d40f09a0 msgr2=0x7f46d40f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:07:03.437 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:03.437+0000 7f46dab47700 1 -- 172.21.15.86:0/2081794735 shutdown_connections 2024-02-19T22:07:03.438 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:03.437+0000 7f46dab47700 1 -- 172.21.15.86:0/2081794735 wait complete. 2024-02-19T22:07:03.438 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T22:07:03.794 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:07:03 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:07:01.960668+0000 mgr.smithi053.fyhcdi (mgr.14162) 324 : cluster [DBG] pgmap v308: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:07:03.794 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:07:03 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:07:03.436246+0000 mon.smithi053 (mon.0) 310 : audit [DBG] from='client.? 172.21.15.86:0/2081794735' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:07:05.122 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:07:05.122 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:07:05.794 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:07:05 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:07:03.961047+0000 mgr.smithi053.fyhcdi (mgr.14162) 325 : cluster [DBG] pgmap v309: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:07:07.080 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:07.079+0000 7f5694d75700 1 -- 172.21.15.86:0/1416004438 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f56900f6510 msgr2=0x7f56900f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:07:07.080 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:07.079+0000 7f5694d75700 1 --2- 172.21.15.86:0/1416004438 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f56900f6510 0x7f56900f6930 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 rx=0x7f5684009fb0 tx=0x7f5684009b30).stop 2024-02-19T22:07:07.080 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:07.079+0000 7f5694d75700 1 -- 172.21.15.86:0/1416004438 shutdown_connections 2024-02-19T22:07:07.080 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:07.079+0000 7f5694d75700 1 --2- 172.21.15.86:0/1416004438 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f56900f6510 0x7f56900f6930 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:07:07.080 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:07.079+0000 7f5694d75700 1 -- 172.21.15.86:0/1416004438 >> 172.21.15.86:0/1416004438 conn(0x7f56900f1c70 msgr2=0x7f56900f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:07:07.080 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:07.079+0000 7f5694d75700 1 -- 172.21.15.86:0/1416004438 shutdown_connections 2024-02-19T22:07:07.080 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:07.079+0000 7f5694d75700 1 -- 172.21.15.86:0/1416004438 wait complete. 2024-02-19T22:07:07.080 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:07.079+0000 7f5694d75700 1 Processor -- start 2024-02-19T22:07:07.080 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:07.080+0000 7f5694d75700 1 -- start start 2024-02-19T22:07:07.080 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:07.080+0000 7f5694d75700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f56900f6510 0x7f5690106810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:07:07.081 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:07.080+0000 7f5694d75700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5690106d00 con 0x7f56900f6510 2024-02-19T22:07:07.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:07.080+0000 7f568e59c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f56900f6510 0x7f5690106810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:07:07.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:07.080+0000 7f568e59c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f56900f6510 0x7f5690106810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:52820/0 (socket says 172.21.15.86:52820) 2024-02-19T22:07:07.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:07.080+0000 7f568e59c700 1 -- 172.21.15.86:0/2299858398 learned_addr learned my addr 172.21.15.86:0/2299858398 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T22:07:07.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:07.080+0000 7f568e59c700 1 -- 172.21.15.86:0/2299858398 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5684005040 con 0x7f56900f6510 2024-02-19T22:07:07.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:07.080+0000 7f568e59c700 1 --2- 172.21.15.86:0/2299858398 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f56900f6510 0x7f5690106810 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0x7f5684005010 tx=0x7f568400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:07:07.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:07.081+0000 7f567effd700 1 -- 172.21.15.86:0/2299858398 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5684009320 con 0x7f56900f6510 2024-02-19T22:07:07.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:07.081+0000 7f5694d75700 1 -- 172.21.15.86:0/2299858398 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5690106f00 con 0x7f56900f6510 2024-02-19T22:07:07.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:07.081+0000 7f567effd700 1 -- 172.21.15.86:0/2299858398 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5684013380 con 0x7f56900f6510 2024-02-19T22:07:07.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:07.081+0000 7f567effd700 1 -- 172.21.15.86:0/2299858398 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5684014e50 con 0x7f56900f6510 2024-02-19T22:07:07.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:07.081+0000 7f5694d75700 1 -- 172.21.15.86:0/2299858398 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5690107390 con 0x7f56900f6510 2024-02-19T22:07:07.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:07.081+0000 7f567effd700 1 -- 172.21.15.86:0/2299858398 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f56840263a0 con 0x7f56900f6510 2024-02-19T22:07:07.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:07.082+0000 7f567effd700 1 --2- 172.21.15.86:0/2299858398 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f567802ff10 0x7f56780323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:07:07.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:07.082+0000 7f567effd700 1 -- 172.21.15.86:0/2299858398 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5684040ec0 con 0x7f56900f6510 2024-02-19T22:07:07.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:07.082+0000 7f5694d75700 1 -- 172.21.15.86:0/2299858398 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5690045ad0 con 0x7f56900f6510 2024-02-19T22:07:07.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:07.082+0000 7f568dd9b700 1 --2- 172.21.15.86:0/2299858398 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f567802ff10 0x7f56780323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:07:07.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:07.082+0000 7f568dd9b700 1 --2- 172.21.15.86:0/2299858398 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f567802ff10 0x7f56780323d0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f56800096a0 tx=0x7f5680006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:07:07.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:07.086+0000 7f567effd700 1 -- 172.21.15.86:0/2299858398 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5684012920 con 0x7f56900f6510 2024-02-19T22:07:07.290 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:07.290+0000 7f5694d75700 1 -- 172.21.15.86:0/2299858398 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f56901080b0 con 0x7f56900f6510 2024-02-19T22:07:07.291 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:07.291+0000 7f567effd700 1 -- 172.21.15.86:0/2299858398 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5684012330 con 0x7f56900f6510 2024-02-19T22:07:07.291 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T22:07:07.291 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:07:07.293 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:07.292+0000 7f5694d75700 1 -- 172.21.15.86:0/2299858398 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f567802ff10 msgr2=0x7f56780323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:07:07.293 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:07.292+0000 7f5694d75700 1 --2- 172.21.15.86:0/2299858398 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f567802ff10 0x7f56780323d0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f56800096a0 tx=0x7f5680006b40).stop 2024-02-19T22:07:07.293 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:07.292+0000 7f5694d75700 1 -- 172.21.15.86:0/2299858398 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f56900f6510 msgr2=0x7f5690106810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:07:07.293 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:07.292+0000 7f5694d75700 1 --2- 172.21.15.86:0/2299858398 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f56900f6510 0x7f5690106810 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0x7f5684005010 tx=0x7f568400b040).stop 2024-02-19T22:07:07.293 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:07.292+0000 7f5694d75700 1 -- 172.21.15.86:0/2299858398 shutdown_connections 2024-02-19T22:07:07.293 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:07.292+0000 7f5694d75700 1 --2- 172.21.15.86:0/2299858398 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f567802ff10 0x7f56780323d0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:07:07.293 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:07.292+0000 7f5694d75700 1 --2- 172.21.15.86:0/2299858398 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f56900f6510 0x7f5690106810 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:07:07.293 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:07.292+0000 7f5694d75700 1 -- 172.21.15.86:0/2299858398 >> 172.21.15.86:0/2299858398 conn(0x7f56900f1c70 msgr2=0x7f56900f26f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:07:07.293 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:07.292+0000 7f5694d75700 1 -- 172.21.15.86:0/2299858398 shutdown_connections 2024-02-19T22:07:07.293 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:07.292+0000 7f5694d75700 1 -- 172.21.15.86:0/2299858398 wait complete. 2024-02-19T22:07:07.294 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T22:07:07.794 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:07:07 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:07:05.961403+0000 mgr.smithi053.fyhcdi (mgr.14162) 326 : cluster [DBG] pgmap v310: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:07:07.794 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:07:07 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:07:07.292236+0000 mon.smithi053 (mon.0) 311 : audit [DBG] from='client.? 172.21.15.86:0/2299858398' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:07:08.981 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:07:08.981 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:07:10.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:07:09 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:07:07.961658+0000 mgr.smithi053.fyhcdi (mgr.14162) 327 : cluster [DBG] pgmap v311: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:07:10.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:10.952+0000 7fe254918700 1 -- 172.21.15.86:0/3678594491 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe2500f5410 msgr2=0x7fe2500f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:07:10.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:10.952+0000 7fe254918700 1 --2- 172.21.15.86:0/3678594491 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe2500f5410 0x7fe2500f5830 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 rx=0x7fe240009fb0 tx=0x7fe240009b30).stop 2024-02-19T22:07:10.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:10.952+0000 7fe254918700 1 -- 172.21.15.86:0/3678594491 shutdown_connections 2024-02-19T22:07:10.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:10.952+0000 7fe254918700 1 --2- 172.21.15.86:0/3678594491 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe2500f5410 0x7fe2500f5830 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:07:10.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:10.952+0000 7fe254918700 1 -- 172.21.15.86:0/3678594491 >> 172.21.15.86:0/3678594491 conn(0x7fe2500f0b70 msgr2=0x7fe2500f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:07:10.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:10.952+0000 7fe254918700 1 -- 172.21.15.86:0/3678594491 shutdown_connections 2024-02-19T22:07:10.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:10.952+0000 7fe254918700 1 -- 172.21.15.86:0/3678594491 wait complete. 2024-02-19T22:07:10.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:10.952+0000 7fe254918700 1 Processor -- start 2024-02-19T22:07:10.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:10.953+0000 7fe254918700 1 -- start start 2024-02-19T22:07:10.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:10.953+0000 7fe254918700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe2500f5410 0x7fe250109520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:07:10.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:10.953+0000 7fe254918700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe250060240 con 0x7fe2500f5410 2024-02-19T22:07:10.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:10.953+0000 7fe24e59c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe2500f5410 0x7fe250109520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:07:10.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:10.953+0000 7fe24e59c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe2500f5410 0x7fe250109520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:40804/0 (socket says 172.21.15.86:40804) 2024-02-19T22:07:10.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:10.953+0000 7fe24e59c700 1 -- 172.21.15.86:0/3236782067 learned_addr learned my addr 172.21.15.86:0/3236782067 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T22:07:10.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:10.953+0000 7fe24e59c700 1 -- 172.21.15.86:0/3236782067 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe240005040 con 0x7fe2500f5410 2024-02-19T22:07:10.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:10.953+0000 7fe24e59c700 1 --2- 172.21.15.86:0/3236782067 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe2500f5410 0x7fe250109520 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 rx=0x7fe240005010 tx=0x7fe24000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:07:10.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:10.954+0000 7fe23effd700 1 -- 172.21.15.86:0/3236782067 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe240009320 con 0x7fe2500f5410 2024-02-19T22:07:10.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:10.954+0000 7fe254918700 1 -- 172.21.15.86:0/3236782067 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe250060440 con 0x7fe2500f5410 2024-02-19T22:07:10.956 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:10.954+0000 7fe23effd700 1 -- 172.21.15.86:0/3236782067 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe240013380 con 0x7fe2500f5410 2024-02-19T22:07:10.956 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:10.954+0000 7fe23effd700 1 -- 172.21.15.86:0/3236782067 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe240014e50 con 0x7fe2500f5410 2024-02-19T22:07:10.956 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:10.954+0000 7fe254918700 1 -- 172.21.15.86:0/3236782067 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe250109bd0 con 0x7fe2500f5410 2024-02-19T22:07:10.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:10.954+0000 7fe23effd700 1 -- 172.21.15.86:0/3236782067 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe2400263a0 con 0x7fe2500f5410 2024-02-19T22:07:10.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:10.955+0000 7fe23effd700 1 --2- 172.21.15.86:0/3236782067 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fe23802ff10 0x7fe2380323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:07:10.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:10.955+0000 7fe23effd700 1 -- 172.21.15.86:0/3236782067 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe240041140 con 0x7fe2500f5410 2024-02-19T22:07:10.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:10.955+0000 7fe254918700 1 -- 172.21.15.86:0/3236782067 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe250045ad0 con 0x7fe2500f5410 2024-02-19T22:07:10.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:10.955+0000 7fe24dd9b700 1 --2- 172.21.15.86:0/3236782067 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fe23802ff10 0x7fe2380323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:07:10.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:10.955+0000 7fe24dd9b700 1 --2- 172.21.15.86:0/3236782067 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fe23802ff10 0x7fe2380323d0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7fe2440096a0 tx=0x7fe244006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:07:10.960 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:10.960+0000 7fe23effd700 1 -- 172.21.15.86:0/3236782067 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe240012920 con 0x7fe2500f5410 2024-02-19T22:07:11.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:11.165+0000 7fe254918700 1 -- 172.21.15.86:0/3236782067 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe25010a330 con 0x7fe2500f5410 2024-02-19T22:07:11.166 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:11.166+0000 7fe23effd700 1 -- 172.21.15.86:0/3236782067 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe240012330 con 0x7fe2500f5410 2024-02-19T22:07:11.166 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T22:07:11.166 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:07:11.168 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:11.167+0000 7fe254918700 1 -- 172.21.15.86:0/3236782067 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fe23802ff10 msgr2=0x7fe2380323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:07:11.168 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:11.167+0000 7fe254918700 1 --2- 172.21.15.86:0/3236782067 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fe23802ff10 0x7fe2380323d0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7fe2440096a0 tx=0x7fe244006b40).stop 2024-02-19T22:07:11.168 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:11.167+0000 7fe254918700 1 -- 172.21.15.86:0/3236782067 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe2500f5410 msgr2=0x7fe250109520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:07:11.168 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:11.167+0000 7fe254918700 1 --2- 172.21.15.86:0/3236782067 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe2500f5410 0x7fe250109520 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 rx=0x7fe240005010 tx=0x7fe24000b040).stop 2024-02-19T22:07:11.168 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:11.167+0000 7fe254918700 1 -- 172.21.15.86:0/3236782067 shutdown_connections 2024-02-19T22:07:11.168 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:11.167+0000 7fe254918700 1 --2- 172.21.15.86:0/3236782067 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fe23802ff10 0x7fe2380323d0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:07:11.168 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:11.167+0000 7fe254918700 1 --2- 172.21.15.86:0/3236782067 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe2500f5410 0x7fe250109520 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:07:11.168 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:11.167+0000 7fe254918700 1 -- 172.21.15.86:0/3236782067 >> 172.21.15.86:0/3236782067 conn(0x7fe2500f0b70 msgr2=0x7fe2500f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:07:11.169 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:11.167+0000 7fe254918700 1 -- 172.21.15.86:0/3236782067 shutdown_connections 2024-02-19T22:07:11.169 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:11.167+0000 7fe254918700 1 -- 172.21.15.86:0/3236782067 wait complete. 2024-02-19T22:07:11.169 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T22:07:12.046 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:07:11 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:07:09.962031+0000 mgr.smithi053.fyhcdi (mgr.14162) 328 : cluster [DBG] pgmap v312: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:07:12.046 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:07:11 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:07:11.166937+0000 mon.smithi053 (mon.0) 312 : audit [DBG] from='client.? 172.21.15.86:0/3236782067' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:07:12.890 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:07:12.891 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:07:14.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:07:13 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:07:11.962454+0000 mgr.smithi053.fyhcdi (mgr.14162) 329 : cluster [DBG] pgmap v313: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:07:14.740 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:14.738+0000 7f50fa9bb700 1 -- 172.21.15.86:0/1398703131 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f50f40f51a0 msgr2=0x7f50f40f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:07:14.740 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:14.738+0000 7f50fa9bb700 1 --2- 172.21.15.86:0/1398703131 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f50f40f51a0 0x7f50f40f55c0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 rx=0x7f50f0009fb0 tx=0x7f50f0009b30).stop 2024-02-19T22:07:14.740 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:14.738+0000 7f50fa9bb700 1 -- 172.21.15.86:0/1398703131 shutdown_connections 2024-02-19T22:07:14.740 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:14.738+0000 7f50fa9bb700 1 --2- 172.21.15.86:0/1398703131 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f50f40f51a0 0x7f50f40f55c0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:07:14.740 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:14.738+0000 7f50fa9bb700 1 -- 172.21.15.86:0/1398703131 >> 172.21.15.86:0/1398703131 conn(0x7f50f40f09a0 msgr2=0x7f50f40f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:07:14.740 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:14.738+0000 7f50fa9bb700 1 -- 172.21.15.86:0/1398703131 shutdown_connections 2024-02-19T22:07:14.741 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:14.738+0000 7f50fa9bb700 1 -- 172.21.15.86:0/1398703131 wait complete. 2024-02-19T22:07:14.741 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:14.739+0000 7f50fa9bb700 1 Processor -- start 2024-02-19T22:07:14.741 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:14.739+0000 7f50fa9bb700 1 -- start start 2024-02-19T22:07:14.741 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:14.739+0000 7f50fa9bb700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f50f40f51a0 0x7f50f4109290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:07:14.741 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:14.739+0000 7f50fa9bb700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50f4060280 con 0x7f50f40f51a0 2024-02-19T22:07:14.741 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:14.739+0000 7f50f99b9700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f50f40f51a0 0x7f50f4109290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:07:14.741 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:14.739+0000 7f50f99b9700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f50f40f51a0 0x7f50f4109290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:40822/0 (socket says 172.21.15.86:40822) 2024-02-19T22:07:14.741 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:14.739+0000 7f50f99b9700 1 -- 172.21.15.86:0/3099347462 learned_addr learned my addr 172.21.15.86:0/3099347462 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T22:07:14.741 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:14.739+0000 7f50f99b9700 1 -- 172.21.15.86:0/3099347462 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50f0005040 con 0x7f50f40f51a0 2024-02-19T22:07:14.741 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:14.739+0000 7f50f99b9700 1 --2- 172.21.15.86:0/3099347462 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f50f40f51a0 0x7f50f4109290 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0x7f50f0009830 tx=0x7f50f000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:07:14.741 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:14.740+0000 7f50ea7fc700 1 -- 172.21.15.86:0/3099347462 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f50f0009320 con 0x7f50f40f51a0 2024-02-19T22:07:14.741 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:14.740+0000 7f50fa9bb700 1 -- 172.21.15.86:0/3099347462 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50f4060480 con 0x7f50f40f51a0 2024-02-19T22:07:14.741 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:14.740+0000 7f50ea7fc700 1 -- 172.21.15.86:0/3099347462 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f50f0013380 con 0x7f50f40f51a0 2024-02-19T22:07:14.741 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:14.740+0000 7f50ea7fc700 1 -- 172.21.15.86:0/3099347462 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f50f0014420 con 0x7f50f40f51a0 2024-02-19T22:07:14.741 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:14.740+0000 7f50fa9bb700 1 -- 172.21.15.86:0/3099347462 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50f41099c0 con 0x7f50f40f51a0 2024-02-19T22:07:14.742 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:14.740+0000 7f50ea7fc700 1 -- 172.21.15.86:0/3099347462 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f50f001e420 con 0x7f50f40f51a0 2024-02-19T22:07:14.742 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:14.740+0000 7f50fa9bb700 1 -- 172.21.15.86:0/3099347462 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50f4045ad0 con 0x7f50f40f51a0 2024-02-19T22:07:14.742 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:14.741+0000 7f50ea7fc700 1 --2- 172.21.15.86:0/3099347462 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f50e002ff60 0x7f50e0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:07:14.742 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:14.741+0000 7f50ea7fc700 1 -- 172.21.15.86:0/3099347462 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f50f0040e30 con 0x7f50f40f51a0 2024-02-19T22:07:14.743 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:14.741+0000 7f50f91b8700 1 --2- 172.21.15.86:0/3099347462 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f50e002ff60 0x7f50e0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:07:14.743 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:14.741+0000 7f50f91b8700 1 --2- 172.21.15.86:0/3099347462 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f50e002ff60 0x7f50e0032420 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f50e40096a0 tx=0x7f50e4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:07:14.745 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:14.745+0000 7f50ea7fc700 1 -- 172.21.15.86:0/3099347462 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f50f001cb10 con 0x7f50f40f51a0 2024-02-19T22:07:14.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:14.950+0000 7f50fa9bb700 1 -- 172.21.15.86:0/3099347462 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f50f40fad30 con 0x7f50f40f51a0 2024-02-19T22:07:14.951 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:14.951+0000 7f50ea7fc700 1 -- 172.21.15.86:0/3099347462 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f50f0004490 con 0x7f50f40f51a0 2024-02-19T22:07:14.952 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T22:07:14.952 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:07:14.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:14.953+0000 7f50fa9bb700 1 -- 172.21.15.86:0/3099347462 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f50e002ff60 msgr2=0x7f50e0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:07:14.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:14.953+0000 7f50fa9bb700 1 --2- 172.21.15.86:0/3099347462 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f50e002ff60 0x7f50e0032420 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f50e40096a0 tx=0x7f50e4006b40).stop 2024-02-19T22:07:14.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:14.953+0000 7f50fa9bb700 1 -- 172.21.15.86:0/3099347462 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f50f40f51a0 msgr2=0x7f50f4109290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:07:14.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:14.953+0000 7f50fa9bb700 1 --2- 172.21.15.86:0/3099347462 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f50f40f51a0 0x7f50f4109290 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0x7f50f0009830 tx=0x7f50f000b040).stop 2024-02-19T22:07:14.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:14.953+0000 7f50fa9bb700 1 -- 172.21.15.86:0/3099347462 shutdown_connections 2024-02-19T22:07:14.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:14.953+0000 7f50fa9bb700 1 --2- 172.21.15.86:0/3099347462 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f50e002ff60 0x7f50e0032420 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:07:14.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:14.953+0000 7f50fa9bb700 1 --2- 172.21.15.86:0/3099347462 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f50f40f51a0 0x7f50f4109290 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:07:14.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:14.953+0000 7f50fa9bb700 1 -- 172.21.15.86:0/3099347462 >> 172.21.15.86:0/3099347462 conn(0x7f50f40f09a0 msgr2=0x7f50f40f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:07:14.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:14.953+0000 7f50fa9bb700 1 -- 172.21.15.86:0/3099347462 shutdown_connections 2024-02-19T22:07:14.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:14.953+0000 7f50fa9bb700 1 -- 172.21.15.86:0/3099347462 wait complete. 2024-02-19T22:07:14.954 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T22:07:16.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:07:15 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:07:13.962932+0000 mgr.smithi053.fyhcdi (mgr.14162) 330 : cluster [DBG] pgmap v314: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:07:16.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:07:15 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:07:14.952285+0000 mon.smithi053 (mon.0) 313 : audit [DBG] from='client.? 172.21.15.86:0/3099347462' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:07:16.746 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:07:16.746 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:07:18.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:07:17 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:07:15.963279+0000 mgr.smithi053.fyhcdi (mgr.14162) 331 : cluster [DBG] pgmap v315: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:07:18.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:18.574+0000 7f04f519e700 1 -- 172.21.15.86:0/195707960 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f04f00f51a0 msgr2=0x7f04f00f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:07:18.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:18.574+0000 7f04f519e700 1 --2- 172.21.15.86:0/195707960 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f04f00f51a0 0x7f04f00f55c0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 rx=0x7f04e0009fb0 tx=0x7f04e0009b30).stop 2024-02-19T22:07:18.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:18.574+0000 7f04f519e700 1 -- 172.21.15.86:0/195707960 shutdown_connections 2024-02-19T22:07:18.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:18.574+0000 7f04f519e700 1 --2- 172.21.15.86:0/195707960 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f04f00f51a0 0x7f04f00f55c0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:07:18.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:18.574+0000 7f04f519e700 1 -- 172.21.15.86:0/195707960 >> 172.21.15.86:0/195707960 conn(0x7f04f00f09a0 msgr2=0x7f04f00f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:07:18.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:18.574+0000 7f04f519e700 1 -- 172.21.15.86:0/195707960 shutdown_connections 2024-02-19T22:07:18.576 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:18.574+0000 7f04f519e700 1 -- 172.21.15.86:0/195707960 wait complete. 2024-02-19T22:07:18.576 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:18.574+0000 7f04f519e700 1 Processor -- start 2024-02-19T22:07:18.576 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:18.575+0000 7f04f519e700 1 -- start start 2024-02-19T22:07:18.576 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:18.575+0000 7f04f519e700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f04f00f51a0 0x7f04f01065d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:07:18.576 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:18.575+0000 7f04f519e700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04f0106ac0 con 0x7f04f00f51a0 2024-02-19T22:07:18.578 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:18.575+0000 7f04effff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f04f00f51a0 0x7f04f01065d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:07:18.578 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:18.575+0000 7f04effff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f04f00f51a0 0x7f04f01065d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:40842/0 (socket says 172.21.15.86:40842) 2024-02-19T22:07:18.578 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:18.575+0000 7f04effff700 1 -- 172.21.15.86:0/3518224068 learned_addr learned my addr 172.21.15.86:0/3518224068 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T22:07:18.578 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:18.575+0000 7f04effff700 1 -- 172.21.15.86:0/3518224068 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04e0005040 con 0x7f04f00f51a0 2024-02-19T22:07:18.578 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:18.576+0000 7f04effff700 1 --2- 172.21.15.86:0/3518224068 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f04f00f51a0 0x7f04f01065d0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 rx=0x7f04e00089f0 tx=0x7f04e000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:07:18.578 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:18.576+0000 7f04ecff9700 1 -- 172.21.15.86:0/3518224068 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f04e0009320 con 0x7f04f00f51a0 2024-02-19T22:07:18.578 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:18.576+0000 7f04f519e700 1 -- 172.21.15.86:0/3518224068 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04f0106cc0 con 0x7f04f00f51a0 2024-02-19T22:07:18.578 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:18.576+0000 7f04ecff9700 1 -- 172.21.15.86:0/3518224068 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f04e0013380 con 0x7f04f00f51a0 2024-02-19T22:07:18.578 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:18.576+0000 7f04ecff9700 1 -- 172.21.15.86:0/3518224068 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f04e0014e20 con 0x7f04f00f51a0 2024-02-19T22:07:18.578 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:18.576+0000 7f04f519e700 1 -- 172.21.15.86:0/3518224068 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04f0107150 con 0x7f04f00f51a0 2024-02-19T22:07:18.578 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:18.577+0000 7f04ecff9700 1 -- 172.21.15.86:0/3518224068 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f04e00095f0 con 0x7f04f00f51a0 2024-02-19T22:07:18.579 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:18.577+0000 7f04f519e700 1 -- 172.21.15.86:0/3518224068 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04f0045ad0 con 0x7f04f00f51a0 2024-02-19T22:07:18.579 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:18.577+0000 7f04ecff9700 1 --2- 172.21.15.86:0/3518224068 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f04dc02ff10 0x7f04dc0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:07:18.579 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:18.577+0000 7f04ecff9700 1 -- 172.21.15.86:0/3518224068 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f04e0040f10 con 0x7f04f00f51a0 2024-02-19T22:07:18.579 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:18.577+0000 7f04ef7fe700 1 --2- 172.21.15.86:0/3518224068 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f04dc02ff10 0x7f04dc0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:07:18.579 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:18.577+0000 7f04ef7fe700 1 --2- 172.21.15.86:0/3518224068 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f04dc02ff10 0x7f04dc0323d0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7f04d80096a0 tx=0x7f04d8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:07:18.581 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:18.581+0000 7f04ecff9700 1 -- 172.21.15.86:0/3518224068 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f04e0014420 con 0x7f04f00f51a0 2024-02-19T22:07:18.790 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:18.789+0000 7f04f519e700 1 -- 172.21.15.86:0/3518224068 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f04f00fad30 con 0x7f04f00f51a0 2024-02-19T22:07:18.790 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:18.790+0000 7f04ecff9700 1 -- 172.21.15.86:0/3518224068 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f04e001d980 con 0x7f04f00f51a0 2024-02-19T22:07:18.790 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T22:07:18.790 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:07:18.792 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:18.791+0000 7f04f519e700 1 -- 172.21.15.86:0/3518224068 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f04dc02ff10 msgr2=0x7f04dc0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:07:18.792 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:18.791+0000 7f04f519e700 1 --2- 172.21.15.86:0/3518224068 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f04dc02ff10 0x7f04dc0323d0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7f04d80096a0 tx=0x7f04d8006b40).stop 2024-02-19T22:07:18.792 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:18.791+0000 7f04f519e700 1 -- 172.21.15.86:0/3518224068 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f04f00f51a0 msgr2=0x7f04f01065d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:07:18.792 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:18.791+0000 7f04f519e700 1 --2- 172.21.15.86:0/3518224068 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f04f00f51a0 0x7f04f01065d0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 rx=0x7f04e00089f0 tx=0x7f04e000b040).stop 2024-02-19T22:07:18.792 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:18.792+0000 7f04f519e700 1 -- 172.21.15.86:0/3518224068 shutdown_connections 2024-02-19T22:07:18.792 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:18.792+0000 7f04f519e700 1 --2- 172.21.15.86:0/3518224068 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f04dc02ff10 0x7f04dc0323d0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:07:18.792 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:18.792+0000 7f04f519e700 1 --2- 172.21.15.86:0/3518224068 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f04f00f51a0 0x7f04f01065d0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:07:18.793 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:18.792+0000 7f04f519e700 1 -- 172.21.15.86:0/3518224068 >> 172.21.15.86:0/3518224068 conn(0x7f04f00f09a0 msgr2=0x7f04f005fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:07:18.793 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:18.792+0000 7f04f519e700 1 -- 172.21.15.86:0/3518224068 shutdown_connections 2024-02-19T22:07:18.793 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:18.792+0000 7f04f519e700 1 -- 172.21.15.86:0/3518224068 wait complete. 2024-02-19T22:07:18.793 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T22:07:20.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:07:19 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:07:17.963592+0000 mgr.smithi053.fyhcdi (mgr.14162) 332 : cluster [DBG] pgmap v316: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:07:20.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:07:19 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:07:18.791182+0000 mon.smithi053 (mon.0) 314 : audit [DBG] from='client.? 172.21.15.86:0/3518224068' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:07:20.511 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:07:20.511 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:07:22.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:07:21 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:07:19.964053+0000 mgr.smithi053.fyhcdi (mgr.14162) 333 : cluster [DBG] pgmap v317: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:07:22.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:22.364+0000 7f0737fff700 1 -- 172.21.15.86:0/228978284 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f07380f62a0 msgr2=0x7f07380f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:07:22.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:22.364+0000 7f0737fff700 1 --2- 172.21.15.86:0/228978284 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f07380f62a0 0x7f07380f66c0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 rx=0x7f0728009fb0 tx=0x7f0728009b30).stop 2024-02-19T22:07:22.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:22.364+0000 7f0737fff700 1 -- 172.21.15.86:0/228978284 shutdown_connections 2024-02-19T22:07:22.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:22.364+0000 7f0737fff700 1 --2- 172.21.15.86:0/228978284 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f07380f62a0 0x7f07380f66c0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:07:22.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:22.364+0000 7f0737fff700 1 -- 172.21.15.86:0/228978284 >> 172.21.15.86:0/228978284 conn(0x7f07380f1aa0 msgr2=0x7f07380f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:07:22.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:22.364+0000 7f0737fff700 1 -- 172.21.15.86:0/228978284 shutdown_connections 2024-02-19T22:07:22.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:22.364+0000 7f0737fff700 1 -- 172.21.15.86:0/228978284 wait complete. 2024-02-19T22:07:22.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:22.364+0000 7f0737fff700 1 Processor -- start 2024-02-19T22:07:22.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:22.364+0000 7f0737fff700 1 -- start start 2024-02-19T22:07:22.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:22.365+0000 7f0737fff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f07380f62a0 0x7f073806b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:07:22.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:22.365+0000 7f0737fff700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0738069e20 con 0x7f07380f62a0 2024-02-19T22:07:22.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:22.365+0000 7f0736ffd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f07380f62a0 0x7f073806b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:07:22.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:22.365+0000 7f0736ffd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f07380f62a0 0x7f073806b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:35608/0 (socket says 172.21.15.86:35608) 2024-02-19T22:07:22.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:22.365+0000 7f0736ffd700 1 -- 172.21.15.86:0/1173117011 learned_addr learned my addr 172.21.15.86:0/1173117011 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T22:07:22.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:22.365+0000 7f0736ffd700 1 -- 172.21.15.86:0/1173117011 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0728005040 con 0x7f07380f62a0 2024-02-19T22:07:22.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:22.365+0000 7f0736ffd700 1 --2- 172.21.15.86:0/1173117011 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f07380f62a0 0x7f073806b9a0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 rx=0x7f07280089f0 tx=0x7f072800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:07:22.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:22.365+0000 7f071f7fe700 1 -- 172.21.15.86:0/1173117011 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0728009320 con 0x7f07380f62a0 2024-02-19T22:07:22.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:22.365+0000 7f0737fff700 1 -- 172.21.15.86:0/1173117011 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0738069fc0 con 0x7f07380f62a0 2024-02-19T22:07:22.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:22.365+0000 7f071f7fe700 1 -- 172.21.15.86:0/1173117011 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0728013380 con 0x7f07380f62a0 2024-02-19T22:07:22.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:22.365+0000 7f071f7fe700 1 -- 172.21.15.86:0/1173117011 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0728014e20 con 0x7f07380f62a0 2024-02-19T22:07:22.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:22.365+0000 7f0737fff700 1 -- 172.21.15.86:0/1173117011 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f073806a450 con 0x7f07380f62a0 2024-02-19T22:07:22.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:22.366+0000 7f071f7fe700 1 -- 172.21.15.86:0/1173117011 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f07280095f0 con 0x7f07380f62a0 2024-02-19T22:07:22.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:22.366+0000 7f0737fff700 1 -- 172.21.15.86:0/1173117011 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0738059100 con 0x7f07380f62a0 2024-02-19T22:07:22.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:22.366+0000 7f071f7fe700 1 --2- 172.21.15.86:0/1173117011 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f072002ff10 0x7f07200323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:07:22.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:22.366+0000 7f071f7fe700 1 -- 172.21.15.86:0/1173117011 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0728040ea0 con 0x7f07380f62a0 2024-02-19T22:07:22.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:22.367+0000 7f07367fc700 1 --2- 172.21.15.86:0/1173117011 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f072002ff10 0x7f07200323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:07:22.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:22.367+0000 7f07367fc700 1 --2- 172.21.15.86:0/1173117011 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f072002ff10 0x7f07200323d0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f072c0096a0 tx=0x7f072c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:07:22.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:22.371+0000 7f071f7fe700 1 -- 172.21.15.86:0/1173117011 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0728014420 con 0x7f07380f62a0 2024-02-19T22:07:22.578 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:22.578+0000 7f0737fff700 1 -- 172.21.15.86:0/1173117011 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f07380fbe30 con 0x7f07380f62a0 2024-02-19T22:07:22.579 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:22.578+0000 7f071f7fe700 1 -- 172.21.15.86:0/1173117011 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0728010230 con 0x7f07380f62a0 2024-02-19T22:07:22.579 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T22:07:22.579 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:07:22.581 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:22.580+0000 7f0737fff700 1 -- 172.21.15.86:0/1173117011 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f072002ff10 msgr2=0x7f07200323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:07:22.581 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:22.580+0000 7f0737fff700 1 --2- 172.21.15.86:0/1173117011 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f072002ff10 0x7f07200323d0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f072c0096a0 tx=0x7f072c006b40).stop 2024-02-19T22:07:22.581 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:22.580+0000 7f0737fff700 1 -- 172.21.15.86:0/1173117011 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f07380f62a0 msgr2=0x7f073806b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:07:22.581 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:22.580+0000 7f0737fff700 1 --2- 172.21.15.86:0/1173117011 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f07380f62a0 0x7f073806b9a0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 rx=0x7f07280089f0 tx=0x7f072800b040).stop 2024-02-19T22:07:22.581 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:22.580+0000 7f0737fff700 1 -- 172.21.15.86:0/1173117011 shutdown_connections 2024-02-19T22:07:22.581 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:22.580+0000 7f0737fff700 1 --2- 172.21.15.86:0/1173117011 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f072002ff10 0x7f07200323d0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:07:22.581 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:22.580+0000 7f0737fff700 1 --2- 172.21.15.86:0/1173117011 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f07380f62a0 0x7f073806b9a0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:07:22.581 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:22.580+0000 7f0737fff700 1 -- 172.21.15.86:0/1173117011 >> 172.21.15.86:0/1173117011 conn(0x7f07380f1aa0 msgr2=0x7f07380f2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:07:22.581 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:22.580+0000 7f0737fff700 1 -- 172.21.15.86:0/1173117011 shutdown_connections 2024-02-19T22:07:22.581 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:22.580+0000 7f0737fff700 1 -- 172.21.15.86:0/1173117011 wait complete. 2024-02-19T22:07:22.582 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T22:07:24.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:07:23 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:07:21.964386+0000 mgr.smithi053.fyhcdi (mgr.14162) 334 : cluster [DBG] pgmap v318: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:07:24.045 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:07:23 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:07:22.579653+0000 mon.smithi053 (mon.0) 315 : audit [DBG] from='client.? 172.21.15.86:0/1173117011' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:07:24.192 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:07:24.193 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:07:26.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:07:25 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:07:23.964730+0000 mgr.smithi053.fyhcdi (mgr.14162) 335 : cluster [DBG] pgmap v319: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:07:26.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:26.082+0000 7f132359e700 1 -- 172.21.15.86:0/468083810 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f13240f51a0 msgr2=0x7f13240f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:07:26.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:26.082+0000 7f132359e700 1 --2- 172.21.15.86:0/468083810 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f13240f51a0 0x7f13240f55c0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 rx=0x7f1318009fb0 tx=0x7f1318009b30).stop 2024-02-19T22:07:26.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:26.082+0000 7f132359e700 1 -- 172.21.15.86:0/468083810 shutdown_connections 2024-02-19T22:07:26.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:26.082+0000 7f132359e700 1 --2- 172.21.15.86:0/468083810 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f13240f51a0 0x7f13240f55c0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:07:26.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:26.082+0000 7f132359e700 1 -- 172.21.15.86:0/468083810 >> 172.21.15.86:0/468083810 conn(0x7f13240f09a0 msgr2=0x7f13240f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:07:26.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:26.082+0000 7f132359e700 1 -- 172.21.15.86:0/468083810 shutdown_connections 2024-02-19T22:07:26.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:26.082+0000 7f132359e700 1 -- 172.21.15.86:0/468083810 wait complete. 2024-02-19T22:07:26.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:26.083+0000 7f132359e700 1 Processor -- start 2024-02-19T22:07:26.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:26.083+0000 7f132359e700 1 -- start start 2024-02-19T22:07:26.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:26.083+0000 7f132359e700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f13240f51a0 0x7f13241054d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:07:26.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:26.083+0000 7f132359e700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13241059c0 con 0x7f13240f51a0 2024-02-19T22:07:26.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:26.083+0000 7f132259c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f13240f51a0 0x7f13241054d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:07:26.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:26.083+0000 7f132259c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f13240f51a0 0x7f13241054d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:35628/0 (socket says 172.21.15.86:35628) 2024-02-19T22:07:26.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:26.083+0000 7f132259c700 1 -- 172.21.15.86:0/2214953327 learned_addr learned my addr 172.21.15.86:0/2214953327 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T22:07:26.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:26.083+0000 7f132259c700 1 -- 172.21.15.86:0/2214953327 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1318005040 con 0x7f13240f51a0 2024-02-19T22:07:26.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:26.083+0000 7f132259c700 1 --2- 172.21.15.86:0/2214953327 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f13240f51a0 0x7f13241054d0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 rx=0x7f13180089f0 tx=0x7f131800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:07:26.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:26.084+0000 7f1312ffd700 1 -- 172.21.15.86:0/2214953327 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1318009320 con 0x7f13240f51a0 2024-02-19T22:07:26.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:26.084+0000 7f132359e700 1 -- 172.21.15.86:0/2214953327 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1324105bc0 con 0x7f13240f51a0 2024-02-19T22:07:26.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:26.084+0000 7f1312ffd700 1 -- 172.21.15.86:0/2214953327 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1318013380 con 0x7f13240f51a0 2024-02-19T22:07:26.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:26.084+0000 7f1312ffd700 1 -- 172.21.15.86:0/2214953327 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1318014e20 con 0x7f13240f51a0 2024-02-19T22:07:26.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:26.084+0000 7f132359e700 1 -- 172.21.15.86:0/2214953327 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1324106050 con 0x7f13240f51a0 2024-02-19T22:07:26.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:26.084+0000 7f1312ffd700 1 -- 172.21.15.86:0/2214953327 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f13180095f0 con 0x7f13240f51a0 2024-02-19T22:07:26.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:26.085+0000 7f132359e700 1 -- 172.21.15.86:0/2214953327 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1324045ad0 con 0x7f13240f51a0 2024-02-19T22:07:26.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:26.085+0000 7f1312ffd700 1 --2- 172.21.15.86:0/2214953327 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f130c02ff10 0x7f130c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:07:26.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:26.085+0000 7f1312ffd700 1 -- 172.21.15.86:0/2214953327 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1318040e90 con 0x7f13240f51a0 2024-02-19T22:07:26.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:26.085+0000 7f1321d9b700 1 --2- 172.21.15.86:0/2214953327 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f130c02ff10 0x7f130c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:07:26.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:26.085+0000 7f1321d9b700 1 --2- 172.21.15.86:0/2214953327 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f130c02ff10 0x7f130c0323d0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7f13140096a0 tx=0x7f1314006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:07:26.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:26.089+0000 7f1312ffd700 1 -- 172.21.15.86:0/2214953327 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1318014420 con 0x7f13240f51a0 2024-02-19T22:07:26.297 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:26.297+0000 7f132359e700 1 -- 172.21.15.86:0/2214953327 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1324106f00 con 0x7f13240f51a0 2024-02-19T22:07:26.298 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:26.297+0000 7f1312ffd700 1 -- 172.21.15.86:0/2214953327 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f131801d980 con 0x7f13240f51a0 2024-02-19T22:07:26.298 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T22:07:26.298 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:07:26.299 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:26.299+0000 7f132359e700 1 -- 172.21.15.86:0/2214953327 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f130c02ff10 msgr2=0x7f130c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:07:26.300 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:26.299+0000 7f132359e700 1 --2- 172.21.15.86:0/2214953327 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f130c02ff10 0x7f130c0323d0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7f13140096a0 tx=0x7f1314006b40).stop 2024-02-19T22:07:26.300 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:26.299+0000 7f132359e700 1 -- 172.21.15.86:0/2214953327 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f13240f51a0 msgr2=0x7f13241054d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:07:26.300 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:26.299+0000 7f132359e700 1 --2- 172.21.15.86:0/2214953327 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f13240f51a0 0x7f13241054d0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 rx=0x7f13180089f0 tx=0x7f131800b040).stop 2024-02-19T22:07:26.300 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:26.299+0000 7f132359e700 1 -- 172.21.15.86:0/2214953327 shutdown_connections 2024-02-19T22:07:26.300 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:26.299+0000 7f132359e700 1 --2- 172.21.15.86:0/2214953327 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f130c02ff10 0x7f130c0323d0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:07:26.300 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:26.299+0000 7f132359e700 1 --2- 172.21.15.86:0/2214953327 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f13240f51a0 0x7f13241054d0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:07:26.300 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:26.299+0000 7f132359e700 1 -- 172.21.15.86:0/2214953327 >> 172.21.15.86:0/2214953327 conn(0x7f13240f09a0 msgr2=0x7f13240f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:07:26.300 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:26.299+0000 7f132359e700 1 -- 172.21.15.86:0/2214953327 shutdown_connections 2024-02-19T22:07:26.300 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:26.299+0000 7f132359e700 1 -- 172.21.15.86:0/2214953327 wait complete. 2024-02-19T22:07:26.301 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T22:07:27.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:07:26 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:07:26.298575+0000 mon.smithi053 (mon.0) 316 : audit [DBG] from='client.? 172.21.15.86:0/2214953327' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:07:27.888 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:07:27.888 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:07:28.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:07:27 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:07:25.965189+0000 mgr.smithi053.fyhcdi (mgr.14162) 336 : cluster [DBG] pgmap v320: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:07:29.894 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:29.893+0000 7f0664b34700 1 -- 172.21.15.86:0/3426057161 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f06600f5410 msgr2=0x7f06600f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:07:29.894 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:29.893+0000 7f0664b34700 1 --2- 172.21.15.86:0/3426057161 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f06600f5410 0x7f06600f5830 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 rx=0x7f0650009fb0 tx=0x7f0650009b30).stop 2024-02-19T22:07:29.894 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:29.893+0000 7f0664b34700 1 -- 172.21.15.86:0/3426057161 shutdown_connections 2024-02-19T22:07:29.895 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:29.893+0000 7f0664b34700 1 --2- 172.21.15.86:0/3426057161 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f06600f5410 0x7f06600f5830 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:07:29.895 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:29.893+0000 7f0664b34700 1 -- 172.21.15.86:0/3426057161 >> 172.21.15.86:0/3426057161 conn(0x7f06600f0b70 msgr2=0x7f06600f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:07:29.895 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:29.893+0000 7f0664b34700 1 -- 172.21.15.86:0/3426057161 shutdown_connections 2024-02-19T22:07:29.895 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:29.893+0000 7f0664b34700 1 -- 172.21.15.86:0/3426057161 wait complete. 2024-02-19T22:07:29.895 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:29.893+0000 7f0664b34700 1 Processor -- start 2024-02-19T22:07:29.895 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:29.894+0000 7f0664b34700 1 -- start start 2024-02-19T22:07:29.895 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:29.894+0000 7f0664b34700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f06600f5410 0x7f06600fd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:07:29.895 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:29.894+0000 7f0664b34700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06600fb960 con 0x7f06600f5410 2024-02-19T22:07:29.895 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:29.894+0000 7f065e59c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f06600f5410 0x7f06600fd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:07:29.895 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:29.894+0000 7f065e59c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f06600f5410 0x7f06600fd4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:54054/0 (socket says 172.21.15.86:54054) 2024-02-19T22:07:29.896 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:29.894+0000 7f065e59c700 1 -- 172.21.15.86:0/1565923314 learned_addr learned my addr 172.21.15.86:0/1565923314 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T22:07:29.897 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:29.894+0000 7f065e59c700 1 -- 172.21.15.86:0/1565923314 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0650005040 con 0x7f06600f5410 2024-02-19T22:07:29.897 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:29.894+0000 7f065e59c700 1 --2- 172.21.15.86:0/1565923314 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f06600f5410 0x7f06600fd4e0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 rx=0x7f0650005010 tx=0x7f065000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:07:29.897 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:29.895+0000 7f064effd700 1 -- 172.21.15.86:0/1565923314 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0650009320 con 0x7f06600f5410 2024-02-19T22:07:29.897 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:29.895+0000 7f0664b34700 1 -- 172.21.15.86:0/1565923314 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06600fbb00 con 0x7f06600f5410 2024-02-19T22:07:29.897 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:29.895+0000 7f064effd700 1 -- 172.21.15.86:0/1565923314 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0650013380 con 0x7f06600f5410 2024-02-19T22:07:29.897 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:29.895+0000 7f064effd700 1 -- 172.21.15.86:0/1565923314 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0650014e50 con 0x7f06600f5410 2024-02-19T22:07:29.897 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:29.895+0000 7f0664b34700 1 -- 172.21.15.86:0/1565923314 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06600fbf90 con 0x7f06600f5410 2024-02-19T22:07:29.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:29.895+0000 7f064effd700 1 -- 172.21.15.86:0/1565923314 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f06500095f0 con 0x7f06600f5410 2024-02-19T22:07:29.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:29.895+0000 7f0664b34700 1 -- 172.21.15.86:0/1565923314 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0660059100 con 0x7f06600f5410 2024-02-19T22:07:29.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:29.896+0000 7f064effd700 1 --2- 172.21.15.86:0/1565923314 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f064802ff10 0x7f06480323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:07:29.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:29.896+0000 7f064effd700 1 -- 172.21.15.86:0/1565923314 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0650040f70 con 0x7f06600f5410 2024-02-19T22:07:29.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:29.896+0000 7f065dd9b700 1 --2- 172.21.15.86:0/1565923314 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f064802ff10 0x7f06480323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:07:29.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:29.896+0000 7f065dd9b700 1 --2- 172.21.15.86:0/1565923314 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f064802ff10 0x7f06480323d0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f06540096a0 tx=0x7f0654006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:07:29.901 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:29.900+0000 7f064effd700 1 -- 172.21.15.86:0/1565923314 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0650014420 con 0x7f06600f5410 2024-02-19T22:07:30.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:07:29 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:07:27.965608+0000 mgr.smithi053.fyhcdi (mgr.14162) 337 : cluster [DBG] pgmap v321: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:07:30.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:30.108+0000 7f0664b34700 1 -- 172.21.15.86:0/1565923314 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0660045ad0 con 0x7f06600f5410 2024-02-19T22:07:30.110 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:30.109+0000 7f064effd700 1 -- 172.21.15.86:0/1565923314 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0650010230 con 0x7f06600f5410 2024-02-19T22:07:30.110 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T22:07:30.110 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:07:30.112 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:30.111+0000 7f0664b34700 1 -- 172.21.15.86:0/1565923314 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f064802ff10 msgr2=0x7f06480323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:07:30.112 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:30.111+0000 7f0664b34700 1 --2- 172.21.15.86:0/1565923314 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f064802ff10 0x7f06480323d0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f06540096a0 tx=0x7f0654006b40).stop 2024-02-19T22:07:30.112 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:30.111+0000 7f0664b34700 1 -- 172.21.15.86:0/1565923314 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f06600f5410 msgr2=0x7f06600fd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:07:30.112 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:30.111+0000 7f0664b34700 1 --2- 172.21.15.86:0/1565923314 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f06600f5410 0x7f06600fd4e0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 rx=0x7f0650005010 tx=0x7f065000b040).stop 2024-02-19T22:07:30.112 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:30.111+0000 7f0664b34700 1 -- 172.21.15.86:0/1565923314 shutdown_connections 2024-02-19T22:07:30.112 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:30.111+0000 7f0664b34700 1 --2- 172.21.15.86:0/1565923314 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f064802ff10 0x7f06480323d0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:07:30.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:30.111+0000 7f0664b34700 1 --2- 172.21.15.86:0/1565923314 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f06600f5410 0x7f06600fd4e0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:07:30.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:30.111+0000 7f0664b34700 1 -- 172.21.15.86:0/1565923314 >> 172.21.15.86:0/1565923314 conn(0x7f06600f0b70 msgr2=0x7f066005fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:07:30.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:30.111+0000 7f0664b34700 1 -- 172.21.15.86:0/1565923314 shutdown_connections 2024-02-19T22:07:30.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:30.111+0000 7f0664b34700 1 -- 172.21.15.86:0/1565923314 wait complete. 2024-02-19T22:07:30.113 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T22:07:31.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:07:30 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:07:30.110529+0000 mon.smithi053 (mon.0) 317 : audit [DBG] from='client.? 172.21.15.86:0/1565923314' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:07:31.731 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:07:31.731 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:07:32.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:07:31 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:07:29.966076+0000 mgr.smithi053.fyhcdi (mgr.14162) 338 : cluster [DBG] pgmap v322: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:07:33.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:33.872+0000 7f98e025b700 1 -- 172.21.15.86:0/2900710015 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f98d80f6480 msgr2=0x7f98d80f68a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:07:33.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:33.872+0000 7f98e025b700 1 --2- 172.21.15.86:0/2900710015 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f98d80f6480 0x7f98d80f68a0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 rx=0x7f98cc009fb0 tx=0x7f98cc009b30).stop 2024-02-19T22:07:33.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:33.872+0000 7f98e025b700 1 -- 172.21.15.86:0/2900710015 shutdown_connections 2024-02-19T22:07:33.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:33.872+0000 7f98e025b700 1 --2- 172.21.15.86:0/2900710015 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f98d80f6480 0x7f98d80f68a0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:07:33.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:33.872+0000 7f98e025b700 1 -- 172.21.15.86:0/2900710015 >> 172.21.15.86:0/2900710015 conn(0x7f98d80f1c00 msgr2=0x7f98d80f4020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:07:33.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:33.872+0000 7f98e025b700 1 -- 172.21.15.86:0/2900710015 shutdown_connections 2024-02-19T22:07:33.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:33.872+0000 7f98e025b700 1 -- 172.21.15.86:0/2900710015 wait complete. 2024-02-19T22:07:33.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:33.873+0000 7f98e025b700 1 Processor -- start 2024-02-19T22:07:33.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:33.873+0000 7f98e025b700 1 -- start start 2024-02-19T22:07:33.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:33.873+0000 7f98e025b700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f98d80f6480 0x7f98d81067f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:07:33.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:33.873+0000 7f98e025b700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98d8106ce0 con 0x7f98d80f6480 2024-02-19T22:07:33.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:33.874+0000 7f98ddff7700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f98d80f6480 0x7f98d81067f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:07:33.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:33.874+0000 7f98ddff7700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f98d80f6480 0x7f98d81067f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:54064/0 (socket says 172.21.15.86:54064) 2024-02-19T22:07:33.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:33.874+0000 7f98ddff7700 1 -- 172.21.15.86:0/2601931356 learned_addr learned my addr 172.21.15.86:0/2601931356 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T22:07:33.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:33.874+0000 7f98ddff7700 1 -- 172.21.15.86:0/2601931356 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98cc005040 con 0x7f98d80f6480 2024-02-19T22:07:33.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:33.874+0000 7f98ddff7700 1 --2- 172.21.15.86:0/2601931356 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f98d80f6480 0x7f98d81067f0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 rx=0x7f98cc0089f0 tx=0x7f98cc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:07:33.876 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:33.874+0000 7f98ca7fc700 1 -- 172.21.15.86:0/2601931356 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f98cc009320 con 0x7f98d80f6480 2024-02-19T22:07:33.877 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:33.874+0000 7f98ca7fc700 1 -- 172.21.15.86:0/2601931356 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f98cc013380 con 0x7f98d80f6480 2024-02-19T22:07:33.877 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:33.874+0000 7f98e025b700 1 -- 172.21.15.86:0/2601931356 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98d8106ee0 con 0x7f98d80f6480 2024-02-19T22:07:33.877 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:33.874+0000 7f98ca7fc700 1 -- 172.21.15.86:0/2601931356 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f98cc01ed80 con 0x7f98d80f6480 2024-02-19T22:07:33.877 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:33.874+0000 7f98e025b700 1 -- 172.21.15.86:0/2601931356 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98d8107370 con 0x7f98d80f6480 2024-02-19T22:07:33.877 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:33.875+0000 7f98ca7fc700 1 -- 172.21.15.86:0/2601931356 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f98cc01e420 con 0x7f98d80f6480 2024-02-19T22:07:33.877 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:33.875+0000 7f98e025b700 1 -- 172.21.15.86:0/2601931356 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98d8045ad0 con 0x7f98d80f6480 2024-02-19T22:07:33.877 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:33.875+0000 7f98ca7fc700 1 --2- 172.21.15.86:0/2601931356 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f98c402ffb0 0x7f98c4032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:07:33.877 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:33.875+0000 7f98ca7fc700 1 -- 172.21.15.86:0/2601931356 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f98cc040fb0 con 0x7f98d80f6480 2024-02-19T22:07:33.877 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:33.875+0000 7f98dd7f6700 1 --2- 172.21.15.86:0/2601931356 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f98c402ffb0 0x7f98c4032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:07:33.877 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:33.876+0000 7f98dd7f6700 1 --2- 172.21.15.86:0/2601931356 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f98c402ffb0 0x7f98c4032470 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f98d40096a0 tx=0x7f98d4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:07:33.880 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:33.880+0000 7f98ca7fc700 1 -- 172.21.15.86:0/2601931356 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f98cc0095f0 con 0x7f98d80f6480 2024-02-19T22:07:34.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:07:33 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:07:31.966480+0000 mgr.smithi053.fyhcdi (mgr.14162) 339 : cluster [DBG] pgmap v323: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:07:34.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:34.087+0000 7f98e025b700 1 -- 172.21.15.86:0/2601931356 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f98d80fc090 con 0x7f98d80f6480 2024-02-19T22:07:34.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:34.088+0000 7f98ca7fc700 1 -- 172.21.15.86:0/2601931356 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f98cc01cb10 con 0x7f98d80f6480 2024-02-19T22:07:34.088 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T22:07:34.089 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:07:34.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:34.089+0000 7f98e025b700 1 -- 172.21.15.86:0/2601931356 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f98c402ffb0 msgr2=0x7f98c4032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:07:34.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:34.089+0000 7f98e025b700 1 --2- 172.21.15.86:0/2601931356 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f98c402ffb0 0x7f98c4032470 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f98d40096a0 tx=0x7f98d4006b40).stop 2024-02-19T22:07:34.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:34.089+0000 7f98e025b700 1 -- 172.21.15.86:0/2601931356 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f98d80f6480 msgr2=0x7f98d81067f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:07:34.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:34.089+0000 7f98e025b700 1 --2- 172.21.15.86:0/2601931356 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f98d80f6480 0x7f98d81067f0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 rx=0x7f98cc0089f0 tx=0x7f98cc00b040).stop 2024-02-19T22:07:34.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:34.089+0000 7f98e025b700 1 -- 172.21.15.86:0/2601931356 shutdown_connections 2024-02-19T22:07:34.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:34.090+0000 7f98e025b700 1 --2- 172.21.15.86:0/2601931356 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f98c402ffb0 0x7f98c4032470 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:07:34.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:34.090+0000 7f98e025b700 1 --2- 172.21.15.86:0/2601931356 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f98d80f6480 0x7f98d81067f0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:07:34.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:34.090+0000 7f98e025b700 1 -- 172.21.15.86:0/2601931356 >> 172.21.15.86:0/2601931356 conn(0x7f98d80f1c00 msgr2=0x7f98d80f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:07:34.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:34.090+0000 7f98e025b700 1 -- 172.21.15.86:0/2601931356 shutdown_connections 2024-02-19T22:07:34.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:34.090+0000 7f98e025b700 1 -- 172.21.15.86:0/2601931356 wait complete. 2024-02-19T22:07:34.091 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T22:07:35.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:07:34 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:07:34.089183+0000 mon.smithi053 (mon.0) 318 : audit [DBG] from='client.? 172.21.15.86:0/2601931356' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:07:35.757 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:07:35.757 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:07:36.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:07:35 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:07:33.967006+0000 mgr.smithi053.fyhcdi (mgr.14162) 340 : cluster [DBG] pgmap v324: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:07:37.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:07:36 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:07:35.607851+0000 mon.smithi053 (mon.0) 319 : audit [INF] from='mgr.14162 172.21.15.53:0/2860358250' entity='mgr.smithi053.fyhcdi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi053.fyhcdi/mirror_snapshot_schedule"}]: dispatch 2024-02-19T22:07:37.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:07:36 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:07:35.610054+0000 mon.smithi053 (mon.0) 320 : audit [INF] from='mgr.14162 172.21.15.53:0/2860358250' entity='mgr.smithi053.fyhcdi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi053.fyhcdi/trash_purge_schedule"}]: dispatch 2024-02-19T22:07:37.759 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:37.758+0000 7f5adf6a2700 1 -- 172.21.15.86:0/732509139 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5ad8060280 msgr2=0x7f5ad80f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:07:37.760 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:37.758+0000 7f5adf6a2700 1 --2- 172.21.15.86:0/732509139 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5ad8060280 0x7f5ad80f90f0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 rx=0x7f5ad4009fb0 tx=0x7f5ad4009b30).stop 2024-02-19T22:07:37.760 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:37.758+0000 7f5adf6a2700 1 -- 172.21.15.86:0/732509139 shutdown_connections 2024-02-19T22:07:37.760 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:37.758+0000 7f5adf6a2700 1 --2- 172.21.15.86:0/732509139 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5ad8060280 0x7f5ad80f90f0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:07:37.760 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:37.758+0000 7f5adf6a2700 1 -- 172.21.15.86:0/732509139 >> 172.21.15.86:0/732509139 conn(0x7f5ad80f0b70 msgr2=0x7f5ad80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:07:37.760 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:37.758+0000 7f5adf6a2700 1 -- 172.21.15.86:0/732509139 shutdown_connections 2024-02-19T22:07:37.760 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:37.758+0000 7f5adf6a2700 1 -- 172.21.15.86:0/732509139 wait complete. 2024-02-19T22:07:37.760 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:37.758+0000 7f5adf6a2700 1 Processor -- start 2024-02-19T22:07:37.760 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:37.758+0000 7f5adf6a2700 1 -- start start 2024-02-19T22:07:37.760 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:37.758+0000 7f5adf6a2700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5ad8060280 0x7f5ad8106780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:07:37.760 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:37.758+0000 7f5adf6a2700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ad8106c70 con 0x7f5ad8060280 2024-02-19T22:07:37.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:37.759+0000 7f5add43e700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5ad8060280 0x7f5ad8106780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:07:37.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:37.759+0000 7f5add43e700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5ad8060280 0x7f5ad8106780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:54084/0 (socket says 172.21.15.86:54084) 2024-02-19T22:07:37.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:37.759+0000 7f5add43e700 1 -- 172.21.15.86:0/1017243108 learned_addr learned my addr 172.21.15.86:0/1017243108 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T22:07:37.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:37.759+0000 7f5add43e700 1 -- 172.21.15.86:0/1017243108 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ad4005040 con 0x7f5ad8060280 2024-02-19T22:07:37.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:37.759+0000 7f5add43e700 1 --2- 172.21.15.86:0/1017243108 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5ad8060280 0x7f5ad8106780 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0x7f5ad40089f0 tx=0x7f5ad400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:07:37.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:37.759+0000 7f5acdffb700 1 -- 172.21.15.86:0/1017243108 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5ad4009320 con 0x7f5ad8060280 2024-02-19T22:07:37.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:37.759+0000 7f5adf6a2700 1 -- 172.21.15.86:0/1017243108 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ad8106e70 con 0x7f5ad8060280 2024-02-19T22:07:37.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:37.759+0000 7f5acdffb700 1 -- 172.21.15.86:0/1017243108 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5ad4013380 con 0x7f5ad8060280 2024-02-19T22:07:37.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:37.759+0000 7f5acdffb700 1 -- 172.21.15.86:0/1017243108 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5ad4014e20 con 0x7f5ad8060280 2024-02-19T22:07:37.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:37.759+0000 7f5adf6a2700 1 -- 172.21.15.86:0/1017243108 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ad8107300 con 0x7f5ad8060280 2024-02-19T22:07:37.762 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:37.760+0000 7f5adf6a2700 1 -- 172.21.15.86:0/1017243108 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ad8045ad0 con 0x7f5ad8060280 2024-02-19T22:07:37.762 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:37.760+0000 7f5acdffb700 1 -- 172.21.15.86:0/1017243108 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5ad40095f0 con 0x7f5ad8060280 2024-02-19T22:07:37.762 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:37.760+0000 7f5acdffb700 1 --2- 172.21.15.86:0/1017243108 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f5ac402ff10 0x7f5ac40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:07:37.762 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:37.760+0000 7f5acdffb700 1 -- 172.21.15.86:0/1017243108 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5ad4040ec0 con 0x7f5ad8060280 2024-02-19T22:07:37.762 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:37.761+0000 7f5adcc3d700 1 --2- 172.21.15.86:0/1017243108 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f5ac402ff10 0x7f5ac40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:07:37.762 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:37.761+0000 7f5adcc3d700 1 --2- 172.21.15.86:0/1017243108 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f5ac402ff10 0x7f5ac40323d0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f5ac80096a0 tx=0x7f5ac8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:07:37.765 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:37.765+0000 7f5acdffb700 1 -- 172.21.15.86:0/1017243108 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5ad4014420 con 0x7f5ad8060280 2024-02-19T22:07:37.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:37.973+0000 7f5adf6a2700 1 -- 172.21.15.86:0/1017243108 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5ad81081b0 con 0x7f5ad8060280 2024-02-19T22:07:37.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:37.974+0000 7f5acdffb700 1 -- 172.21.15.86:0/1017243108 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5ad4010230 con 0x7f5ad8060280 2024-02-19T22:07:37.974 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T22:07:37.975 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:07:37.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:37.976+0000 7f5adf6a2700 1 -- 172.21.15.86:0/1017243108 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f5ac402ff10 msgr2=0x7f5ac40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:07:37.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:37.976+0000 7f5adf6a2700 1 --2- 172.21.15.86:0/1017243108 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f5ac402ff10 0x7f5ac40323d0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f5ac80096a0 tx=0x7f5ac8006b40).stop 2024-02-19T22:07:37.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:37.976+0000 7f5adf6a2700 1 -- 172.21.15.86:0/1017243108 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5ad8060280 msgr2=0x7f5ad8106780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:07:37.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:37.976+0000 7f5adf6a2700 1 --2- 172.21.15.86:0/1017243108 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5ad8060280 0x7f5ad8106780 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0x7f5ad40089f0 tx=0x7f5ad400b040).stop 2024-02-19T22:07:37.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:37.976+0000 7f5adf6a2700 1 -- 172.21.15.86:0/1017243108 shutdown_connections 2024-02-19T22:07:37.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:37.976+0000 7f5adf6a2700 1 --2- 172.21.15.86:0/1017243108 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f5ac402ff10 0x7f5ac40323d0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:07:37.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:37.976+0000 7f5adf6a2700 1 --2- 172.21.15.86:0/1017243108 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5ad8060280 0x7f5ad8106780 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:07:37.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:37.976+0000 7f5adf6a2700 1 -- 172.21.15.86:0/1017243108 >> 172.21.15.86:0/1017243108 conn(0x7f5ad80f0b70 msgr2=0x7f5ad80f15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:07:37.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:37.976+0000 7f5adf6a2700 1 -- 172.21.15.86:0/1017243108 shutdown_connections 2024-02-19T22:07:37.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:37.976+0000 7f5adf6a2700 1 -- 172.21.15.86:0/1017243108 wait complete. 2024-02-19T22:07:37.977 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T22:07:38.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:07:37 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:07:35.967294+0000 mgr.smithi053.fyhcdi (mgr.14162) 341 : cluster [DBG] pgmap v325: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:07:39.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:07:38 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:07:37.975213+0000 mon.smithi053 (mon.0) 321 : audit [DBG] from='client.? 172.21.15.86:0/1017243108' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:07:39.694 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:07:39.694 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:07:40.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:07:39 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:07:37.967616+0000 mgr.smithi053.fyhcdi (mgr.14162) 342 : cluster [DBG] pgmap v326: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:07:41.645 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:41.643+0000 7f31ca19f700 1 -- 172.21.15.86:0/2421944297 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f31c40f62a0 msgr2=0x7f31c40f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:07:41.646 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:41.643+0000 7f31ca19f700 1 --2- 172.21.15.86:0/2421944297 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f31c40f62a0 0x7f31c40f66c0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 rx=0x7f31b4009fb0 tx=0x7f31b4009b30).stop 2024-02-19T22:07:41.646 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:41.643+0000 7f31ca19f700 1 -- 172.21.15.86:0/2421944297 shutdown_connections 2024-02-19T22:07:41.646 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:41.643+0000 7f31ca19f700 1 --2- 172.21.15.86:0/2421944297 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f31c40f62a0 0x7f31c40f66c0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:07:41.646 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:41.643+0000 7f31ca19f700 1 -- 172.21.15.86:0/2421944297 >> 172.21.15.86:0/2421944297 conn(0x7f31c40f1aa0 msgr2=0x7f31c40f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:07:41.646 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:41.644+0000 7f31ca19f700 1 -- 172.21.15.86:0/2421944297 shutdown_connections 2024-02-19T22:07:41.646 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:41.644+0000 7f31ca19f700 1 -- 172.21.15.86:0/2421944297 wait complete. 2024-02-19T22:07:41.646 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:41.644+0000 7f31ca19f700 1 Processor -- start 2024-02-19T22:07:41.646 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:41.644+0000 7f31ca19f700 1 -- start start 2024-02-19T22:07:41.646 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:41.644+0000 7f31ca19f700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f31c40f62a0 0x7f31c406b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:07:41.646 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:41.644+0000 7f31ca19f700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31c4069e20 con 0x7f31c40f62a0 2024-02-19T22:07:41.646 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:41.644+0000 7f31c919d700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f31c40f62a0 0x7f31c406b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:07:41.646 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:41.644+0000 7f31c919d700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f31c40f62a0 0x7f31c406b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:53772/0 (socket says 172.21.15.86:53772) 2024-02-19T22:07:41.646 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:41.644+0000 7f31c919d700 1 -- 172.21.15.86:0/3353187712 learned_addr learned my addr 172.21.15.86:0/3353187712 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T22:07:41.647 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:41.645+0000 7f31c919d700 1 -- 172.21.15.86:0/3353187712 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31b4005040 con 0x7f31c40f62a0 2024-02-19T22:07:41.647 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:41.645+0000 7f31c919d700 1 --2- 172.21.15.86:0/3353187712 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f31c40f62a0 0x7f31c406b9a0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 rx=0x7f31b40089f0 tx=0x7f31b400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:07:41.647 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:41.645+0000 7f31b9ffb700 1 -- 172.21.15.86:0/3353187712 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f31b4009320 con 0x7f31c40f62a0 2024-02-19T22:07:41.647 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:41.645+0000 7f31ca19f700 1 -- 172.21.15.86:0/3353187712 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31c4069fc0 con 0x7f31c40f62a0 2024-02-19T22:07:41.647 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:41.645+0000 7f31b9ffb700 1 -- 172.21.15.86:0/3353187712 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f31b4013380 con 0x7f31c40f62a0 2024-02-19T22:07:41.647 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:41.645+0000 7f31b9ffb700 1 -- 172.21.15.86:0/3353187712 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f31b4014e20 con 0x7f31c40f62a0 2024-02-19T22:07:41.647 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:41.645+0000 7f31ca19f700 1 -- 172.21.15.86:0/3353187712 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31c406a450 con 0x7f31c40f62a0 2024-02-19T22:07:41.648 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:41.646+0000 7f31b9ffb700 1 -- 172.21.15.86:0/3353187712 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f31b40095f0 con 0x7f31c40f62a0 2024-02-19T22:07:41.648 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:41.646+0000 7f31ca19f700 1 -- 172.21.15.86:0/3353187712 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31c4059100 con 0x7f31c40f62a0 2024-02-19T22:07:41.648 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:41.646+0000 7f31b9ffb700 1 --2- 172.21.15.86:0/3353187712 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f31b002ff10 0x7f31b00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:07:41.648 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:41.646+0000 7f31b9ffb700 1 -- 172.21.15.86:0/3353187712 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f31b4040f80 con 0x7f31c40f62a0 2024-02-19T22:07:41.648 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:41.646+0000 7f31c899c700 1 --2- 172.21.15.86:0/3353187712 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f31b002ff10 0x7f31b00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:07:41.648 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:41.647+0000 7f31c899c700 1 --2- 172.21.15.86:0/3353187712 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f31b002ff10 0x7f31b00323d0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7f31c00096a0 tx=0x7f31c0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:07:41.651 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:41.651+0000 7f31b9ffb700 1 -- 172.21.15.86:0/3353187712 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f31b4014420 con 0x7f31c40f62a0 2024-02-19T22:07:41.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:41.856+0000 7f31ca19f700 1 -- 172.21.15.86:0/3353187712 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f31c4045ad0 con 0x7f31c40f62a0 2024-02-19T22:07:41.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:41.856+0000 7f31b9ffb700 1 -- 172.21.15.86:0/3353187712 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f31b4010230 con 0x7f31c40f62a0 2024-02-19T22:07:41.857 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T22:07:41.857 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:07:41.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:41.858+0000 7f31ca19f700 1 -- 172.21.15.86:0/3353187712 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f31b002ff10 msgr2=0x7f31b00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:07:41.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:41.858+0000 7f31ca19f700 1 --2- 172.21.15.86:0/3353187712 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f31b002ff10 0x7f31b00323d0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7f31c00096a0 tx=0x7f31c0006b40).stop 2024-02-19T22:07:41.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:41.858+0000 7f31ca19f700 1 -- 172.21.15.86:0/3353187712 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f31c40f62a0 msgr2=0x7f31c406b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:07:41.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:41.858+0000 7f31ca19f700 1 --2- 172.21.15.86:0/3353187712 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f31c40f62a0 0x7f31c406b9a0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 rx=0x7f31b40089f0 tx=0x7f31b400b040).stop 2024-02-19T22:07:41.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:41.858+0000 7f31ca19f700 1 -- 172.21.15.86:0/3353187712 shutdown_connections 2024-02-19T22:07:41.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:41.858+0000 7f31ca19f700 1 --2- 172.21.15.86:0/3353187712 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f31b002ff10 0x7f31b00323d0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:07:41.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:41.858+0000 7f31ca19f700 1 --2- 172.21.15.86:0/3353187712 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f31c40f62a0 0x7f31c406b9a0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:07:41.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:41.858+0000 7f31ca19f700 1 -- 172.21.15.86:0/3353187712 >> 172.21.15.86:0/3353187712 conn(0x7f31c40f1aa0 msgr2=0x7f31c40f2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:07:41.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:41.858+0000 7f31ca19f700 1 -- 172.21.15.86:0/3353187712 shutdown_connections 2024-02-19T22:07:41.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:41.858+0000 7f31ca19f700 1 -- 172.21.15.86:0/3353187712 wait complete. 2024-02-19T22:07:41.859 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T22:07:42.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:07:41 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:07:39.968012+0000 mgr.smithi053.fyhcdi (mgr.14162) 343 : cluster [DBG] pgmap v327: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:07:43.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:07:42 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:07:41.857465+0000 mon.smithi053 (mon.0) 322 : audit [DBG] from='client.? 172.21.15.86:0/3353187712' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:07:43.660 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:07:43.660 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:07:43.794 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:07:43 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:07:41.968540+0000 mgr.smithi053.fyhcdi (mgr.14162) 344 : cluster [DBG] pgmap v328: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:07:45.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:45.508+0000 7f7414eeb700 1 -- 172.21.15.86:0/3394702393 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f74100f62a0 msgr2=0x7f74100f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:07:45.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:45.508+0000 7f7414eeb700 1 --2- 172.21.15.86:0/3394702393 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f74100f62a0 0x7f74100f66c0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 rx=0x7f7400009fb0 tx=0x7f7400009b30).stop 2024-02-19T22:07:45.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:45.508+0000 7f7414eeb700 1 -- 172.21.15.86:0/3394702393 shutdown_connections 2024-02-19T22:07:45.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:45.508+0000 7f7414eeb700 1 --2- 172.21.15.86:0/3394702393 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f74100f62a0 0x7f74100f66c0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:07:45.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:45.508+0000 7f7414eeb700 1 -- 172.21.15.86:0/3394702393 >> 172.21.15.86:0/3394702393 conn(0x7f74100f1aa0 msgr2=0x7f74100f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:07:45.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:45.508+0000 7f7414eeb700 1 -- 172.21.15.86:0/3394702393 shutdown_connections 2024-02-19T22:07:45.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:45.508+0000 7f7414eeb700 1 -- 172.21.15.86:0/3394702393 wait complete. 2024-02-19T22:07:45.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:45.508+0000 7f7414eeb700 1 Processor -- start 2024-02-19T22:07:45.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:45.509+0000 7f7414eeb700 1 -- start start 2024-02-19T22:07:45.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:45.509+0000 7f7414eeb700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f74100f62a0 0x7f741006b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:07:45.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:45.509+0000 7f7414eeb700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7410069e20 con 0x7f74100f62a0 2024-02-19T22:07:45.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:45.509+0000 7f740f7fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f74100f62a0 0x7f741006b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:07:45.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:45.509+0000 7f740f7fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f74100f62a0 0x7f741006b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:53798/0 (socket says 172.21.15.86:53798) 2024-02-19T22:07:45.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:45.509+0000 7f740f7fe700 1 -- 172.21.15.86:0/2426179902 learned_addr learned my addr 172.21.15.86:0/2426179902 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T22:07:45.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:45.509+0000 7f740f7fe700 1 -- 172.21.15.86:0/2426179902 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7400005040 con 0x7f74100f62a0 2024-02-19T22:07:45.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:45.510+0000 7f740f7fe700 1 --2- 172.21.15.86:0/2426179902 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f74100f62a0 0x7f741006b9a0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 rx=0x7f74000089f0 tx=0x7f740000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:07:45.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:45.510+0000 7f7407fff700 1 -- 172.21.15.86:0/2426179902 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7400009320 con 0x7f74100f62a0 2024-02-19T22:07:45.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:45.510+0000 7f7407fff700 1 -- 172.21.15.86:0/2426179902 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7400013380 con 0x7f74100f62a0 2024-02-19T22:07:45.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:45.510+0000 7f7414eeb700 1 -- 172.21.15.86:0/2426179902 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7410069fc0 con 0x7f74100f62a0 2024-02-19T22:07:45.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:45.510+0000 7f7407fff700 1 -- 172.21.15.86:0/2426179902 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7400014e20 con 0x7f74100f62a0 2024-02-19T22:07:45.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:45.510+0000 7f7414eeb700 1 -- 172.21.15.86:0/2426179902 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f741006a450 con 0x7f74100f62a0 2024-02-19T22:07:45.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:45.511+0000 7f7407fff700 1 -- 172.21.15.86:0/2426179902 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f74000095f0 con 0x7f74100f62a0 2024-02-19T22:07:45.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:45.511+0000 7f7414eeb700 1 -- 172.21.15.86:0/2426179902 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7410059100 con 0x7f74100f62a0 2024-02-19T22:07:45.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:45.511+0000 7f7407fff700 1 --2- 172.21.15.86:0/2426179902 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f73fc02ff10 0x7f73fc0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:07:45.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:45.511+0000 7f7407fff700 1 -- 172.21.15.86:0/2426179902 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7400040e90 con 0x7f74100f62a0 2024-02-19T22:07:45.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:45.511+0000 7f740effd700 1 --2- 172.21.15.86:0/2426179902 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f73fc02ff10 0x7f73fc0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:07:45.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:45.511+0000 7f740effd700 1 --2- 172.21.15.86:0/2426179902 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f73fc02ff10 0x7f73fc0323d0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f73f80096a0 tx=0x7f73f8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:07:45.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:45.516+0000 7f7407fff700 1 -- 172.21.15.86:0/2426179902 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f740001db60 con 0x7f74100f62a0 2024-02-19T22:07:45.719 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:45.720+0000 7f7414eeb700 1 -- 172.21.15.86:0/2426179902 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f74100fbe30 con 0x7f74100f62a0 2024-02-19T22:07:45.720 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:45.721+0000 7f7407fff700 1 -- 172.21.15.86:0/2426179902 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f740001dd70 con 0x7f74100f62a0 2024-02-19T22:07:45.720 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T22:07:45.720 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:07:45.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:45.722+0000 7f7414eeb700 1 -- 172.21.15.86:0/2426179902 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f73fc02ff10 msgr2=0x7f73fc0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:07:45.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:45.722+0000 7f7414eeb700 1 --2- 172.21.15.86:0/2426179902 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f73fc02ff10 0x7f73fc0323d0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f73f80096a0 tx=0x7f73f8006b40).stop 2024-02-19T22:07:45.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:45.722+0000 7f7414eeb700 1 -- 172.21.15.86:0/2426179902 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f74100f62a0 msgr2=0x7f741006b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:07:45.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:45.722+0000 7f7414eeb700 1 --2- 172.21.15.86:0/2426179902 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f74100f62a0 0x7f741006b9a0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 rx=0x7f74000089f0 tx=0x7f740000b040).stop 2024-02-19T22:07:45.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:45.722+0000 7f7414eeb700 1 -- 172.21.15.86:0/2426179902 shutdown_connections 2024-02-19T22:07:45.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:45.722+0000 7f7414eeb700 1 --2- 172.21.15.86:0/2426179902 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f73fc02ff10 0x7f73fc0323d0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:07:45.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:45.722+0000 7f7414eeb700 1 --2- 172.21.15.86:0/2426179902 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f74100f62a0 0x7f741006b9a0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:07:45.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:45.722+0000 7f7414eeb700 1 -- 172.21.15.86:0/2426179902 >> 172.21.15.86:0/2426179902 conn(0x7f74100f1aa0 msgr2=0x7f74100f2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:07:45.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:45.722+0000 7f7414eeb700 1 -- 172.21.15.86:0/2426179902 shutdown_connections 2024-02-19T22:07:45.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:45.722+0000 7f7414eeb700 1 -- 172.21.15.86:0/2426179902 wait complete. 2024-02-19T22:07:45.723 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T22:07:46.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:07:45 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:07:43.969053+0000 mgr.smithi053.fyhcdi (mgr.14162) 345 : cluster [DBG] pgmap v329: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:07:47.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:07:46 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:07:45.720955+0000 mon.smithi053 (mon.0) 323 : audit [DBG] from='client.? 172.21.15.86:0/2426179902' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:07:47.481 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:07:47.482 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:07:48.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:07:47 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:07:45.969580+0000 mgr.smithi053.fyhcdi (mgr.14162) 346 : cluster [DBG] pgmap v330: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:07:49.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:49.365+0000 7f25fa6eb700 1 -- 172.21.15.86:0/4090080569 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f25f40f51a0 msgr2=0x7f25f40f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:07:49.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:49.365+0000 7f25fa6eb700 1 --2- 172.21.15.86:0/4090080569 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f25f40f51a0 0x7f25f40f55c0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 rx=0x7f25f0009fb0 tx=0x7f25f0009b30).stop 2024-02-19T22:07:49.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:49.365+0000 7f25fa6eb700 1 -- 172.21.15.86:0/4090080569 shutdown_connections 2024-02-19T22:07:49.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:49.365+0000 7f25fa6eb700 1 --2- 172.21.15.86:0/4090080569 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f25f40f51a0 0x7f25f40f55c0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:07:49.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:49.365+0000 7f25fa6eb700 1 -- 172.21.15.86:0/4090080569 >> 172.21.15.86:0/4090080569 conn(0x7f25f40f09a0 msgr2=0x7f25f40f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:07:49.368 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:49.365+0000 7f25fa6eb700 1 -- 172.21.15.86:0/4090080569 shutdown_connections 2024-02-19T22:07:49.368 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:49.365+0000 7f25fa6eb700 1 -- 172.21.15.86:0/4090080569 wait complete. 2024-02-19T22:07:49.368 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:49.365+0000 7f25fa6eb700 1 Processor -- start 2024-02-19T22:07:49.368 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:49.366+0000 7f25fa6eb700 1 -- start start 2024-02-19T22:07:49.368 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:49.366+0000 7f25fa6eb700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f25f40f51a0 0x7f25f4109290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:07:49.368 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:49.366+0000 7f25fa6eb700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25f4060280 con 0x7f25f40f51a0 2024-02-19T22:07:49.368 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:49.366+0000 7f25f96e9700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f25f40f51a0 0x7f25f4109290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:07:49.368 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:49.366+0000 7f25f96e9700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f25f40f51a0 0x7f25f4109290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:60812/0 (socket says 172.21.15.86:60812) 2024-02-19T22:07:49.368 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:49.366+0000 7f25f96e9700 1 -- 172.21.15.86:0/2358038523 learned_addr learned my addr 172.21.15.86:0/2358038523 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T22:07:49.368 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:49.366+0000 7f25f96e9700 1 -- 172.21.15.86:0/2358038523 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25f0005040 con 0x7f25f40f51a0 2024-02-19T22:07:49.368 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:49.366+0000 7f25f96e9700 1 --2- 172.21.15.86:0/2358038523 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f25f40f51a0 0x7f25f4109290 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 rx=0x7f25f00089f0 tx=0x7f25f000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:07:49.368 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:49.367+0000 7f25e9ffb700 1 -- 172.21.15.86:0/2358038523 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f25f0009320 con 0x7f25f40f51a0 2024-02-19T22:07:49.368 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:49.367+0000 7f25fa6eb700 1 -- 172.21.15.86:0/2358038523 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25f4060480 con 0x7f25f40f51a0 2024-02-19T22:07:49.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:49.367+0000 7f25e9ffb700 1 -- 172.21.15.86:0/2358038523 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f25f0013380 con 0x7f25f40f51a0 2024-02-19T22:07:49.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:49.367+0000 7f25e9ffb700 1 -- 172.21.15.86:0/2358038523 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f25f0014e20 con 0x7f25f40f51a0 2024-02-19T22:07:49.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:49.367+0000 7f25fa6eb700 1 -- 172.21.15.86:0/2358038523 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25f41099c0 con 0x7f25f40f51a0 2024-02-19T22:07:49.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:49.368+0000 7f25e9ffb700 1 -- 172.21.15.86:0/2358038523 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f25f00095f0 con 0x7f25f40f51a0 2024-02-19T22:07:49.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:49.368+0000 7f25fa6eb700 1 -- 172.21.15.86:0/2358038523 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25f4045ad0 con 0x7f25f40f51a0 2024-02-19T22:07:49.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:49.368+0000 7f25e9ffb700 1 --2- 172.21.15.86:0/2358038523 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f25e002ff10 0x7f25e00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:07:49.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:49.368+0000 7f25e9ffb700 1 -- 172.21.15.86:0/2358038523 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f25f0040ef0 con 0x7f25f40f51a0 2024-02-19T22:07:49.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:49.368+0000 7f25f8ee8700 1 --2- 172.21.15.86:0/2358038523 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f25e002ff10 0x7f25e00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:07:49.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:49.368+0000 7f25f8ee8700 1 --2- 172.21.15.86:0/2358038523 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f25e002ff10 0x7f25e00323d0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7f25e40096a0 tx=0x7f25e4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:07:49.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:49.372+0000 7f25e9ffb700 1 -- 172.21.15.86:0/2358038523 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f25f0014420 con 0x7f25f40f51a0 2024-02-19T22:07:49.578 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:49.577+0000 7f25fa6eb700 1 -- 172.21.15.86:0/2358038523 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f25f40fad30 con 0x7f25f40f51a0 2024-02-19T22:07:49.578 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:49.578+0000 7f25e9ffb700 1 -- 172.21.15.86:0/2358038523 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f25f0010230 con 0x7f25f40f51a0 2024-02-19T22:07:49.579 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T22:07:49.579 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:07:49.580 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:49.580+0000 7f25fa6eb700 1 -- 172.21.15.86:0/2358038523 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f25e002ff10 msgr2=0x7f25e00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:07:49.580 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:49.580+0000 7f25fa6eb700 1 --2- 172.21.15.86:0/2358038523 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f25e002ff10 0x7f25e00323d0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7f25e40096a0 tx=0x7f25e4006b40).stop 2024-02-19T22:07:49.580 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:49.580+0000 7f25fa6eb700 1 -- 172.21.15.86:0/2358038523 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f25f40f51a0 msgr2=0x7f25f4109290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:07:49.580 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:49.580+0000 7f25fa6eb700 1 --2- 172.21.15.86:0/2358038523 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f25f40f51a0 0x7f25f4109290 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 rx=0x7f25f00089f0 tx=0x7f25f000b040).stop 2024-02-19T22:07:49.581 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:49.580+0000 7f25fa6eb700 1 -- 172.21.15.86:0/2358038523 shutdown_connections 2024-02-19T22:07:49.581 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:49.580+0000 7f25fa6eb700 1 --2- 172.21.15.86:0/2358038523 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f25e002ff10 0x7f25e00323d0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:07:49.581 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:49.580+0000 7f25fa6eb700 1 --2- 172.21.15.86:0/2358038523 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f25f40f51a0 0x7f25f4109290 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:07:49.581 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:49.580+0000 7f25fa6eb700 1 -- 172.21.15.86:0/2358038523 >> 172.21.15.86:0/2358038523 conn(0x7f25f40f09a0 msgr2=0x7f25f40f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:07:49.581 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:49.580+0000 7f25fa6eb700 1 -- 172.21.15.86:0/2358038523 shutdown_connections 2024-02-19T22:07:49.581 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:49.580+0000 7f25fa6eb700 1 -- 172.21.15.86:0/2358038523 wait complete. 2024-02-19T22:07:49.581 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T22:07:50.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:07:49 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:07:47.970024+0000 mgr.smithi053.fyhcdi (mgr.14162) 347 : cluster [DBG] pgmap v331: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:07:50.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:07:49 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:07:49.579496+0000 mon.smithi053 (mon.0) 324 : audit [DBG] from='client.? 172.21.15.86:0/2358038523' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:07:51.210 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:07:51.210 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:07:52.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:07:51 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:07:49.970501+0000 mgr.smithi053.fyhcdi (mgr.14162) 348 : cluster [DBG] pgmap v332: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:07:53.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:53.114+0000 7f915398f700 1 -- 172.21.15.86:0/789423932 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f914c0f6510 msgr2=0x7f914c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:07:53.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:53.114+0000 7f915398f700 1 --2- 172.21.15.86:0/789423932 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f914c0f6510 0x7f914c0f6930 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 rx=0x7f913c009fb0 tx=0x7f913c009b30).stop 2024-02-19T22:07:53.117 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:53.114+0000 7f915398f700 1 -- 172.21.15.86:0/789423932 shutdown_connections 2024-02-19T22:07:53.117 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:53.114+0000 7f915398f700 1 --2- 172.21.15.86:0/789423932 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f914c0f6510 0x7f914c0f6930 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:07:53.117 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:53.114+0000 7f915398f700 1 -- 172.21.15.86:0/789423932 >> 172.21.15.86:0/789423932 conn(0x7f914c0f1c70 msgr2=0x7f914c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:07:53.117 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:53.114+0000 7f915398f700 1 -- 172.21.15.86:0/789423932 shutdown_connections 2024-02-19T22:07:53.117 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:53.114+0000 7f915398f700 1 -- 172.21.15.86:0/789423932 wait complete. 2024-02-19T22:07:53.117 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:53.115+0000 7f915398f700 1 Processor -- start 2024-02-19T22:07:53.117 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:53.115+0000 7f915398f700 1 -- start start 2024-02-19T22:07:53.117 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:53.115+0000 7f915398f700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f914c0f6510 0x7f914c06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:07:53.117 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:53.115+0000 7f915398f700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f914c069e20 con 0x7f914c0f6510 2024-02-19T22:07:53.117 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:53.115+0000 7f915172b700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f914c0f6510 0x7f914c06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:07:53.117 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:53.115+0000 7f915172b700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f914c0f6510 0x7f914c06b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:60828/0 (socket says 172.21.15.86:60828) 2024-02-19T22:07:53.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:53.115+0000 7f915172b700 1 -- 172.21.15.86:0/4243684005 learned_addr learned my addr 172.21.15.86:0/4243684005 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T22:07:53.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:53.116+0000 7f915172b700 1 -- 172.21.15.86:0/4243684005 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f913c005040 con 0x7f914c0f6510 2024-02-19T22:07:53.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:53.116+0000 7f915172b700 1 --2- 172.21.15.86:0/4243684005 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f914c0f6510 0x7f914c06b9a0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 rx=0x7f913c005010 tx=0x7f913c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:07:53.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:53.116+0000 7f9141ffb700 1 -- 172.21.15.86:0/4243684005 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f913c009320 con 0x7f914c0f6510 2024-02-19T22:07:53.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:53.116+0000 7f915398f700 1 -- 172.21.15.86:0/4243684005 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f914c069fc0 con 0x7f914c0f6510 2024-02-19T22:07:53.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:53.116+0000 7f9141ffb700 1 -- 172.21.15.86:0/4243684005 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f913c013380 con 0x7f914c0f6510 2024-02-19T22:07:53.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:53.116+0000 7f9141ffb700 1 -- 172.21.15.86:0/4243684005 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f913c014e50 con 0x7f914c0f6510 2024-02-19T22:07:53.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:53.116+0000 7f915398f700 1 -- 172.21.15.86:0/4243684005 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f914c06a450 con 0x7f914c0f6510 2024-02-19T22:07:53.119 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:53.117+0000 7f915398f700 1 -- 172.21.15.86:0/4243684005 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f914c059100 con 0x7f914c0f6510 2024-02-19T22:07:53.119 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:53.117+0000 7f9141ffb700 1 -- 172.21.15.86:0/4243684005 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f913c0095f0 con 0x7f914c0f6510 2024-02-19T22:07:53.119 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:53.117+0000 7f9141ffb700 1 --2- 172.21.15.86:0/4243684005 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f913802ff10 0x7f91380323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:07:53.119 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:53.117+0000 7f9141ffb700 1 -- 172.21.15.86:0/4243684005 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f913c040e70 con 0x7f914c0f6510 2024-02-19T22:07:53.119 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:53.117+0000 7f9150f2a700 1 --2- 172.21.15.86:0/4243684005 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f913802ff10 0x7f91380323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:07:53.119 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:53.118+0000 7f9150f2a700 1 --2- 172.21.15.86:0/4243684005 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f913802ff10 0x7f91380323d0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f91480096a0 tx=0x7f9148006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:07:53.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:53.122+0000 7f9141ffb700 1 -- 172.21.15.86:0/4243684005 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f913c014420 con 0x7f914c0f6510 2024-02-19T22:07:53.326 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:53.325+0000 7f915398f700 1 -- 172.21.15.86:0/4243684005 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f914c0fc0a0 con 0x7f914c0f6510 2024-02-19T22:07:53.326 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:53.326+0000 7f9141ffb700 1 -- 172.21.15.86:0/4243684005 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f913c010230 con 0x7f914c0f6510 2024-02-19T22:07:53.327 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T22:07:53.327 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:07:53.328 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:53.328+0000 7f915398f700 1 -- 172.21.15.86:0/4243684005 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f913802ff10 msgr2=0x7f91380323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:07:53.329 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:53.328+0000 7f915398f700 1 --2- 172.21.15.86:0/4243684005 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f913802ff10 0x7f91380323d0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f91480096a0 tx=0x7f9148006b40).stop 2024-02-19T22:07:53.329 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:53.328+0000 7f915398f700 1 -- 172.21.15.86:0/4243684005 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f914c0f6510 msgr2=0x7f914c06b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:07:53.329 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:53.328+0000 7f915398f700 1 --2- 172.21.15.86:0/4243684005 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f914c0f6510 0x7f914c06b9a0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 rx=0x7f913c005010 tx=0x7f913c00b040).stop 2024-02-19T22:07:53.329 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:53.328+0000 7f915398f700 1 -- 172.21.15.86:0/4243684005 shutdown_connections 2024-02-19T22:07:53.329 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:53.328+0000 7f915398f700 1 --2- 172.21.15.86:0/4243684005 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f913802ff10 0x7f91380323d0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:07:53.329 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:53.328+0000 7f915398f700 1 --2- 172.21.15.86:0/4243684005 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f914c0f6510 0x7f914c06b9a0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:07:53.329 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:53.328+0000 7f915398f700 1 -- 172.21.15.86:0/4243684005 >> 172.21.15.86:0/4243684005 conn(0x7f914c0f1c70 msgr2=0x7f914c0f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:07:53.329 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:53.328+0000 7f915398f700 1 -- 172.21.15.86:0/4243684005 shutdown_connections 2024-02-19T22:07:53.329 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:53.328+0000 7f915398f700 1 -- 172.21.15.86:0/4243684005 wait complete. 2024-02-19T22:07:53.329 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T22:07:54.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:07:53 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:07:51.970886+0000 mgr.smithi053.fyhcdi (mgr.14162) 349 : cluster [DBG] pgmap v333: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:07:54.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:07:53 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:07:53.327350+0000 mon.smithi053 (mon.0) 325 : audit [DBG] from='client.? 172.21.15.86:0/4243684005' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:07:54.953 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:07:54.953 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:07:56.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:07:55 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:07:53.971355+0000 mgr.smithi053.fyhcdi (mgr.14162) 350 : cluster [DBG] pgmap v334: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:07:56.901 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:56.887+0000 7f767ea3c700 1 -- 172.21.15.86:0/3804771793 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f76780f51a0 msgr2=0x7f76780f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:07:56.902 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:56.887+0000 7f767ea3c700 1 --2- 172.21.15.86:0/3804771793 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f76780f51a0 0x7f76780f55c0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 rx=0x7f7674009fb0 tx=0x7f7674009b30).stop 2024-02-19T22:07:56.902 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:56.887+0000 7f767ea3c700 1 -- 172.21.15.86:0/3804771793 shutdown_connections 2024-02-19T22:07:56.902 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:56.887+0000 7f767ea3c700 1 --2- 172.21.15.86:0/3804771793 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f76780f51a0 0x7f76780f55c0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:07:56.902 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:56.887+0000 7f767ea3c700 1 -- 172.21.15.86:0/3804771793 >> 172.21.15.86:0/3804771793 conn(0x7f76780f09a0 msgr2=0x7f76780f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:07:56.902 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:56.887+0000 7f767ea3c700 1 -- 172.21.15.86:0/3804771793 shutdown_connections 2024-02-19T22:07:56.902 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:56.887+0000 7f767ea3c700 1 -- 172.21.15.86:0/3804771793 wait complete. 2024-02-19T22:07:56.902 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:56.887+0000 7f767ea3c700 1 Processor -- start 2024-02-19T22:07:56.902 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:56.887+0000 7f767ea3c700 1 -- start start 2024-02-19T22:07:56.902 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:56.887+0000 7f767ea3c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f76780f51a0 0x7f76781054d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:07:56.902 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:56.887+0000 7f767ea3c700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76781059c0 con 0x7f76780f51a0 2024-02-19T22:07:56.902 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:56.888+0000 7f767da3a700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f76780f51a0 0x7f76781054d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:07:56.902 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:56.888+0000 7f767da3a700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f76780f51a0 0x7f76781054d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:60840/0 (socket says 172.21.15.86:60840) 2024-02-19T22:07:56.902 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:56.888+0000 7f767da3a700 1 -- 172.21.15.86:0/990151894 learned_addr learned my addr 172.21.15.86:0/990151894 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T22:07:56.903 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:56.888+0000 7f767da3a700 1 -- 172.21.15.86:0/990151894 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7674005040 con 0x7f76780f51a0 2024-02-19T22:07:56.903 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:56.888+0000 7f767da3a700 1 --2- 172.21.15.86:0/990151894 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f76780f51a0 0x7f76781054d0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 rx=0x7f76740089f0 tx=0x7f767400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:07:56.903 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:56.888+0000 7f766e7fc700 1 -- 172.21.15.86:0/990151894 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7674009320 con 0x7f76780f51a0 2024-02-19T22:07:56.903 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:56.888+0000 7f767ea3c700 1 -- 172.21.15.86:0/990151894 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7678105bc0 con 0x7f76780f51a0 2024-02-19T22:07:56.903 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:56.888+0000 7f766e7fc700 1 -- 172.21.15.86:0/990151894 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7674013380 con 0x7f76780f51a0 2024-02-19T22:07:56.903 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:56.888+0000 7f766e7fc700 1 -- 172.21.15.86:0/990151894 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7674014e20 con 0x7f76780f51a0 2024-02-19T22:07:56.903 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:56.888+0000 7f767ea3c700 1 -- 172.21.15.86:0/990151894 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7678106050 con 0x7f76780f51a0 2024-02-19T22:07:56.903 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:56.889+0000 7f766e7fc700 1 -- 172.21.15.86:0/990151894 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f76740095f0 con 0x7f76780f51a0 2024-02-19T22:07:56.903 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:56.889+0000 7f767ea3c700 1 -- 172.21.15.86:0/990151894 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7678045ad0 con 0x7f76780f51a0 2024-02-19T22:07:56.903 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:56.889+0000 7f766e7fc700 1 --2- 172.21.15.86:0/990151894 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f766402ff10 0x7f76640323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:07:56.903 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:56.889+0000 7f766e7fc700 1 -- 172.21.15.86:0/990151894 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7674040fb0 con 0x7f76780f51a0 2024-02-19T22:07:56.903 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:56.889+0000 7f767d239700 1 --2- 172.21.15.86:0/990151894 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f766402ff10 0x7f76640323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:07:56.903 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:56.890+0000 7f767d239700 1 --2- 172.21.15.86:0/990151894 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f766402ff10 0x7f76640323d0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7f76680096a0 tx=0x7f7668006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:07:56.903 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:56.894+0000 7f766e7fc700 1 -- 172.21.15.86:0/990151894 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f767406d2b0 con 0x7f76780f51a0 2024-02-19T22:07:57.099 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:57.098+0000 7f767ea3c700 1 -- 172.21.15.86:0/990151894 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7678106ea0 con 0x7f76780f51a0 2024-02-19T22:07:57.099 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:57.099+0000 7f766e7fc700 1 -- 172.21.15.86:0/990151894 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7674014420 con 0x7f76780f51a0 2024-02-19T22:07:57.099 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T22:07:57.100 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:07:57.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:57.100+0000 7f767ea3c700 1 -- 172.21.15.86:0/990151894 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f766402ff10 msgr2=0x7f76640323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:07:57.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:57.100+0000 7f767ea3c700 1 --2- 172.21.15.86:0/990151894 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f766402ff10 0x7f76640323d0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7f76680096a0 tx=0x7f7668006b40).stop 2024-02-19T22:07:57.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:57.100+0000 7f767ea3c700 1 -- 172.21.15.86:0/990151894 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f76780f51a0 msgr2=0x7f76781054d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:07:57.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:57.100+0000 7f767ea3c700 1 --2- 172.21.15.86:0/990151894 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f76780f51a0 0x7f76781054d0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 rx=0x7f76740089f0 tx=0x7f767400b040).stop 2024-02-19T22:07:57.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:57.100+0000 7f767ea3c700 1 -- 172.21.15.86:0/990151894 shutdown_connections 2024-02-19T22:07:57.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:57.100+0000 7f767ea3c700 1 --2- 172.21.15.86:0/990151894 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f766402ff10 0x7f76640323d0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:07:57.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:57.100+0000 7f767ea3c700 1 --2- 172.21.15.86:0/990151894 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f76780f51a0 0x7f76781054d0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:07:57.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:57.100+0000 7f767ea3c700 1 -- 172.21.15.86:0/990151894 >> 172.21.15.86:0/990151894 conn(0x7f76780f09a0 msgr2=0x7f76780f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:07:57.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:57.101+0000 7f767ea3c700 1 -- 172.21.15.86:0/990151894 shutdown_connections 2024-02-19T22:07:57.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:07:57.101+0000 7f767ea3c700 1 -- 172.21.15.86:0/990151894 wait complete. 2024-02-19T22:07:57.102 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T22:07:58.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:07:57 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:07:55.971836+0000 mgr.smithi053.fyhcdi (mgr.14162) 351 : cluster [DBG] pgmap v335: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:07:58.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:07:57 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:07:57.100253+0000 mon.smithi053 (mon.0) 326 : audit [DBG] from='client.? 172.21.15.86:0/990151894' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:07:58.747 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:07:58.747 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:08:00.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:07:59 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:07:57.972313+0000 mgr.smithi053.fyhcdi (mgr.14162) 352 : cluster [DBG] pgmap v336: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:08:00.589 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:00.587+0000 7f9b62bf6700 1 -- 172.21.15.86:0/3160881902 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9b5c0f5410 msgr2=0x7f9b5c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:08:00.589 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:00.587+0000 7f9b62bf6700 1 --2- 172.21.15.86:0/3160881902 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9b5c0f5410 0x7f9b5c0f5830 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 rx=0x7f9b50009fb0 tx=0x7f9b50009b30).stop 2024-02-19T22:08:00.589 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:00.587+0000 7f9b62bf6700 1 -- 172.21.15.86:0/3160881902 shutdown_connections 2024-02-19T22:08:00.589 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:00.587+0000 7f9b62bf6700 1 --2- 172.21.15.86:0/3160881902 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9b5c0f5410 0x7f9b5c0f5830 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:08:00.590 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:00.587+0000 7f9b62bf6700 1 -- 172.21.15.86:0/3160881902 >> 172.21.15.86:0/3160881902 conn(0x7f9b5c0f0b70 msgr2=0x7f9b5c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:08:00.590 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:00.587+0000 7f9b62bf6700 1 -- 172.21.15.86:0/3160881902 shutdown_connections 2024-02-19T22:08:00.590 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:00.587+0000 7f9b62bf6700 1 -- 172.21.15.86:0/3160881902 wait complete. 2024-02-19T22:08:00.590 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:00.588+0000 7f9b62bf6700 1 Processor -- start 2024-02-19T22:08:00.590 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:00.588+0000 7f9b62bf6700 1 -- start start 2024-02-19T22:08:00.590 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:00.588+0000 7f9b62bf6700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9b5c0f5410 0x7f9b5c106810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:08:00.590 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:00.588+0000 7f9b62bf6700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b5c106d00 con 0x7f9b5c0f5410 2024-02-19T22:08:00.590 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:00.588+0000 7f9b60992700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9b5c0f5410 0x7f9b5c106810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:08:00.590 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:00.588+0000 7f9b60992700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9b5c0f5410 0x7f9b5c106810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:34822/0 (socket says 172.21.15.86:34822) 2024-02-19T22:08:00.590 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:00.588+0000 7f9b60992700 1 -- 172.21.15.86:0/961058021 learned_addr learned my addr 172.21.15.86:0/961058021 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T22:08:00.590 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:00.588+0000 7f9b60992700 1 -- 172.21.15.86:0/961058021 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b50005040 con 0x7f9b5c0f5410 2024-02-19T22:08:00.590 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:00.589+0000 7f9b60992700 1 --2- 172.21.15.86:0/961058021 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9b5c0f5410 0x7f9b5c106810 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 rx=0x7f9b50005010 tx=0x7f9b5000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:08:00.590 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:00.589+0000 7f9b597fa700 1 -- 172.21.15.86:0/961058021 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9b50009320 con 0x7f9b5c0f5410 2024-02-19T22:08:00.590 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:00.589+0000 7f9b62bf6700 1 -- 172.21.15.86:0/961058021 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b5c106f00 con 0x7f9b5c0f5410 2024-02-19T22:08:00.590 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:00.589+0000 7f9b597fa700 1 -- 172.21.15.86:0/961058021 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9b50013380 con 0x7f9b5c0f5410 2024-02-19T22:08:00.591 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:00.589+0000 7f9b597fa700 1 -- 172.21.15.86:0/961058021 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9b50014e50 con 0x7f9b5c0f5410 2024-02-19T22:08:00.591 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:00.589+0000 7f9b62bf6700 1 -- 172.21.15.86:0/961058021 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b5c107390 con 0x7f9b5c0f5410 2024-02-19T22:08:00.591 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:00.590+0000 7f9b597fa700 1 -- 172.21.15.86:0/961058021 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9b500095f0 con 0x7f9b5c0f5410 2024-02-19T22:08:00.591 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:00.590+0000 7f9b62bf6700 1 -- 172.21.15.86:0/961058021 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b5c045ad0 con 0x7f9b5c0f5410 2024-02-19T22:08:00.592 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:00.590+0000 7f9b597fa700 1 --2- 172.21.15.86:0/961058021 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f9b4c02ff10 0x7f9b4c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:08:00.592 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:00.590+0000 7f9b597fa700 1 -- 172.21.15.86:0/961058021 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9b50040f80 con 0x7f9b5c0f5410 2024-02-19T22:08:00.592 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:00.590+0000 7f9b5bfff700 1 --2- 172.21.15.86:0/961058021 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f9b4c02ff10 0x7f9b4c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:08:00.592 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:00.590+0000 7f9b5bfff700 1 --2- 172.21.15.86:0/961058021 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f9b4c02ff10 0x7f9b4c0323d0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f9b480096a0 tx=0x7f9b48006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:08:00.595 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:00.595+0000 7f9b597fa700 1 -- 172.21.15.86:0/961058021 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9b50014420 con 0x7f9b5c0f5410 2024-02-19T22:08:00.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:00.800+0000 7f9b62bf6700 1 -- 172.21.15.86:0/961058021 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9b5c0fafa0 con 0x7f9b5c0f5410 2024-02-19T22:08:00.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:00.801+0000 7f9b597fa700 1 -- 172.21.15.86:0/961058021 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9b50010230 con 0x7f9b5c0f5410 2024-02-19T22:08:00.802 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T22:08:00.802 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:08:00.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:00.803+0000 7f9b62bf6700 1 -- 172.21.15.86:0/961058021 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f9b4c02ff10 msgr2=0x7f9b4c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:08:00.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:00.803+0000 7f9b62bf6700 1 --2- 172.21.15.86:0/961058021 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f9b4c02ff10 0x7f9b4c0323d0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f9b480096a0 tx=0x7f9b48006b40).stop 2024-02-19T22:08:00.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:00.803+0000 7f9b62bf6700 1 -- 172.21.15.86:0/961058021 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9b5c0f5410 msgr2=0x7f9b5c106810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:08:00.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:00.803+0000 7f9b62bf6700 1 --2- 172.21.15.86:0/961058021 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9b5c0f5410 0x7f9b5c106810 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 rx=0x7f9b50005010 tx=0x7f9b5000b040).stop 2024-02-19T22:08:00.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:00.803+0000 7f9b62bf6700 1 -- 172.21.15.86:0/961058021 shutdown_connections 2024-02-19T22:08:00.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:00.803+0000 7f9b62bf6700 1 --2- 172.21.15.86:0/961058021 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f9b4c02ff10 0x7f9b4c0323d0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:08:00.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:00.803+0000 7f9b62bf6700 1 --2- 172.21.15.86:0/961058021 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9b5c0f5410 0x7f9b5c106810 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:08:00.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:00.803+0000 7f9b62bf6700 1 -- 172.21.15.86:0/961058021 >> 172.21.15.86:0/961058021 conn(0x7f9b5c0f0b70 msgr2=0x7f9b5c05fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:08:00.805 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:00.803+0000 7f9b62bf6700 1 -- 172.21.15.86:0/961058021 shutdown_connections 2024-02-19T22:08:00.805 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:00.803+0000 7f9b62bf6700 1 -- 172.21.15.86:0/961058021 wait complete. 2024-02-19T22:08:00.805 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T22:08:02.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:08:01 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:07:59.972798+0000 mgr.smithi053.fyhcdi (mgr.14162) 353 : cluster [DBG] pgmap v337: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:08:02.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:08:01 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:08:00.802371+0000 mon.smithi053 (mon.0) 327 : audit [DBG] from='client.? 172.21.15.86:0/961058021' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:08:02.350 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:08:02.350 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:08:04.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:08:03 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:08:01.973279+0000 mgr.smithi053.fyhcdi (mgr.14162) 354 : cluster [DBG] pgmap v338: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:08:04.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:04.331+0000 7f18b9a75700 1 -- 172.21.15.86:0/657957233 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f18b40f6510 msgr2=0x7f18b40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:08:04.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:04.331+0000 7f18b9a75700 1 --2- 172.21.15.86:0/657957233 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f18b40f6510 0x7f18b40f6930 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 rx=0x7f18a8009fb0 tx=0x7f18a8009b30).stop 2024-02-19T22:08:04.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:04.331+0000 7f18b9a75700 1 -- 172.21.15.86:0/657957233 shutdown_connections 2024-02-19T22:08:04.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:04.332+0000 7f18b9a75700 1 --2- 172.21.15.86:0/657957233 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f18b40f6510 0x7f18b40f6930 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:08:04.333 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:04.332+0000 7f18b9a75700 1 -- 172.21.15.86:0/657957233 >> 172.21.15.86:0/657957233 conn(0x7f18b40f1c70 msgr2=0x7f18b40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:08:04.333 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:04.332+0000 7f18b9a75700 1 -- 172.21.15.86:0/657957233 shutdown_connections 2024-02-19T22:08:04.333 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:04.332+0000 7f18b9a75700 1 -- 172.21.15.86:0/657957233 wait complete. 2024-02-19T22:08:04.335 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:04.332+0000 7f18b9a75700 1 Processor -- start 2024-02-19T22:08:04.335 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:04.332+0000 7f18b9a75700 1 -- start start 2024-02-19T22:08:04.335 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:04.333+0000 7f18b9a75700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f18b40f6510 0x7f18b41067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:08:04.335 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:04.333+0000 7f18b9a75700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18b4106cd0 con 0x7f18b40f6510 2024-02-19T22:08:04.335 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:04.333+0000 7f18b37fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f18b40f6510 0x7f18b41067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:08:04.335 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:04.333+0000 7f18b37fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f18b40f6510 0x7f18b41067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:34842/0 (socket says 172.21.15.86:34842) 2024-02-19T22:08:04.335 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:04.333+0000 7f18b37fe700 1 -- 172.21.15.86:0/972902524 learned_addr learned my addr 172.21.15.86:0/972902524 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T22:08:04.335 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:04.333+0000 7f18b37fe700 1 -- 172.21.15.86:0/972902524 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18a8005040 con 0x7f18b40f6510 2024-02-19T22:08:04.335 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:04.333+0000 7f18b37fe700 1 --2- 172.21.15.86:0/972902524 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f18b40f6510 0x7f18b41067e0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 rx=0x7f18a80089f0 tx=0x7f18a800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:08:04.335 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:04.333+0000 7f189bfff700 1 -- 172.21.15.86:0/972902524 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f18a8009320 con 0x7f18b40f6510 2024-02-19T22:08:04.336 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:04.333+0000 7f189bfff700 1 -- 172.21.15.86:0/972902524 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f18a8013380 con 0x7f18b40f6510 2024-02-19T22:08:04.336 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:04.333+0000 7f18b9a75700 1 -- 172.21.15.86:0/972902524 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18b4106ed0 con 0x7f18b40f6510 2024-02-19T22:08:04.336 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:04.333+0000 7f189bfff700 1 -- 172.21.15.86:0/972902524 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f18a801ed80 con 0x7f18b40f6510 2024-02-19T22:08:04.336 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:04.334+0000 7f18b9a75700 1 -- 172.21.15.86:0/972902524 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18b4107360 con 0x7f18b40f6510 2024-02-19T22:08:04.336 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:04.334+0000 7f189bfff700 1 -- 172.21.15.86:0/972902524 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f18a801e420 con 0x7f18b40f6510 2024-02-19T22:08:04.336 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:04.334+0000 7f18b9a75700 1 -- 172.21.15.86:0/972902524 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18b4045ad0 con 0x7f18b40f6510 2024-02-19T22:08:04.336 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:04.334+0000 7f189bfff700 1 --2- 172.21.15.86:0/972902524 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f189c02ff60 0x7f189c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:08:04.336 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:04.334+0000 7f189bfff700 1 -- 172.21.15.86:0/972902524 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f18a8040e10 con 0x7f18b40f6510 2024-02-19T22:08:04.336 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:04.335+0000 7f18b2ffd700 1 --2- 172.21.15.86:0/972902524 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f189c02ff60 0x7f189c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:08:04.337 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:04.335+0000 7f18b2ffd700 1 --2- 172.21.15.86:0/972902524 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f189c02ff60 0x7f189c032420 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7f18a40096a0 tx=0x7f18a4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:08:04.340 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:04.339+0000 7f189bfff700 1 -- 172.21.15.86:0/972902524 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f18a801c930 con 0x7f18b40f6510 2024-02-19T22:08:04.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:04.546+0000 7f18b9a75700 1 -- 172.21.15.86:0/972902524 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f18b4108210 con 0x7f18b40f6510 2024-02-19T22:08:04.547 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:04.547+0000 7f189bfff700 1 -- 172.21.15.86:0/972902524 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f18a801cb40 con 0x7f18b40f6510 2024-02-19T22:08:04.548 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T22:08:04.548 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:08:04.549 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:04.548+0000 7f18b9a75700 1 -- 172.21.15.86:0/972902524 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f189c02ff60 msgr2=0x7f189c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:08:04.549 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:04.548+0000 7f18b9a75700 1 --2- 172.21.15.86:0/972902524 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f189c02ff60 0x7f189c032420 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7f18a40096a0 tx=0x7f18a4006b40).stop 2024-02-19T22:08:04.550 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:04.548+0000 7f18b9a75700 1 -- 172.21.15.86:0/972902524 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f18b40f6510 msgr2=0x7f18b41067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:08:04.550 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:04.548+0000 7f18b9a75700 1 --2- 172.21.15.86:0/972902524 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f18b40f6510 0x7f18b41067e0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 rx=0x7f18a80089f0 tx=0x7f18a800b040).stop 2024-02-19T22:08:04.550 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:04.548+0000 7f18b9a75700 1 -- 172.21.15.86:0/972902524 shutdown_connections 2024-02-19T22:08:04.550 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:04.548+0000 7f18b9a75700 1 --2- 172.21.15.86:0/972902524 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f189c02ff60 0x7f189c032420 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:08:04.550 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:04.548+0000 7f18b9a75700 1 --2- 172.21.15.86:0/972902524 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f18b40f6510 0x7f18b41067e0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:08:04.550 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:04.548+0000 7f18b9a75700 1 -- 172.21.15.86:0/972902524 >> 172.21.15.86:0/972902524 conn(0x7f18b40f1c70 msgr2=0x7f18b40f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:08:04.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:04.549+0000 7f18b9a75700 1 -- 172.21.15.86:0/972902524 shutdown_connections 2024-02-19T22:08:04.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:04.549+0000 7f18b9a75700 1 -- 172.21.15.86:0/972902524 wait complete. 2024-02-19T22:08:04.551 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T22:08:05.794 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:08:05 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:08:03.973573+0000 mgr.smithi053.fyhcdi (mgr.14162) 355 : cluster [DBG] pgmap v339: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:08:05.794 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:08:05 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:08:04.547867+0000 mon.smithi053 (mon.0) 328 : audit [DBG] from='client.? 172.21.15.86:0/972902524' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:08:06.321 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:08:06.321 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:08:07.794 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:08:07 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:08:05.973920+0000 mgr.smithi053.fyhcdi (mgr.14162) 356 : cluster [DBG] pgmap v340: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:08:08.153 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:08.152+0000 7f4993187700 1 -- 172.21.15.86:0/1325188725 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f498c0f5410 msgr2=0x7f498c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:08:08.153 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:08.152+0000 7f4993187700 1 --2- 172.21.15.86:0/1325188725 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f498c0f5410 0x7f498c0f5830 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 rx=0x7f4980009fb0 tx=0x7f4980009b30).stop 2024-02-19T22:08:08.153 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:08.152+0000 7f4993187700 1 -- 172.21.15.86:0/1325188725 shutdown_connections 2024-02-19T22:08:08.154 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:08.152+0000 7f4993187700 1 --2- 172.21.15.86:0/1325188725 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f498c0f5410 0x7f498c0f5830 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:08:08.154 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:08.152+0000 7f4993187700 1 -- 172.21.15.86:0/1325188725 >> 172.21.15.86:0/1325188725 conn(0x7f498c0f0b70 msgr2=0x7f498c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:08:08.154 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:08.152+0000 7f4993187700 1 -- 172.21.15.86:0/1325188725 shutdown_connections 2024-02-19T22:08:08.154 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:08.152+0000 7f4993187700 1 -- 172.21.15.86:0/1325188725 wait complete. 2024-02-19T22:08:08.154 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:08.153+0000 7f4993187700 1 Processor -- start 2024-02-19T22:08:08.154 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:08.153+0000 7f4993187700 1 -- start start 2024-02-19T22:08:08.155 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:08.153+0000 7f4993187700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f498c0f5410 0x7f498c109520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:08:08.156 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:08.153+0000 7f4993187700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f498c060240 con 0x7f498c0f5410 2024-02-19T22:08:08.156 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:08.153+0000 7f4990f23700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f498c0f5410 0x7f498c109520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:08:08.156 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:08.153+0000 7f4990f23700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f498c0f5410 0x7f498c109520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:34862/0 (socket says 172.21.15.86:34862) 2024-02-19T22:08:08.156 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:08.154+0000 7f4990f23700 1 -- 172.21.15.86:0/2452492350 learned_addr learned my addr 172.21.15.86:0/2452492350 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T22:08:08.156 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:08.154+0000 7f4990f23700 1 -- 172.21.15.86:0/2452492350 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4980005040 con 0x7f498c0f5410 2024-02-19T22:08:08.156 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:08.154+0000 7f4990f23700 1 --2- 172.21.15.86:0/2452492350 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f498c0f5410 0x7f498c109520 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 rx=0x7f4980005010 tx=0x7f498000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:08:08.156 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:08.154+0000 7f49897fa700 1 -- 172.21.15.86:0/2452492350 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4980009320 con 0x7f498c0f5410 2024-02-19T22:08:08.156 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:08.154+0000 7f49897fa700 1 -- 172.21.15.86:0/2452492350 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4980013380 con 0x7f498c0f5410 2024-02-19T22:08:08.156 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:08.154+0000 7f4993187700 1 -- 172.21.15.86:0/2452492350 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f498c060440 con 0x7f498c0f5410 2024-02-19T22:08:08.156 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:08.154+0000 7f49897fa700 1 -- 172.21.15.86:0/2452492350 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4980014e50 con 0x7f498c0f5410 2024-02-19T22:08:08.156 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:08.154+0000 7f4993187700 1 -- 172.21.15.86:0/2452492350 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f498c109bd0 con 0x7f498c0f5410 2024-02-19T22:08:08.156 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:08.155+0000 7f49897fa700 1 -- 172.21.15.86:0/2452492350 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f49800095f0 con 0x7f498c0f5410 2024-02-19T22:08:08.156 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:08.155+0000 7f4993187700 1 -- 172.21.15.86:0/2452492350 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f498c045ad0 con 0x7f498c0f5410 2024-02-19T22:08:08.157 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:08.155+0000 7f49897fa700 1 --2- 172.21.15.86:0/2452492350 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f497c02ff10 0x7f497c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:08:08.157 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:08.155+0000 7f49897fa700 1 -- 172.21.15.86:0/2452492350 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4980040fa0 con 0x7f498c0f5410 2024-02-19T22:08:08.157 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:08.155+0000 7f498bfff700 1 --2- 172.21.15.86:0/2452492350 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f497c02ff10 0x7f497c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:08:08.157 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:08.156+0000 7f498bfff700 1 --2- 172.21.15.86:0/2452492350 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f497c02ff10 0x7f497c0323d0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f49780096a0 tx=0x7f4978006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:08:08.160 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:08.160+0000 7f49897fa700 1 -- 172.21.15.86:0/2452492350 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4980014420 con 0x7f498c0f5410 2024-02-19T22:08:08.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:08.363+0000 7f4993187700 1 -- 172.21.15.86:0/2452492350 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f498c10a3f0 con 0x7f498c0f5410 2024-02-19T22:08:08.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:08.364+0000 7f49897fa700 1 -- 172.21.15.86:0/2452492350 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4980010230 con 0x7f498c0f5410 2024-02-19T22:08:08.365 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T22:08:08.365 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:08:08.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:08.365+0000 7f4993187700 1 -- 172.21.15.86:0/2452492350 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f497c02ff10 msgr2=0x7f497c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:08:08.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:08.365+0000 7f4993187700 1 --2- 172.21.15.86:0/2452492350 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f497c02ff10 0x7f497c0323d0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f49780096a0 tx=0x7f4978006b40).stop 2024-02-19T22:08:08.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:08.365+0000 7f4993187700 1 -- 172.21.15.86:0/2452492350 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f498c0f5410 msgr2=0x7f498c109520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:08:08.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:08.365+0000 7f4993187700 1 --2- 172.21.15.86:0/2452492350 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f498c0f5410 0x7f498c109520 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 rx=0x7f4980005010 tx=0x7f498000b040).stop 2024-02-19T22:08:08.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:08.366+0000 7f4993187700 1 -- 172.21.15.86:0/2452492350 shutdown_connections 2024-02-19T22:08:08.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:08.366+0000 7f4993187700 1 --2- 172.21.15.86:0/2452492350 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f497c02ff10 0x7f497c0323d0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:08:08.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:08.366+0000 7f4993187700 1 --2- 172.21.15.86:0/2452492350 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f498c0f5410 0x7f498c109520 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:08:08.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:08.366+0000 7f4993187700 1 -- 172.21.15.86:0/2452492350 >> 172.21.15.86:0/2452492350 conn(0x7f498c0f0b70 msgr2=0x7f498c0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:08:08.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:08.366+0000 7f4993187700 1 -- 172.21.15.86:0/2452492350 shutdown_connections 2024-02-19T22:08:08.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:08.366+0000 7f4993187700 1 -- 172.21.15.86:0/2452492350 wait complete. 2024-02-19T22:08:08.367 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T22:08:08.794 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:08:08 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:08:08.365295+0000 mon.smithi053 (mon.0) 329 : audit [DBG] from='client.? 172.21.15.86:0/2452492350' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:08:09.794 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:08:09 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:08:07.974394+0000 mgr.smithi053.fyhcdi (mgr.14162) 357 : cluster [DBG] pgmap v341: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:08:10.030 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:08:10.030 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:08:11.998 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:11.996+0000 7fab6bdb7700 1 -- 172.21.15.86:0/2860507777 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fab640f5410 msgr2=0x7fab640f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:08:11.998 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:11.996+0000 7fab6bdb7700 1 --2- 172.21.15.86:0/2860507777 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fab640f5410 0x7fab640f5830 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 rx=0x7fab58009fb0 tx=0x7fab58009b30).stop 2024-02-19T22:08:11.999 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:11.996+0000 7fab6bdb7700 1 -- 172.21.15.86:0/2860507777 shutdown_connections 2024-02-19T22:08:11.999 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:11.996+0000 7fab6bdb7700 1 --2- 172.21.15.86:0/2860507777 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fab640f5410 0x7fab640f5830 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:08:11.999 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:11.996+0000 7fab6bdb7700 1 -- 172.21.15.86:0/2860507777 >> 172.21.15.86:0/2860507777 conn(0x7fab640f0b70 msgr2=0x7fab640f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:08:11.999 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:11.997+0000 7fab6bdb7700 1 -- 172.21.15.86:0/2860507777 shutdown_connections 2024-02-19T22:08:11.999 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:11.997+0000 7fab6bdb7700 1 -- 172.21.15.86:0/2860507777 wait complete. 2024-02-19T22:08:11.999 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:11.997+0000 7fab6bdb7700 1 Processor -- start 2024-02-19T22:08:11.999 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:11.997+0000 7fab6bdb7700 1 -- start start 2024-02-19T22:08:11.999 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:11.997+0000 7fab6bdb7700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fab640f5410 0x7fab641067d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:08:11.999 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:11.997+0000 7fab6bdb7700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab64106cc0 con 0x7fab640f5410 2024-02-19T22:08:11.999 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:11.997+0000 7fab69b53700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fab640f5410 0x7fab641067d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:08:11.999 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:11.997+0000 7fab69b53700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fab640f5410 0x7fab641067d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:57038/0 (socket says 172.21.15.86:57038) 2024-02-19T22:08:12.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:11.997+0000 7fab69b53700 1 -- 172.21.15.86:0/2083644580 learned_addr learned my addr 172.21.15.86:0/2083644580 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T22:08:12.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:11.998+0000 7fab69b53700 1 -- 172.21.15.86:0/2083644580 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab58005040 con 0x7fab640f5410 2024-02-19T22:08:12.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:11.998+0000 7fab69b53700 1 --2- 172.21.15.86:0/2083644580 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fab640f5410 0x7fab641067d0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 rx=0x7fab58005010 tx=0x7fab5800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:08:12.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:11.998+0000 7fab567fc700 1 -- 172.21.15.86:0/2083644580 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fab58009320 con 0x7fab640f5410 2024-02-19T22:08:12.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:11.998+0000 7fab6bdb7700 1 -- 172.21.15.86:0/2083644580 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab64106ec0 con 0x7fab640f5410 2024-02-19T22:08:12.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:11.998+0000 7fab567fc700 1 -- 172.21.15.86:0/2083644580 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fab58013380 con 0x7fab640f5410 2024-02-19T22:08:12.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:11.998+0000 7fab567fc700 1 -- 172.21.15.86:0/2083644580 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fab58014e50 con 0x7fab640f5410 2024-02-19T22:08:12.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:11.998+0000 7fab6bdb7700 1 -- 172.21.15.86:0/2083644580 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab64107350 con 0x7fab640f5410 2024-02-19T22:08:12.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:11.999+0000 7fab567fc700 1 -- 172.21.15.86:0/2083644580 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fab580095f0 con 0x7fab640f5410 2024-02-19T22:08:12.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:11.999+0000 7fab6bdb7700 1 -- 172.21.15.86:0/2083644580 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab64045ad0 con 0x7fab640f5410 2024-02-19T22:08:12.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:11.999+0000 7fab567fc700 1 --2- 172.21.15.86:0/2083644580 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fab5002ff10 0x7fab500323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:08:12.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:11.999+0000 7fab567fc700 1 -- 172.21.15.86:0/2083644580 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fab58040ed0 con 0x7fab640f5410 2024-02-19T22:08:12.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:11.999+0000 7fab69352700 1 --2- 172.21.15.86:0/2083644580 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fab5002ff10 0x7fab500323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:08:12.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:12.000+0000 7fab69352700 1 --2- 172.21.15.86:0/2083644580 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fab5002ff10 0x7fab500323d0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7fab600096a0 tx=0x7fab60006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:08:12.004 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:12.004+0000 7fab567fc700 1 -- 172.21.15.86:0/2083644580 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fab58014420 con 0x7fab640f5410 2024-02-19T22:08:12.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:08:11 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:08:09.974846+0000 mgr.smithi053.fyhcdi (mgr.14162) 358 : cluster [DBG] pgmap v342: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:08:12.208 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:12.208+0000 7fab6bdb7700 1 -- 172.21.15.86:0/2083644580 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fab64108200 con 0x7fab640f5410 2024-02-19T22:08:12.209 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:12.209+0000 7fab567fc700 1 -- 172.21.15.86:0/2083644580 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fab58010230 con 0x7fab640f5410 2024-02-19T22:08:12.209 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T22:08:12.209 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:08:12.211 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:12.210+0000 7fab6bdb7700 1 -- 172.21.15.86:0/2083644580 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fab5002ff10 msgr2=0x7fab500323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:08:12.211 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:12.210+0000 7fab6bdb7700 1 --2- 172.21.15.86:0/2083644580 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fab5002ff10 0x7fab500323d0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7fab600096a0 tx=0x7fab60006b40).stop 2024-02-19T22:08:12.211 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:12.210+0000 7fab6bdb7700 1 -- 172.21.15.86:0/2083644580 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fab640f5410 msgr2=0x7fab641067d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:08:12.211 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:12.210+0000 7fab6bdb7700 1 --2- 172.21.15.86:0/2083644580 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fab640f5410 0x7fab641067d0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 rx=0x7fab58005010 tx=0x7fab5800b040).stop 2024-02-19T22:08:12.211 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:12.210+0000 7fab6bdb7700 1 -- 172.21.15.86:0/2083644580 shutdown_connections 2024-02-19T22:08:12.211 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:12.210+0000 7fab6bdb7700 1 --2- 172.21.15.86:0/2083644580 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fab5002ff10 0x7fab500323d0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:08:12.211 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:12.210+0000 7fab6bdb7700 1 --2- 172.21.15.86:0/2083644580 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fab640f5410 0x7fab641067d0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:08:12.211 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:12.210+0000 7fab6bdb7700 1 -- 172.21.15.86:0/2083644580 >> 172.21.15.86:0/2083644580 conn(0x7fab640f0b70 msgr2=0x7fab6405fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:08:12.211 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:12.210+0000 7fab6bdb7700 1 -- 172.21.15.86:0/2083644580 shutdown_connections 2024-02-19T22:08:12.211 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:12.210+0000 7fab6bdb7700 1 -- 172.21.15.86:0/2083644580 wait complete. 2024-02-19T22:08:12.212 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T22:08:13.047 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:08:12 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:08:12.209847+0000 mon.smithi053 (mon.0) 330 : audit [DBG] from='client.? 172.21.15.86:0/2083644580' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:08:13.805 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:08:13.805 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:08:14.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:08:13 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:08:11.975304+0000 mgr.smithi053.fyhcdi (mgr.14162) 359 : cluster [DBG] pgmap v343: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:08:15.756 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:15.753+0000 7f519408d700 1 -- 172.21.15.86:0/1802255086 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f518c0f5410 msgr2=0x7f518c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:08:15.756 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:15.753+0000 7f519408d700 1 --2- 172.21.15.86:0/1802255086 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f518c0f5410 0x7f518c0f5830 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 rx=0x7f517c009fb0 tx=0x7f517c009b30).stop 2024-02-19T22:08:15.756 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:15.753+0000 7f519408d700 1 -- 172.21.15.86:0/1802255086 shutdown_connections 2024-02-19T22:08:15.756 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:15.753+0000 7f519408d700 1 --2- 172.21.15.86:0/1802255086 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f518c0f5410 0x7f518c0f5830 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:08:15.756 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:15.753+0000 7f519408d700 1 -- 172.21.15.86:0/1802255086 >> 172.21.15.86:0/1802255086 conn(0x7f518c0f0b70 msgr2=0x7f518c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:08:15.756 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:15.753+0000 7f519408d700 1 -- 172.21.15.86:0/1802255086 shutdown_connections 2024-02-19T22:08:15.756 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:15.753+0000 7f519408d700 1 -- 172.21.15.86:0/1802255086 wait complete. 2024-02-19T22:08:15.756 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:15.754+0000 7f519408d700 1 Processor -- start 2024-02-19T22:08:15.756 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:15.754+0000 7f519408d700 1 -- start start 2024-02-19T22:08:15.756 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:15.754+0000 7f519408d700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f518c0f5410 0x7f518c10b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:08:15.756 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:15.754+0000 7f519408d700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f518c1072f0 con 0x7f518c0f5410 2024-02-19T22:08:15.757 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:15.754+0000 7f5191e29700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f518c0f5410 0x7f518c10b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:08:15.757 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:15.754+0000 7f5191e29700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f518c0f5410 0x7f518c10b900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:57064/0 (socket says 172.21.15.86:57064) 2024-02-19T22:08:15.757 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:15.754+0000 7f5191e29700 1 -- 172.21.15.86:0/3485384816 learned_addr learned my addr 172.21.15.86:0/3485384816 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T22:08:15.757 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:15.755+0000 7f5191e29700 1 -- 172.21.15.86:0/3485384816 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f517c005040 con 0x7f518c0f5410 2024-02-19T22:08:15.757 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:15.755+0000 7f5191e29700 1 --2- 172.21.15.86:0/3485384816 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f518c0f5410 0x7f518c10b900 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 rx=0x7f517c005010 tx=0x7f517c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:08:15.757 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:15.755+0000 7f51827fc700 1 -- 172.21.15.86:0/3485384816 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f517c009320 con 0x7f518c0f5410 2024-02-19T22:08:15.757 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:15.755+0000 7f519408d700 1 -- 172.21.15.86:0/3485384816 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f518c060220 con 0x7f518c0f5410 2024-02-19T22:08:15.757 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:15.755+0000 7f51827fc700 1 -- 172.21.15.86:0/3485384816 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f517c013380 con 0x7f518c0f5410 2024-02-19T22:08:15.757 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:15.755+0000 7f51827fc700 1 -- 172.21.15.86:0/3485384816 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f517c014e50 con 0x7f518c0f5410 2024-02-19T22:08:15.757 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:15.755+0000 7f519408d700 1 -- 172.21.15.86:0/3485384816 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f518c10bdf0 con 0x7f518c0f5410 2024-02-19T22:08:15.758 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:15.756+0000 7f51827fc700 1 -- 172.21.15.86:0/3485384816 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f517c0263a0 con 0x7f518c0f5410 2024-02-19T22:08:15.758 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:15.756+0000 7f51827fc700 1 --2- 172.21.15.86:0/3485384816 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f517802ff10 0x7f51780323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:08:15.758 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:15.756+0000 7f519408d700 1 -- 172.21.15.86:0/3485384816 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f518c045ad0 con 0x7f518c0f5410 2024-02-19T22:08:15.758 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:15.756+0000 7f51827fc700 1 -- 172.21.15.86:0/3485384816 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f517c040e80 con 0x7f518c0f5410 2024-02-19T22:08:15.759 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:15.756+0000 7f5191628700 1 --2- 172.21.15.86:0/3485384816 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f517802ff10 0x7f51780323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:08:15.759 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:15.757+0000 7f5191628700 1 --2- 172.21.15.86:0/3485384816 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f517802ff10 0x7f51780323d0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f51880096a0 tx=0x7f5188006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:08:15.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:15.761+0000 7f51827fc700 1 -- 172.21.15.86:0/3485384816 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f517c012920 con 0x7f518c0f5410 2024-02-19T22:08:15.967 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:15.967+0000 7f519408d700 1 -- 172.21.15.86:0/3485384816 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f518c10c5e0 con 0x7f518c0f5410 2024-02-19T22:08:15.968 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:15.967+0000 7f51827fc700 1 -- 172.21.15.86:0/3485384816 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f517c02c090 con 0x7f518c0f5410 2024-02-19T22:08:15.968 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T22:08:15.968 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:08:15.970 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:15.969+0000 7f519408d700 1 -- 172.21.15.86:0/3485384816 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f517802ff10 msgr2=0x7f51780323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:08:15.970 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:15.969+0000 7f519408d700 1 --2- 172.21.15.86:0/3485384816 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f517802ff10 0x7f51780323d0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f51880096a0 tx=0x7f5188006b40).stop 2024-02-19T22:08:15.970 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:15.969+0000 7f519408d700 1 -- 172.21.15.86:0/3485384816 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f518c0f5410 msgr2=0x7f518c10b900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:08:15.970 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:15.969+0000 7f519408d700 1 --2- 172.21.15.86:0/3485384816 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f518c0f5410 0x7f518c10b900 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 rx=0x7f517c005010 tx=0x7f517c00b040).stop 2024-02-19T22:08:15.970 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:15.969+0000 7f519408d700 1 -- 172.21.15.86:0/3485384816 shutdown_connections 2024-02-19T22:08:15.970 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:15.969+0000 7f519408d700 1 --2- 172.21.15.86:0/3485384816 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f517802ff10 0x7f51780323d0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:08:15.970 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:15.969+0000 7f519408d700 1 --2- 172.21.15.86:0/3485384816 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f518c0f5410 0x7f518c10b900 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:08:15.970 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:15.969+0000 7f519408d700 1 -- 172.21.15.86:0/3485384816 >> 172.21.15.86:0/3485384816 conn(0x7f518c0f0b70 msgr2=0x7f518c0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:08:15.970 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:15.969+0000 7f519408d700 1 -- 172.21.15.86:0/3485384816 shutdown_connections 2024-02-19T22:08:15.970 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:15.969+0000 7f519408d700 1 -- 172.21.15.86:0/3485384816 wait complete. 2024-02-19T22:08:15.971 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T22:08:16.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:08:15 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:08:13.975747+0000 mgr.smithi053.fyhcdi (mgr.14162) 360 : cluster [DBG] pgmap v344: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:08:17.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:08:16 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:08:15.968682+0000 mon.smithi053 (mon.0) 331 : audit [DBG] from='client.? 172.21.15.86:0/3485384816' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:08:17.638 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:08:17.639 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:08:18.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:08:17 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:08:15.976086+0000 mgr.smithi053.fyhcdi (mgr.14162) 361 : cluster [DBG] pgmap v345: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:08:19.573 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:19.571+0000 7f4732d0b700 1 -- 172.21.15.86:0/1340933685 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f472c0f51a0 msgr2=0x7f472c0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:08:19.573 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:19.571+0000 7f4732d0b700 1 --2- 172.21.15.86:0/1340933685 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f472c0f51a0 0x7f472c0f55c0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 rx=0x7f4728009fb0 tx=0x7f4728009b30).stop 2024-02-19T22:08:19.574 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:19.571+0000 7f4732d0b700 1 -- 172.21.15.86:0/1340933685 shutdown_connections 2024-02-19T22:08:19.574 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:19.571+0000 7f4732d0b700 1 --2- 172.21.15.86:0/1340933685 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f472c0f51a0 0x7f472c0f55c0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:08:19.574 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:19.571+0000 7f4732d0b700 1 -- 172.21.15.86:0/1340933685 >> 172.21.15.86:0/1340933685 conn(0x7f472c0f09a0 msgr2=0x7f472c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:08:19.574 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:19.571+0000 7f4732d0b700 1 -- 172.21.15.86:0/1340933685 shutdown_connections 2024-02-19T22:08:19.574 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:19.571+0000 7f4732d0b700 1 -- 172.21.15.86:0/1340933685 wait complete. 2024-02-19T22:08:19.574 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:19.572+0000 7f4732d0b700 1 Processor -- start 2024-02-19T22:08:19.574 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:19.572+0000 7f4732d0b700 1 -- start start 2024-02-19T22:08:19.574 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:19.572+0000 7f4732d0b700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f472c0f51a0 0x7f472c1065c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:08:19.574 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:19.572+0000 7f4732d0b700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f472c106ab0 con 0x7f472c0f51a0 2024-02-19T22:08:19.574 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:19.572+0000 7f4731d09700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f472c0f51a0 0x7f472c1065c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:08:19.574 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:19.572+0000 7f4731d09700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f472c0f51a0 0x7f472c1065c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:35726/0 (socket says 172.21.15.86:35726) 2024-02-19T22:08:19.574 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:19.572+0000 7f4731d09700 1 -- 172.21.15.86:0/3308295149 learned_addr learned my addr 172.21.15.86:0/3308295149 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T22:08:19.574 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:19.572+0000 7f4731d09700 1 -- 172.21.15.86:0/3308295149 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4728005040 con 0x7f472c0f51a0 2024-02-19T22:08:19.574 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:19.573+0000 7f4731d09700 1 --2- 172.21.15.86:0/3308295149 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f472c0f51a0 0x7f472c1065c0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 rx=0x7f4728009830 tx=0x7f472800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:08:19.574 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:19.573+0000 7f47227fc700 1 -- 172.21.15.86:0/3308295149 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4728009320 con 0x7f472c0f51a0 2024-02-19T22:08:19.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:19.573+0000 7f47227fc700 1 -- 172.21.15.86:0/3308295149 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4728013380 con 0x7f472c0f51a0 2024-02-19T22:08:19.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:19.573+0000 7f4732d0b700 1 -- 172.21.15.86:0/3308295149 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f472c106cb0 con 0x7f472c0f51a0 2024-02-19T22:08:19.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:19.573+0000 7f47227fc700 1 -- 172.21.15.86:0/3308295149 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4728014420 con 0x7f472c0f51a0 2024-02-19T22:08:19.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:19.573+0000 7f4732d0b700 1 -- 172.21.15.86:0/3308295149 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f472c107140 con 0x7f472c0f51a0 2024-02-19T22:08:19.576 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:19.574+0000 7f47227fc700 1 -- 172.21.15.86:0/3308295149 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f472801e420 con 0x7f472c0f51a0 2024-02-19T22:08:19.576 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:19.574+0000 7f4732d0b700 1 -- 172.21.15.86:0/3308295149 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f472c045ad0 con 0x7f472c0f51a0 2024-02-19T22:08:19.576 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:19.574+0000 7f47227fc700 1 --2- 172.21.15.86:0/3308295149 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f471802ff60 0x7f4718032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:08:19.576 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:19.574+0000 7f47227fc700 1 -- 172.21.15.86:0/3308295149 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4728040e30 con 0x7f472c0f51a0 2024-02-19T22:08:19.576 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:19.574+0000 7f4731508700 1 --2- 172.21.15.86:0/3308295149 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f471802ff60 0x7f4718032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:08:19.576 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:19.575+0000 7f4731508700 1 --2- 172.21.15.86:0/3308295149 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f471802ff60 0x7f4718032420 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f471c0096a0 tx=0x7f471c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:08:19.580 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:19.579+0000 7f47227fc700 1 -- 172.21.15.86:0/3308295149 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f472801cb10 con 0x7f472c0f51a0 2024-02-19T22:08:19.785 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:19.785+0000 7f4732d0b700 1 -- 172.21.15.86:0/3308295149 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f472c0fad30 con 0x7f472c0f51a0 2024-02-19T22:08:19.786 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:19.785+0000 7f47227fc700 1 -- 172.21.15.86:0/3308295149 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4728004490 con 0x7f472c0f51a0 2024-02-19T22:08:19.786 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T22:08:19.786 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:08:19.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:19.787+0000 7f4732d0b700 1 -- 172.21.15.86:0/3308295149 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f471802ff60 msgr2=0x7f4718032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:08:19.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:19.787+0000 7f4732d0b700 1 --2- 172.21.15.86:0/3308295149 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f471802ff60 0x7f4718032420 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f471c0096a0 tx=0x7f471c006b40).stop 2024-02-19T22:08:19.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:19.787+0000 7f4732d0b700 1 -- 172.21.15.86:0/3308295149 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f472c0f51a0 msgr2=0x7f472c1065c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:08:19.788 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:19.787+0000 7f4732d0b700 1 --2- 172.21.15.86:0/3308295149 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f472c0f51a0 0x7f472c1065c0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 rx=0x7f4728009830 tx=0x7f472800b040).stop 2024-02-19T22:08:19.788 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:19.787+0000 7f4732d0b700 1 -- 172.21.15.86:0/3308295149 shutdown_connections 2024-02-19T22:08:19.788 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:19.787+0000 7f4732d0b700 1 --2- 172.21.15.86:0/3308295149 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f471802ff60 0x7f4718032420 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:08:19.788 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:19.787+0000 7f4732d0b700 1 --2- 172.21.15.86:0/3308295149 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f472c0f51a0 0x7f472c1065c0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:08:19.788 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:19.787+0000 7f4732d0b700 1 -- 172.21.15.86:0/3308295149 >> 172.21.15.86:0/3308295149 conn(0x7f472c0f09a0 msgr2=0x7f472c05fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:08:19.788 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:19.787+0000 7f4732d0b700 1 -- 172.21.15.86:0/3308295149 shutdown_connections 2024-02-19T22:08:19.788 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:19.787+0000 7f4732d0b700 1 -- 172.21.15.86:0/3308295149 wait complete. 2024-02-19T22:08:19.788 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T22:08:20.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:08:19 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:08:17.976531+0000 mgr.smithi053.fyhcdi (mgr.14162) 362 : cluster [DBG] pgmap v346: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:08:20.948 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:08:20 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:08:19.786531+0000 mon.smithi053 (mon.0) 332 : audit [DBG] from='client.? 172.21.15.86:0/3308295149' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:08:21.482 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:08:21.482 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:08:22.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:08:21 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:08:19.976920+0000 mgr.smithi053.fyhcdi (mgr.14162) 363 : cluster [DBG] pgmap v347: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:08:23.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:23.357+0000 7f9bbd40d700 1 -- 172.21.15.86:0/589058201 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9bb80f4740 msgr2=0x7f9bb80f4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:08:23.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:23.357+0000 7f9bbd40d700 1 --2- 172.21.15.86:0/589058201 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9bb80f4740 0x7f9bb80f4b60 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 rx=0x7f9bac009fb0 tx=0x7f9bac009b30).stop 2024-02-19T22:08:23.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:23.357+0000 7f9bbd40d700 1 -- 172.21.15.86:0/589058201 shutdown_connections 2024-02-19T22:08:23.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:23.357+0000 7f9bbd40d700 1 --2- 172.21.15.86:0/589058201 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9bb80f4740 0x7f9bb80f4b60 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:08:23.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:23.357+0000 7f9bbd40d700 1 -- 172.21.15.86:0/589058201 >> 172.21.15.86:0/589058201 conn(0x7f9bb80f0260 msgr2=0x7f9bb80f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:08:23.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:23.357+0000 7f9bbd40d700 1 -- 172.21.15.86:0/589058201 shutdown_connections 2024-02-19T22:08:23.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:23.357+0000 7f9bbd40d700 1 -- 172.21.15.86:0/589058201 wait complete. 2024-02-19T22:08:23.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:23.358+0000 7f9bbd40d700 1 Processor -- start 2024-02-19T22:08:23.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:23.358+0000 7f9bbd40d700 1 -- start start 2024-02-19T22:08:23.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:23.358+0000 7f9bbd40d700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9bb80f4740 0x7f9bb81067b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:08:23.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:23.358+0000 7f9bbd40d700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bb8106ca0 con 0x7f9bb80f4740 2024-02-19T22:08:23.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:23.359+0000 7f9bb6ffd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9bb80f4740 0x7f9bb81067b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:08:23.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:23.359+0000 7f9bb6ffd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9bb80f4740 0x7f9bb81067b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:35744/0 (socket says 172.21.15.86:35744) 2024-02-19T22:08:23.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:23.359+0000 7f9bb6ffd700 1 -- 172.21.15.86:0/2677957207 learned_addr learned my addr 172.21.15.86:0/2677957207 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T22:08:23.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:23.359+0000 7f9bb6ffd700 1 -- 172.21.15.86:0/2677957207 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bac005040 con 0x7f9bb80f4740 2024-02-19T22:08:23.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:23.359+0000 7f9bb6ffd700 1 --2- 172.21.15.86:0/2677957207 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9bb80f4740 0x7f9bb81067b0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0x7f9bac0089f0 tx=0x7f9bac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:08:23.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:23.359+0000 7f9b9f7fe700 1 -- 172.21.15.86:0/2677957207 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9bac009320 con 0x7f9bb80f4740 2024-02-19T22:08:23.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:23.359+0000 7f9b9f7fe700 1 -- 172.21.15.86:0/2677957207 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9bac013380 con 0x7f9bb80f4740 2024-02-19T22:08:23.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:23.359+0000 7f9bbd40d700 1 -- 172.21.15.86:0/2677957207 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bb8106ea0 con 0x7f9bb80f4740 2024-02-19T22:08:23.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:23.359+0000 7f9b9f7fe700 1 -- 172.21.15.86:0/2677957207 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9bac01ed80 con 0x7f9bb80f4740 2024-02-19T22:08:23.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:23.359+0000 7f9bbd40d700 1 -- 172.21.15.86:0/2677957207 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bb8107330 con 0x7f9bb80f4740 2024-02-19T22:08:23.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:23.360+0000 7f9b9f7fe700 1 -- 172.21.15.86:0/2677957207 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9bac01e420 con 0x7f9bb80f4740 2024-02-19T22:08:23.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:23.360+0000 7f9bbd40d700 1 -- 172.21.15.86:0/2677957207 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bb8045ad0 con 0x7f9bb80f4740 2024-02-19T22:08:23.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:23.360+0000 7f9b9f7fe700 1 --2- 172.21.15.86:0/2677957207 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f9ba002ff60 0x7f9ba0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:08:23.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:23.360+0000 7f9b9f7fe700 1 -- 172.21.15.86:0/2677957207 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9bac041010 con 0x7f9bb80f4740 2024-02-19T22:08:23.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:23.360+0000 7f9bb67fc700 1 --2- 172.21.15.86:0/2677957207 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f9ba002ff60 0x7f9ba0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:08:23.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:23.361+0000 7f9bb67fc700 1 --2- 172.21.15.86:0/2677957207 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f9ba002ff60 0x7f9ba0032420 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f9ba80096a0 tx=0x7f9ba8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:08:23.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:23.365+0000 7f9b9f7fe700 1 -- 172.21.15.86:0/2677957207 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9bac0095f0 con 0x7f9bb80f4740 2024-02-19T22:08:23.572 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:23.571+0000 7f9bbd40d700 1 -- 172.21.15.86:0/2677957207 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9bb81081e0 con 0x7f9bb80f4740 2024-02-19T22:08:23.574 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:23.572+0000 7f9b9f7fe700 1 -- 172.21.15.86:0/2677957207 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9bac02c2d0 con 0x7f9bb80f4740 2024-02-19T22:08:23.574 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T22:08:23.575 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:08:23.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:23.574+0000 7f9bbd40d700 1 -- 172.21.15.86:0/2677957207 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f9ba002ff60 msgr2=0x7f9ba0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:08:23.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:23.574+0000 7f9bbd40d700 1 --2- 172.21.15.86:0/2677957207 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f9ba002ff60 0x7f9ba0032420 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f9ba80096a0 tx=0x7f9ba8006b40).stop 2024-02-19T22:08:23.576 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:23.574+0000 7f9bbd40d700 1 -- 172.21.15.86:0/2677957207 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9bb80f4740 msgr2=0x7f9bb81067b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:08:23.576 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:23.574+0000 7f9bbd40d700 1 --2- 172.21.15.86:0/2677957207 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9bb80f4740 0x7f9bb81067b0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0x7f9bac0089f0 tx=0x7f9bac00b040).stop 2024-02-19T22:08:23.576 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:23.574+0000 7f9bbd40d700 1 -- 172.21.15.86:0/2677957207 shutdown_connections 2024-02-19T22:08:23.576 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:23.574+0000 7f9bbd40d700 1 --2- 172.21.15.86:0/2677957207 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f9ba002ff60 0x7f9ba0032420 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:08:23.576 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:23.574+0000 7f9bbd40d700 1 --2- 172.21.15.86:0/2677957207 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9bb80f4740 0x7f9bb81067b0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:08:23.576 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:23.574+0000 7f9bbd40d700 1 -- 172.21.15.86:0/2677957207 >> 172.21.15.86:0/2677957207 conn(0x7f9bb80f0260 msgr2=0x7f9bb80f0cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:08:23.576 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:23.574+0000 7f9bbd40d700 1 -- 172.21.15.86:0/2677957207 shutdown_connections 2024-02-19T22:08:23.576 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:23.574+0000 7f9bbd40d700 1 -- 172.21.15.86:0/2677957207 wait complete. 2024-02-19T22:08:23.576 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T22:08:24.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:08:23 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:08:21.977320+0000 mgr.smithi053.fyhcdi (mgr.14162) 364 : cluster [DBG] pgmap v348: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:08:25.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:08:24 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:08:23.573262+0000 mon.smithi053 (mon.0) 333 : audit [DBG] from='client.? 172.21.15.86:0/2677957207' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:08:25.267 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:08:25.267 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:08:26.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:08:25 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:08:23.977652+0000 mgr.smithi053.fyhcdi (mgr.14162) 365 : cluster [DBG] pgmap v349: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:08:27.198 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:27.188+0000 7f80c9e0f700 1 -- 172.21.15.86:0/3353227001 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f80c40f6510 msgr2=0x7f80c40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:08:27.198 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:27.188+0000 7f80c9e0f700 1 --2- 172.21.15.86:0/3353227001 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f80c40f6510 0x7f80c40f6930 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 rx=0x7f80b4009fb0 tx=0x7f80b4009b30).stop 2024-02-19T22:08:27.198 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:27.188+0000 7f80c9e0f700 1 -- 172.21.15.86:0/3353227001 shutdown_connections 2024-02-19T22:08:27.198 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:27.188+0000 7f80c9e0f700 1 --2- 172.21.15.86:0/3353227001 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f80c40f6510 0x7f80c40f6930 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:08:27.198 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:27.188+0000 7f80c9e0f700 1 -- 172.21.15.86:0/3353227001 >> 172.21.15.86:0/3353227001 conn(0x7f80c40f1c70 msgr2=0x7f80c40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:08:27.198 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:27.188+0000 7f80c9e0f700 1 -- 172.21.15.86:0/3353227001 shutdown_connections 2024-02-19T22:08:27.199 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:27.188+0000 7f80c9e0f700 1 -- 172.21.15.86:0/3353227001 wait complete. 2024-02-19T22:08:27.199 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:27.188+0000 7f80c9e0f700 1 Processor -- start 2024-02-19T22:08:27.199 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:27.189+0000 7f80c9e0f700 1 -- start start 2024-02-19T22:08:27.199 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:27.189+0000 7f80c9e0f700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f80c40f6510 0x7f80c41067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:08:27.199 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:27.189+0000 7f80c9e0f700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80c4106cd0 con 0x7f80c40f6510 2024-02-19T22:08:27.199 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:27.189+0000 7f80c37fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f80c40f6510 0x7f80c41067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:08:27.199 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:27.189+0000 7f80c37fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f80c40f6510 0x7f80c41067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:35764/0 (socket says 172.21.15.86:35764) 2024-02-19T22:08:27.199 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:27.189+0000 7f80c37fe700 1 -- 172.21.15.86:0/2613217110 learned_addr learned my addr 172.21.15.86:0/2613217110 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T22:08:27.199 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:27.190+0000 7f80c37fe700 1 -- 172.21.15.86:0/2613217110 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80b4005040 con 0x7f80c40f6510 2024-02-19T22:08:27.199 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:27.190+0000 7f80c37fe700 1 --2- 172.21.15.86:0/2613217110 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f80c40f6510 0x7f80c41067e0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 rx=0x7f80b4005010 tx=0x7f80b400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:08:27.199 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:27.190+0000 7f80abfff700 1 -- 172.21.15.86:0/2613217110 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f80b4009320 con 0x7f80c40f6510 2024-02-19T22:08:27.199 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:27.190+0000 7f80abfff700 1 -- 172.21.15.86:0/2613217110 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f80b4013380 con 0x7f80c40f6510 2024-02-19T22:08:27.199 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:27.190+0000 7f80abfff700 1 -- 172.21.15.86:0/2613217110 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f80b4014e50 con 0x7f80c40f6510 2024-02-19T22:08:27.199 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:27.190+0000 7f80c9e0f700 1 -- 172.21.15.86:0/2613217110 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80c4106ed0 con 0x7f80c40f6510 2024-02-19T22:08:27.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:27.190+0000 7f80c9e0f700 1 -- 172.21.15.86:0/2613217110 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80c4107360 con 0x7f80c40f6510 2024-02-19T22:08:27.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:27.191+0000 7f80abfff700 1 -- 172.21.15.86:0/2613217110 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f80b40095f0 con 0x7f80c40f6510 2024-02-19T22:08:27.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:27.191+0000 7f80c9e0f700 1 -- 172.21.15.86:0/2613217110 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80c4045ad0 con 0x7f80c40f6510 2024-02-19T22:08:27.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:27.191+0000 7f80abfff700 1 --2- 172.21.15.86:0/2613217110 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f80ac02ff10 0x7f80ac0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:08:27.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:27.191+0000 7f80abfff700 1 -- 172.21.15.86:0/2613217110 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f80b4040ee0 con 0x7f80c40f6510 2024-02-19T22:08:27.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:27.191+0000 7f80c2ffd700 1 --2- 172.21.15.86:0/2613217110 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f80ac02ff10 0x7f80ac0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:08:27.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:27.191+0000 7f80c2ffd700 1 --2- 172.21.15.86:0/2613217110 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f80ac02ff10 0x7f80ac0323d0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f80b80096a0 tx=0x7f80b8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:08:27.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:27.196+0000 7f80abfff700 1 -- 172.21.15.86:0/2613217110 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f80b4014420 con 0x7f80c40f6510 2024-02-19T22:08:27.399 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:27.399+0000 7f80c9e0f700 1 -- 172.21.15.86:0/2613217110 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f80c4108210 con 0x7f80c40f6510 2024-02-19T22:08:27.400 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:27.399+0000 7f80abfff700 1 -- 172.21.15.86:0/2613217110 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f80b401d980 con 0x7f80c40f6510 2024-02-19T22:08:27.400 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T22:08:27.400 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:08:27.401 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:27.401+0000 7f80c9e0f700 1 -- 172.21.15.86:0/2613217110 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f80ac02ff10 msgr2=0x7f80ac0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:08:27.402 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:27.401+0000 7f80c9e0f700 1 --2- 172.21.15.86:0/2613217110 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f80ac02ff10 0x7f80ac0323d0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f80b80096a0 tx=0x7f80b8006b40).stop 2024-02-19T22:08:27.402 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:27.401+0000 7f80c9e0f700 1 -- 172.21.15.86:0/2613217110 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f80c40f6510 msgr2=0x7f80c41067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:08:27.402 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:27.401+0000 7f80c9e0f700 1 --2- 172.21.15.86:0/2613217110 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f80c40f6510 0x7f80c41067e0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 rx=0x7f80b4005010 tx=0x7f80b400b040).stop 2024-02-19T22:08:27.402 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:27.401+0000 7f80c9e0f700 1 -- 172.21.15.86:0/2613217110 shutdown_connections 2024-02-19T22:08:27.402 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:27.401+0000 7f80c9e0f700 1 --2- 172.21.15.86:0/2613217110 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f80ac02ff10 0x7f80ac0323d0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:08:27.402 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:27.401+0000 7f80c9e0f700 1 --2- 172.21.15.86:0/2613217110 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f80c40f6510 0x7f80c41067e0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:08:27.402 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:27.401+0000 7f80c9e0f700 1 -- 172.21.15.86:0/2613217110 >> 172.21.15.86:0/2613217110 conn(0x7f80c40f1c70 msgr2=0x7f80c40f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:08:27.402 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:27.401+0000 7f80c9e0f700 1 -- 172.21.15.86:0/2613217110 shutdown_connections 2024-02-19T22:08:27.402 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:27.401+0000 7f80c9e0f700 1 -- 172.21.15.86:0/2613217110 wait complete. 2024-02-19T22:08:27.403 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T22:08:28.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:08:27 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:08:25.978027+0000 mgr.smithi053.fyhcdi (mgr.14162) 366 : cluster [DBG] pgmap v350: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:08:28.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:08:27 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:08:27.400633+0000 mon.smithi053 (mon.0) 334 : audit [DBG] from='client.? 172.21.15.86:0/2613217110' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:08:29.039 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:08:29.039 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:08:30.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:08:29 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:08:27.978341+0000 mgr.smithi053.fyhcdi (mgr.14162) 367 : cluster [DBG] pgmap v351: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:08:31.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:31.087+0000 7f75b811b700 1 -- 172.21.15.86:0/1195158401 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f75b00f5410 msgr2=0x7f75b00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:08:31.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:31.087+0000 7f75b811b700 1 --2- 172.21.15.86:0/1195158401 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f75b00f5410 0x7f75b00f5830 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 rx=0x7f75a0009fb0 tx=0x7f75a0009b30).stop 2024-02-19T22:08:31.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:31.087+0000 7f75b811b700 1 -- 172.21.15.86:0/1195158401 shutdown_connections 2024-02-19T22:08:31.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:31.087+0000 7f75b811b700 1 --2- 172.21.15.86:0/1195158401 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f75b00f5410 0x7f75b00f5830 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:08:31.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:31.087+0000 7f75b811b700 1 -- 172.21.15.86:0/1195158401 >> 172.21.15.86:0/1195158401 conn(0x7f75b00f0b70 msgr2=0x7f75b00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:08:31.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:31.087+0000 7f75b811b700 1 -- 172.21.15.86:0/1195158401 shutdown_connections 2024-02-19T22:08:31.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:31.087+0000 7f75b811b700 1 -- 172.21.15.86:0/1195158401 wait complete. 2024-02-19T22:08:31.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:31.088+0000 7f75b811b700 1 Processor -- start 2024-02-19T22:08:31.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:31.088+0000 7f75b811b700 1 -- start start 2024-02-19T22:08:31.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:31.088+0000 7f75b811b700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f75b00f5410 0x7f75b01056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:08:31.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:31.088+0000 7f75b811b700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75b0105bd0 con 0x7f75b00f5410 2024-02-19T22:08:31.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:31.088+0000 7f75b5eb7700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f75b00f5410 0x7f75b01056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:08:31.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:31.088+0000 7f75b5eb7700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f75b00f5410 0x7f75b01056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:56062/0 (socket says 172.21.15.86:56062) 2024-02-19T22:08:31.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:31.088+0000 7f75b5eb7700 1 -- 172.21.15.86:0/4094992250 learned_addr learned my addr 172.21.15.86:0/4094992250 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T22:08:31.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:31.089+0000 7f75b5eb7700 1 -- 172.21.15.86:0/4094992250 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75a0005040 con 0x7f75b00f5410 2024-02-19T22:08:31.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:31.089+0000 7f75b5eb7700 1 --2- 172.21.15.86:0/4094992250 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f75b00f5410 0x7f75b01056e0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 rx=0x7f75a0005010 tx=0x7f75a000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:08:31.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:31.089+0000 7f75a67fc700 1 -- 172.21.15.86:0/4094992250 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f75a0009320 con 0x7f75b00f5410 2024-02-19T22:08:31.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:31.089+0000 7f75b811b700 1 -- 172.21.15.86:0/4094992250 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75b0105dd0 con 0x7f75b00f5410 2024-02-19T22:08:31.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:31.089+0000 7f75a67fc700 1 -- 172.21.15.86:0/4094992250 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f75a0013380 con 0x7f75b00f5410 2024-02-19T22:08:31.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:31.089+0000 7f75a67fc700 1 -- 172.21.15.86:0/4094992250 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f75a0014e50 con 0x7f75b00f5410 2024-02-19T22:08:31.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:31.089+0000 7f75b811b700 1 -- 172.21.15.86:0/4094992250 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75b0106260 con 0x7f75b00f5410 2024-02-19T22:08:31.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:31.090+0000 7f75a67fc700 1 -- 172.21.15.86:0/4094992250 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f75a00095f0 con 0x7f75b00f5410 2024-02-19T22:08:31.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:31.090+0000 7f75b811b700 1 -- 172.21.15.86:0/4094992250 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75b0045ad0 con 0x7f75b00f5410 2024-02-19T22:08:31.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:31.090+0000 7f75a67fc700 1 --2- 172.21.15.86:0/4094992250 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f759c02ff10 0x7f759c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:08:31.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:31.090+0000 7f75a67fc700 1 -- 172.21.15.86:0/4094992250 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f75a0040e90 con 0x7f75b00f5410 2024-02-19T22:08:31.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:31.090+0000 7f75b56b6700 1 --2- 172.21.15.86:0/4094992250 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f759c02ff10 0x7f759c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:08:31.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:31.091+0000 7f75b56b6700 1 --2- 172.21.15.86:0/4094992250 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f759c02ff10 0x7f759c0323d0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f75ac0096a0 tx=0x7f75ac006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:08:31.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:31.095+0000 7f75a67fc700 1 -- 172.21.15.86:0/4094992250 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f75a0014420 con 0x7f75b00f5410 2024-02-19T22:08:31.301 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:31.302+0000 7f75b811b700 1 -- 172.21.15.86:0/4094992250 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f75b0107110 con 0x7f75b00f5410 2024-02-19T22:08:31.302 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:31.303+0000 7f75a67fc700 1 -- 172.21.15.86:0/4094992250 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f75a0010230 con 0x7f75b00f5410 2024-02-19T22:08:31.302 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T22:08:31.302 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:08:31.305 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:31.304+0000 7f75b811b700 1 -- 172.21.15.86:0/4094992250 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f759c02ff10 msgr2=0x7f759c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:08:31.305 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:31.304+0000 7f75b811b700 1 --2- 172.21.15.86:0/4094992250 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f759c02ff10 0x7f759c0323d0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f75ac0096a0 tx=0x7f75ac006b40).stop 2024-02-19T22:08:31.305 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:31.304+0000 7f75b811b700 1 -- 172.21.15.86:0/4094992250 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f75b00f5410 msgr2=0x7f75b01056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:08:31.305 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:31.304+0000 7f75b811b700 1 --2- 172.21.15.86:0/4094992250 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f75b00f5410 0x7f75b01056e0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 rx=0x7f75a0005010 tx=0x7f75a000b040).stop 2024-02-19T22:08:31.305 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:31.304+0000 7f75b811b700 1 -- 172.21.15.86:0/4094992250 shutdown_connections 2024-02-19T22:08:31.305 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:31.304+0000 7f75b811b700 1 --2- 172.21.15.86:0/4094992250 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f759c02ff10 0x7f759c0323d0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:08:31.306 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:31.304+0000 7f75b811b700 1 --2- 172.21.15.86:0/4094992250 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f75b00f5410 0x7f75b01056e0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:08:31.306 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:31.305+0000 7f75b811b700 1 -- 172.21.15.86:0/4094992250 >> 172.21.15.86:0/4094992250 conn(0x7f75b00f0b70 msgr2=0x7f75b00f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:08:31.306 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:31.305+0000 7f75b811b700 1 -- 172.21.15.86:0/4094992250 shutdown_connections 2024-02-19T22:08:31.306 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:31.305+0000 7f75b811b700 1 -- 172.21.15.86:0/4094992250 wait complete. 2024-02-19T22:08:31.306 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T22:08:32.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:08:31 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:08:29.978780+0000 mgr.smithi053.fyhcdi (mgr.14162) 368 : cluster [DBG] pgmap v352: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:08:32.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:08:31 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:08:31.302791+0000 mon.smithi053 (mon.0) 335 : audit [DBG] from='client.? 172.21.15.86:0/4094992250' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:08:32.936 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:08:32.936 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:08:34.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:08:33 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:08:31.979186+0000 mgr.smithi053.fyhcdi (mgr.14162) 369 : cluster [DBG] pgmap v353: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:08:34.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:34.881+0000 7f50619f7700 1 -- 172.21.15.86:0/2422145846 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f505c0f5410 msgr2=0x7f505c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:08:34.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:34.881+0000 7f50619f7700 1 --2- 172.21.15.86:0/2422145846 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f505c0f5410 0x7f505c0f5830 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 rx=0x7f504c009fb0 tx=0x7f504c009b30).stop 2024-02-19T22:08:34.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:34.881+0000 7f50619f7700 1 -- 172.21.15.86:0/2422145846 shutdown_connections 2024-02-19T22:08:34.892 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:34.881+0000 7f50619f7700 1 --2- 172.21.15.86:0/2422145846 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f505c0f5410 0x7f505c0f5830 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:08:34.892 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:34.881+0000 7f50619f7700 1 -- 172.21.15.86:0/2422145846 >> 172.21.15.86:0/2422145846 conn(0x7f505c0f0b70 msgr2=0x7f505c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:08:34.892 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:34.881+0000 7f50619f7700 1 -- 172.21.15.86:0/2422145846 shutdown_connections 2024-02-19T22:08:34.892 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:34.881+0000 7f50619f7700 1 -- 172.21.15.86:0/2422145846 wait complete. 2024-02-19T22:08:34.892 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:34.881+0000 7f50619f7700 1 Processor -- start 2024-02-19T22:08:34.892 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:34.882+0000 7f50619f7700 1 -- start start 2024-02-19T22:08:34.892 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:34.882+0000 7f50619f7700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f505c0f5410 0x7f505c105720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:08:34.892 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:34.882+0000 7f50619f7700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f505c105c10 con 0x7f505c0f5410 2024-02-19T22:08:34.892 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:34.882+0000 7f505affd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f505c0f5410 0x7f505c105720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:08:34.892 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:34.882+0000 7f505affd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f505c0f5410 0x7f505c105720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:56068/0 (socket says 172.21.15.86:56068) 2024-02-19T22:08:34.892 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:34.882+0000 7f505affd700 1 -- 172.21.15.86:0/3943981286 learned_addr learned my addr 172.21.15.86:0/3943981286 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T22:08:34.892 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:34.882+0000 7f505affd700 1 -- 172.21.15.86:0/3943981286 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f504c005040 con 0x7f505c0f5410 2024-02-19T22:08:34.892 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:34.882+0000 7f505affd700 1 --2- 172.21.15.86:0/3943981286 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f505c0f5410 0x7f505c105720 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 rx=0x7f504c005010 tx=0x7f504c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:08:34.892 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:34.882+0000 7f50437fe700 1 -- 172.21.15.86:0/3943981286 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f504c009320 con 0x7f505c0f5410 2024-02-19T22:08:34.893 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:34.882+0000 7f50619f7700 1 -- 172.21.15.86:0/3943981286 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f505c105e10 con 0x7f505c0f5410 2024-02-19T22:08:34.893 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:34.883+0000 7f50437fe700 1 -- 172.21.15.86:0/3943981286 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f504c013380 con 0x7f505c0f5410 2024-02-19T22:08:34.893 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:34.883+0000 7f50437fe700 1 -- 172.21.15.86:0/3943981286 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f504c014e50 con 0x7f505c0f5410 2024-02-19T22:08:34.893 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:34.883+0000 7f50619f7700 1 -- 172.21.15.86:0/3943981286 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f505c1062a0 con 0x7f505c0f5410 2024-02-19T22:08:34.893 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:34.883+0000 7f50437fe700 1 -- 172.21.15.86:0/3943981286 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f504c0095f0 con 0x7f505c0f5410 2024-02-19T22:08:34.893 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:34.883+0000 7f50619f7700 1 -- 172.21.15.86:0/3943981286 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f505c045ad0 con 0x7f505c0f5410 2024-02-19T22:08:34.893 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:34.884+0000 7f50437fe700 1 --2- 172.21.15.86:0/3943981286 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f504402ff10 0x7f50440323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:08:34.893 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:34.884+0000 7f50437fe700 1 -- 172.21.15.86:0/3943981286 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f504c040fa0 con 0x7f505c0f5410 2024-02-19T22:08:34.893 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:34.884+0000 7f505a7fc700 1 --2- 172.21.15.86:0/3943981286 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f504402ff10 0x7f50440323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:08:34.893 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:34.884+0000 7f505a7fc700 1 --2- 172.21.15.86:0/3943981286 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f504402ff10 0x7f50440323d0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f50500096a0 tx=0x7f5050006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:08:34.893 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:34.888+0000 7f50437fe700 1 -- 172.21.15.86:0/3943981286 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f504c014420 con 0x7f505c0f5410 2024-02-19T22:08:35.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:35.095+0000 7f50619f7700 1 -- 172.21.15.86:0/3943981286 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f505c1070f0 con 0x7f505c0f5410 2024-02-19T22:08:35.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:35.095+0000 7f50437fe700 1 -- 172.21.15.86:0/3943981286 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f504c010230 con 0x7f505c0f5410 2024-02-19T22:08:35.095 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T22:08:35.095 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:08:35.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:35.097+0000 7f50619f7700 1 -- 172.21.15.86:0/3943981286 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f504402ff10 msgr2=0x7f50440323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:08:35.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:35.097+0000 7f50619f7700 1 --2- 172.21.15.86:0/3943981286 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f504402ff10 0x7f50440323d0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f50500096a0 tx=0x7f5050006b40).stop 2024-02-19T22:08:35.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:35.097+0000 7f50619f7700 1 -- 172.21.15.86:0/3943981286 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f505c0f5410 msgr2=0x7f505c105720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:08:35.097 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:35.097+0000 7f50619f7700 1 --2- 172.21.15.86:0/3943981286 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f505c0f5410 0x7f505c105720 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 rx=0x7f504c005010 tx=0x7f504c00b040).stop 2024-02-19T22:08:35.097 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:35.097+0000 7f50619f7700 1 -- 172.21.15.86:0/3943981286 shutdown_connections 2024-02-19T22:08:35.097 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:35.097+0000 7f50619f7700 1 --2- 172.21.15.86:0/3943981286 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f504402ff10 0x7f50440323d0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:08:35.097 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:35.097+0000 7f50619f7700 1 --2- 172.21.15.86:0/3943981286 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f505c0f5410 0x7f505c105720 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:08:35.097 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:35.097+0000 7f50619f7700 1 -- 172.21.15.86:0/3943981286 >> 172.21.15.86:0/3943981286 conn(0x7f505c0f0b70 msgr2=0x7f505c0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:08:35.097 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:35.097+0000 7f50619f7700 1 -- 172.21.15.86:0/3943981286 shutdown_connections 2024-02-19T22:08:35.097 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:35.097+0000 7f50619f7700 1 -- 172.21.15.86:0/3943981286 wait complete. 2024-02-19T22:08:35.097 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T22:08:36.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:08:35 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:08:33.979674+0000 mgr.smithi053.fyhcdi (mgr.14162) 370 : cluster [DBG] pgmap v354: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:08:36.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:08:35 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:08:35.095542+0000 mon.smithi053 (mon.0) 336 : audit [DBG] from='client.? 172.21.15.86:0/3943981286' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:08:36.781 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:08:36.782 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:08:37.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:08:36 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:08:35.609044+0000 mon.smithi053 (mon.0) 337 : audit [INF] from='mgr.14162 172.21.15.53:0/2860358250' entity='mgr.smithi053.fyhcdi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi053.fyhcdi/mirror_snapshot_schedule"}]: dispatch 2024-02-19T22:08:37.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:08:36 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:08:35.610884+0000 mon.smithi053 (mon.0) 338 : audit [INF] from='mgr.14162 172.21.15.53:0/2860358250' entity='mgr.smithi053.fyhcdi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi053.fyhcdi/trash_purge_schedule"}]: dispatch 2024-02-19T22:08:38.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:08:37 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:08:35.979982+0000 mgr.smithi053.fyhcdi (mgr.14162) 371 : cluster [DBG] pgmap v355: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:08:38.786 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:38.784+0000 7fe086627700 1 -- 172.21.15.86:0/1497995276 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe0800f5410 msgr2=0x7fe0800f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:08:38.786 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:38.784+0000 7fe086627700 1 --2- 172.21.15.86:0/1497995276 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe0800f5410 0x7fe0800f5830 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 rx=0x7fe074009fb0 tx=0x7fe074009b30).stop 2024-02-19T22:08:38.786 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:38.784+0000 7fe086627700 1 -- 172.21.15.86:0/1497995276 shutdown_connections 2024-02-19T22:08:38.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:38.784+0000 7fe086627700 1 --2- 172.21.15.86:0/1497995276 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe0800f5410 0x7fe0800f5830 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:08:38.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:38.784+0000 7fe086627700 1 -- 172.21.15.86:0/1497995276 >> 172.21.15.86:0/1497995276 conn(0x7fe0800f0b70 msgr2=0x7fe0800f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:08:38.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:38.784+0000 7fe086627700 1 -- 172.21.15.86:0/1497995276 shutdown_connections 2024-02-19T22:08:38.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:38.784+0000 7fe086627700 1 -- 172.21.15.86:0/1497995276 wait complete. 2024-02-19T22:08:38.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:38.784+0000 7fe086627700 1 Processor -- start 2024-02-19T22:08:38.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:38.785+0000 7fe086627700 1 -- start start 2024-02-19T22:08:38.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:38.785+0000 7fe086627700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe0800f5410 0x7fe0801056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:08:38.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:38.785+0000 7fe086627700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe080105bd0 con 0x7fe0800f5410 2024-02-19T22:08:38.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:38.785+0000 7fe07ffff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe0800f5410 0x7fe0801056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:08:38.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:38.785+0000 7fe07ffff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe0800f5410 0x7fe0801056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:56078/0 (socket says 172.21.15.86:56078) 2024-02-19T22:08:38.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:38.785+0000 7fe07ffff700 1 -- 172.21.15.86:0/3939640134 learned_addr learned my addr 172.21.15.86:0/3939640134 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T22:08:38.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:38.785+0000 7fe07ffff700 1 -- 172.21.15.86:0/3939640134 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe074005040 con 0x7fe0800f5410 2024-02-19T22:08:38.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:38.785+0000 7fe07ffff700 1 --2- 172.21.15.86:0/3939640134 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe0800f5410 0x7fe0801056e0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 rx=0x7fe074005010 tx=0x7fe07400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:08:38.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:38.785+0000 7fe07cff9700 1 -- 172.21.15.86:0/3939640134 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe074009320 con 0x7fe0800f5410 2024-02-19T22:08:38.788 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:38.786+0000 7fe07cff9700 1 -- 172.21.15.86:0/3939640134 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe074013380 con 0x7fe0800f5410 2024-02-19T22:08:38.788 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:38.786+0000 7fe086627700 1 -- 172.21.15.86:0/3939640134 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe080105dd0 con 0x7fe0800f5410 2024-02-19T22:08:38.788 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:38.786+0000 7fe07cff9700 1 -- 172.21.15.86:0/3939640134 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe074014e50 con 0x7fe0800f5410 2024-02-19T22:08:38.788 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:38.786+0000 7fe086627700 1 -- 172.21.15.86:0/3939640134 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe080106260 con 0x7fe0800f5410 2024-02-19T22:08:38.789 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:38.787+0000 7fe07cff9700 1 -- 172.21.15.86:0/3939640134 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe0740263a0 con 0x7fe0800f5410 2024-02-19T22:08:38.789 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:38.787+0000 7fe086627700 1 -- 172.21.15.86:0/3939640134 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe080045ad0 con 0x7fe0800f5410 2024-02-19T22:08:38.789 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:38.787+0000 7fe07cff9700 1 --2- 172.21.15.86:0/3939640134 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fe06802ff10 0x7fe0680323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:08:38.789 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:38.787+0000 7fe07cff9700 1 -- 172.21.15.86:0/3939640134 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe074041130 con 0x7fe0800f5410 2024-02-19T22:08:38.789 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:38.787+0000 7fe07f7fe700 1 --2- 172.21.15.86:0/3939640134 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fe06802ff10 0x7fe0680323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:08:38.789 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:38.787+0000 7fe07f7fe700 1 --2- 172.21.15.86:0/3939640134 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fe06802ff10 0x7fe0680323d0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7fe0700096a0 tx=0x7fe070006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:08:38.792 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:38.791+0000 7fe07cff9700 1 -- 172.21.15.86:0/3939640134 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe0740126f0 con 0x7fe0800f5410 2024-02-19T22:08:38.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:38.997+0000 7fe086627700 1 -- 172.21.15.86:0/3939640134 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe080107110 con 0x7fe0800f5410 2024-02-19T22:08:38.998 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:38.997+0000 7fe07cff9700 1 -- 172.21.15.86:0/3939640134 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe0740126f0 con 0x7fe0800f5410 2024-02-19T22:08:38.998 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T22:08:38.998 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:08:39.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:38.999+0000 7fe086627700 1 -- 172.21.15.86:0/3939640134 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fe06802ff10 msgr2=0x7fe0680323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:08:39.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:38.999+0000 7fe086627700 1 --2- 172.21.15.86:0/3939640134 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fe06802ff10 0x7fe0680323d0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7fe0700096a0 tx=0x7fe070006b40).stop 2024-02-19T22:08:39.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:38.999+0000 7fe086627700 1 -- 172.21.15.86:0/3939640134 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe0800f5410 msgr2=0x7fe0801056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:08:39.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:38.999+0000 7fe086627700 1 --2- 172.21.15.86:0/3939640134 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe0800f5410 0x7fe0801056e0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 rx=0x7fe074005010 tx=0x7fe07400b040).stop 2024-02-19T22:08:39.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:38.999+0000 7fe086627700 1 -- 172.21.15.86:0/3939640134 shutdown_connections 2024-02-19T22:08:39.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:38.999+0000 7fe086627700 1 --2- 172.21.15.86:0/3939640134 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fe06802ff10 0x7fe0680323d0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:08:39.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:38.999+0000 7fe086627700 1 --2- 172.21.15.86:0/3939640134 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe0800f5410 0x7fe0801056e0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:08:39.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:38.999+0000 7fe086627700 1 -- 172.21.15.86:0/3939640134 >> 172.21.15.86:0/3939640134 conn(0x7fe0800f0b70 msgr2=0x7fe0800f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:08:39.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:38.999+0000 7fe086627700 1 -- 172.21.15.86:0/3939640134 shutdown_connections 2024-02-19T22:08:39.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:38.999+0000 7fe086627700 1 -- 172.21.15.86:0/3939640134 wait complete. 2024-02-19T22:08:39.001 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T22:08:40.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:08:39 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:08:37.980249+0000 mgr.smithi053.fyhcdi (mgr.14162) 372 : cluster [DBG] pgmap v356: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:08:40.045 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:08:39 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:08:38.998686+0000 mon.smithi053 (mon.0) 339 : audit [DBG] from='client.? 172.21.15.86:0/3939640134' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:08:40.684 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:08:40.685 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:08:42.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:08:41 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:08:39.980524+0000 mgr.smithi053.fyhcdi (mgr.14162) 373 : cluster [DBG] pgmap v357: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:08:42.419 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:42.417+0000 7fe49704e700 1 -- 172.21.15.86:0/636950997 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe4900f5410 msgr2=0x7fe4900f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:08:42.420 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:42.417+0000 7fe49704e700 1 --2- 172.21.15.86:0/636950997 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe4900f5410 0x7fe4900f5830 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 rx=0x7fe484009fb0 tx=0x7fe484009b30).stop 2024-02-19T22:08:42.420 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:42.417+0000 7fe49704e700 1 -- 172.21.15.86:0/636950997 shutdown_connections 2024-02-19T22:08:42.420 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:42.417+0000 7fe49704e700 1 --2- 172.21.15.86:0/636950997 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe4900f5410 0x7fe4900f5830 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:08:42.420 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:42.417+0000 7fe49704e700 1 -- 172.21.15.86:0/636950997 >> 172.21.15.86:0/636950997 conn(0x7fe4900f0b70 msgr2=0x7fe4900f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:08:42.420 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:42.417+0000 7fe49704e700 1 -- 172.21.15.86:0/636950997 shutdown_connections 2024-02-19T22:08:42.420 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:42.417+0000 7fe49704e700 1 -- 172.21.15.86:0/636950997 wait complete. 2024-02-19T22:08:42.420 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:42.418+0000 7fe49704e700 1 Processor -- start 2024-02-19T22:08:42.420 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:42.418+0000 7fe49704e700 1 -- start start 2024-02-19T22:08:42.420 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:42.418+0000 7fe49704e700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe4900f5410 0x7fe4900fd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:08:42.420 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:42.418+0000 7fe49704e700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4900fb960 con 0x7fe4900f5410 2024-02-19T22:08:42.420 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:42.418+0000 7fe494dea700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe4900f5410 0x7fe4900fd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:08:42.420 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:42.418+0000 7fe494dea700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe4900f5410 0x7fe4900fd4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:36636/0 (socket says 172.21.15.86:36636) 2024-02-19T22:08:42.421 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:42.418+0000 7fe494dea700 1 -- 172.21.15.86:0/3719781273 learned_addr learned my addr 172.21.15.86:0/3719781273 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T22:08:42.421 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:42.419+0000 7fe494dea700 1 -- 172.21.15.86:0/3719781273 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe484005040 con 0x7fe4900f5410 2024-02-19T22:08:42.421 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:42.419+0000 7fe494dea700 1 --2- 172.21.15.86:0/3719781273 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe4900f5410 0x7fe4900fd4e0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 rx=0x7fe484005010 tx=0x7fe48400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:08:42.421 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:42.419+0000 7fe48d7fa700 1 -- 172.21.15.86:0/3719781273 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe484009320 con 0x7fe4900f5410 2024-02-19T22:08:42.421 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:42.419+0000 7fe48d7fa700 1 -- 172.21.15.86:0/3719781273 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe484013380 con 0x7fe4900f5410 2024-02-19T22:08:42.421 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:42.419+0000 7fe49704e700 1 -- 172.21.15.86:0/3719781273 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4900fbb00 con 0x7fe4900f5410 2024-02-19T22:08:42.421 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:42.419+0000 7fe48d7fa700 1 -- 172.21.15.86:0/3719781273 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe484014e50 con 0x7fe4900f5410 2024-02-19T22:08:42.421 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:42.419+0000 7fe49704e700 1 -- 172.21.15.86:0/3719781273 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4900fbf90 con 0x7fe4900f5410 2024-02-19T22:08:42.422 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:42.420+0000 7fe48d7fa700 1 -- 172.21.15.86:0/3719781273 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe4840263a0 con 0x7fe4900f5410 2024-02-19T22:08:42.422 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:42.420+0000 7fe49704e700 1 -- 172.21.15.86:0/3719781273 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe490059100 con 0x7fe4900f5410 2024-02-19T22:08:42.422 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:42.420+0000 7fe48d7fa700 1 --2- 172.21.15.86:0/3719781273 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fe48002ff10 0x7fe4800323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:08:42.422 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:42.420+0000 7fe48d7fa700 1 -- 172.21.15.86:0/3719781273 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe484040fd0 con 0x7fe4900f5410 2024-02-19T22:08:42.422 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:42.420+0000 7fe48ffff700 1 --2- 172.21.15.86:0/3719781273 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fe48002ff10 0x7fe4800323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:08:42.422 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:42.421+0000 7fe48ffff700 1 --2- 172.21.15.86:0/3719781273 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fe48002ff10 0x7fe4800323d0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7fe47c0096a0 tx=0x7fe47c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:08:42.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:42.425+0000 7fe48d7fa700 1 -- 172.21.15.86:0/3719781273 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe4840126f0 con 0x7fe4900f5410 2024-02-19T22:08:42.629 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:42.629+0000 7fe49704e700 1 -- 172.21.15.86:0/3719781273 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe4900fafa0 con 0x7fe4900f5410 2024-02-19T22:08:42.630 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:42.629+0000 7fe48d7fa700 1 -- 172.21.15.86:0/3719781273 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe484012330 con 0x7fe4900f5410 2024-02-19T22:08:42.630 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T22:08:42.630 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:08:42.631 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:42.631+0000 7fe49704e700 1 -- 172.21.15.86:0/3719781273 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fe48002ff10 msgr2=0x7fe4800323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:08:42.631 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:42.631+0000 7fe49704e700 1 --2- 172.21.15.86:0/3719781273 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fe48002ff10 0x7fe4800323d0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7fe47c0096a0 tx=0x7fe47c006b40).stop 2024-02-19T22:08:42.632 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:42.631+0000 7fe49704e700 1 -- 172.21.15.86:0/3719781273 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe4900f5410 msgr2=0x7fe4900fd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:08:42.632 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:42.631+0000 7fe49704e700 1 --2- 172.21.15.86:0/3719781273 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe4900f5410 0x7fe4900fd4e0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 rx=0x7fe484005010 tx=0x7fe48400b040).stop 2024-02-19T22:08:42.632 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:42.631+0000 7fe49704e700 1 -- 172.21.15.86:0/3719781273 shutdown_connections 2024-02-19T22:08:42.632 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:42.631+0000 7fe49704e700 1 --2- 172.21.15.86:0/3719781273 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7fe48002ff10 0x7fe4800323d0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:08:42.632 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:42.631+0000 7fe49704e700 1 --2- 172.21.15.86:0/3719781273 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe4900f5410 0x7fe4900fd4e0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:08:42.632 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:42.631+0000 7fe49704e700 1 -- 172.21.15.86:0/3719781273 >> 172.21.15.86:0/3719781273 conn(0x7fe4900f0b70 msgr2=0x7fe49005fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:08:42.632 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:42.631+0000 7fe49704e700 1 -- 172.21.15.86:0/3719781273 shutdown_connections 2024-02-19T22:08:42.632 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:42.631+0000 7fe49704e700 1 -- 172.21.15.86:0/3719781273 wait complete. 2024-02-19T22:08:42.632 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T22:08:44.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:08:43 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:08:41.980972+0000 mgr.smithi053.fyhcdi (mgr.14162) 374 : cluster [DBG] pgmap v358: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:08:44.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:08:43 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:08:42.630682+0000 mon.smithi053 (mon.0) 340 : audit [DBG] from='client.? 172.21.15.86:0/3719781273' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:08:44.394 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:08:44.395 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:08:46.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:08:45 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:08:43.981373+0000 mgr.smithi053.fyhcdi (mgr.14162) 375 : cluster [DBG] pgmap v359: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:08:46.198 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:46.182+0000 7f6d5b59e700 1 -- 172.21.15.86:0/3312915877 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6d5c06b920 msgr2=0x7f6d5c069ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:08:46.198 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:46.182+0000 7f6d5b59e700 1 --2- 172.21.15.86:0/3312915877 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6d5c06b920 0x7f6d5c069ec0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 rx=0x7f6d4c009fb0 tx=0x7f6d4c009b30).stop 2024-02-19T22:08:46.198 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:46.182+0000 7f6d5b59e700 1 -- 172.21.15.86:0/3312915877 shutdown_connections 2024-02-19T22:08:46.198 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:46.182+0000 7f6d5b59e700 1 --2- 172.21.15.86:0/3312915877 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6d5c06b920 0x7f6d5c069ec0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:08:46.198 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:46.182+0000 7f6d5b59e700 1 -- 172.21.15.86:0/3312915877 >> 172.21.15.86:0/3312915877 conn(0x7f6d5c0f1aa0 msgr2=0x7f6d5c0f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:08:46.198 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:46.182+0000 7f6d5b59e700 1 -- 172.21.15.86:0/3312915877 shutdown_connections 2024-02-19T22:08:46.198 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:46.182+0000 7f6d5b59e700 1 -- 172.21.15.86:0/3312915877 wait complete. 2024-02-19T22:08:46.198 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:46.182+0000 7f6d5b59e700 1 Processor -- start 2024-02-19T22:08:46.198 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:46.182+0000 7f6d5b59e700 1 -- start start 2024-02-19T22:08:46.198 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:46.182+0000 7f6d5b59e700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6d5c06b920 0x7f6d5c1087b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:08:46.199 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:46.182+0000 7f6d5b59e700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d5c108ca0 con 0x7f6d5c06b920 2024-02-19T22:08:46.199 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:46.183+0000 7f6d5a59c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6d5c06b920 0x7f6d5c1087b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:08:46.199 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:46.183+0000 7f6d5a59c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6d5c06b920 0x7f6d5c1087b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:36652/0 (socket says 172.21.15.86:36652) 2024-02-19T22:08:46.199 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:46.183+0000 7f6d5a59c700 1 -- 172.21.15.86:0/1238443610 learned_addr learned my addr 172.21.15.86:0/1238443610 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T22:08:46.199 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:46.183+0000 7f6d5a59c700 1 -- 172.21.15.86:0/1238443610 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d4c005040 con 0x7f6d5c06b920 2024-02-19T22:08:46.199 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:46.183+0000 7f6d5a59c700 1 --2- 172.21.15.86:0/1238443610 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6d5c06b920 0x7f6d5c1087b0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 rx=0x7f6d4c005010 tx=0x7f6d4c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:08:46.199 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:46.183+0000 7f6d4affd700 1 -- 172.21.15.86:0/1238443610 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6d4c009320 con 0x7f6d5c06b920 2024-02-19T22:08:46.199 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:46.183+0000 7f6d5b59e700 1 -- 172.21.15.86:0/1238443610 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d5c108ea0 con 0x7f6d5c06b920 2024-02-19T22:08:46.199 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:46.183+0000 7f6d4affd700 1 -- 172.21.15.86:0/1238443610 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6d4c013380 con 0x7f6d5c06b920 2024-02-19T22:08:46.199 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:46.183+0000 7f6d4affd700 1 -- 172.21.15.86:0/1238443610 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6d4c014e50 con 0x7f6d5c06b920 2024-02-19T22:08:46.199 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:46.183+0000 7f6d5b59e700 1 -- 172.21.15.86:0/1238443610 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d5c109330 con 0x7f6d5c06b920 2024-02-19T22:08:46.199 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:46.184+0000 7f6d4affd700 1 -- 172.21.15.86:0/1238443610 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6d4c0263a0 con 0x7f6d5c06b920 2024-02-19T22:08:46.199 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:46.184+0000 7f6d4affd700 1 --2- 172.21.15.86:0/1238443610 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f6d4402ff10 0x7f6d440323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:08:46.199 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:46.184+0000 7f6d5b59e700 1 -- 172.21.15.86:0/1238443610 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d5c045ad0 con 0x7f6d5c06b920 2024-02-19T22:08:46.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:46.184+0000 7f6d4affd700 1 -- 172.21.15.86:0/1238443610 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6d4c040f50 con 0x7f6d5c06b920 2024-02-19T22:08:46.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:46.185+0000 7f6d59d9b700 1 --2- 172.21.15.86:0/1238443610 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f6d4402ff10 0x7f6d440323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:08:46.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:46.185+0000 7f6d59d9b700 1 --2- 172.21.15.86:0/1238443610 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f6d4402ff10 0x7f6d440323d0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f6d500096a0 tx=0x7f6d50006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:08:46.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:46.189+0000 7f6d4affd700 1 -- 172.21.15.86:0/1238443610 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6d4c01ddc0 con 0x7f6d5c06b920 2024-02-19T22:08:46.397 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:46.396+0000 7f6d5b59e700 1 -- 172.21.15.86:0/1238443610 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6d5c10a1e0 con 0x7f6d5c06b920 2024-02-19T22:08:46.398 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:46.397+0000 7f6d4affd700 1 -- 172.21.15.86:0/1238443610 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6d4c01d9d0 con 0x7f6d5c06b920 2024-02-19T22:08:46.398 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T22:08:46.398 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:08:46.399 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:46.398+0000 7f6d5b59e700 1 -- 172.21.15.86:0/1238443610 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f6d4402ff10 msgr2=0x7f6d440323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:08:46.399 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:46.398+0000 7f6d5b59e700 1 --2- 172.21.15.86:0/1238443610 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f6d4402ff10 0x7f6d440323d0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f6d500096a0 tx=0x7f6d50006b40).stop 2024-02-19T22:08:46.399 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:46.398+0000 7f6d5b59e700 1 -- 172.21.15.86:0/1238443610 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6d5c06b920 msgr2=0x7f6d5c1087b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:08:46.399 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:46.398+0000 7f6d5b59e700 1 --2- 172.21.15.86:0/1238443610 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6d5c06b920 0x7f6d5c1087b0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 rx=0x7f6d4c005010 tx=0x7f6d4c00b040).stop 2024-02-19T22:08:46.400 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:46.398+0000 7f6d5b59e700 1 -- 172.21.15.86:0/1238443610 shutdown_connections 2024-02-19T22:08:46.400 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:46.398+0000 7f6d5b59e700 1 --2- 172.21.15.86:0/1238443610 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f6d4402ff10 0x7f6d440323d0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:08:46.400 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:46.398+0000 7f6d5b59e700 1 --2- 172.21.15.86:0/1238443610 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6d5c06b920 0x7f6d5c1087b0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:08:46.400 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:46.398+0000 7f6d5b59e700 1 -- 172.21.15.86:0/1238443610 >> 172.21.15.86:0/1238443610 conn(0x7f6d5c0f1aa0 msgr2=0x7f6d5c0f24f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:08:46.400 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:46.398+0000 7f6d5b59e700 1 -- 172.21.15.86:0/1238443610 shutdown_connections 2024-02-19T22:08:46.400 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:46.399+0000 7f6d5b59e700 1 -- 172.21.15.86:0/1238443610 wait complete. 2024-02-19T22:08:46.400 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T22:08:47.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:08:46 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:08:46.398183+0000 mon.smithi053 (mon.0) 341 : audit [DBG] from='client.? 172.21.15.86:0/1238443610' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:08:48.040 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:08:48.040 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:08:48.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:08:47 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:08:45.981875+0000 mgr.smithi053.fyhcdi (mgr.14162) 376 : cluster [DBG] pgmap v360: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:08:49.873 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:49.872+0000 7f6b410a4700 1 -- 172.21.15.86:0/2602549485 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6b3c0f5410 msgr2=0x7f6b3c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:08:49.873 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:49.872+0000 7f6b410a4700 1 --2- 172.21.15.86:0/2602549485 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6b3c0f5410 0x7f6b3c0f5830 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 rx=0x7f6b30009fb0 tx=0x7f6b30009b30).stop 2024-02-19T22:08:49.873 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:49.872+0000 7f6b410a4700 1 -- 172.21.15.86:0/2602549485 shutdown_connections 2024-02-19T22:08:49.873 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:49.872+0000 7f6b410a4700 1 --2- 172.21.15.86:0/2602549485 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6b3c0f5410 0x7f6b3c0f5830 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:08:49.873 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:49.872+0000 7f6b410a4700 1 -- 172.21.15.86:0/2602549485 >> 172.21.15.86:0/2602549485 conn(0x7f6b3c0f0b70 msgr2=0x7f6b3c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:08:49.873 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:49.872+0000 7f6b410a4700 1 -- 172.21.15.86:0/2602549485 shutdown_connections 2024-02-19T22:08:49.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:49.872+0000 7f6b410a4700 1 -- 172.21.15.86:0/2602549485 wait complete. 2024-02-19T22:08:49.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:49.873+0000 7f6b410a4700 1 Processor -- start 2024-02-19T22:08:49.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:49.873+0000 7f6b410a4700 1 -- start start 2024-02-19T22:08:49.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:49.873+0000 7f6b410a4700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6b3c0f5410 0x7f6b3c1094a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:08:49.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:49.873+0000 7f6b410a4700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b3c060280 con 0x7f6b3c0f5410 2024-02-19T22:08:49.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:49.873+0000 7f6b3ad9d700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6b3c0f5410 0x7f6b3c1094a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:08:49.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:49.873+0000 7f6b3ad9d700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6b3c0f5410 0x7f6b3c1094a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.86:32994/0 (socket says 172.21.15.86:32994) 2024-02-19T22:08:49.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:49.873+0000 7f6b3ad9d700 1 -- 172.21.15.86:0/1734872501 learned_addr learned my addr 172.21.15.86:0/1734872501 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T22:08:49.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:49.874+0000 7f6b3ad9d700 1 -- 172.21.15.86:0/1734872501 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b30005040 con 0x7f6b3c0f5410 2024-02-19T22:08:49.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:49.874+0000 7f6b3ad9d700 1 --2- 172.21.15.86:0/1734872501 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6b3c0f5410 0x7f6b3c1094a0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 rx=0x7f6b30005010 tx=0x7f6b3000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:08:49.876 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:49.874+0000 7f6b237fe700 1 -- 172.21.15.86:0/1734872501 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6b30009320 con 0x7f6b3c0f5410 2024-02-19T22:08:49.876 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:49.874+0000 7f6b410a4700 1 -- 172.21.15.86:0/1734872501 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b3c060480 con 0x7f6b3c0f5410 2024-02-19T22:08:49.876 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:49.874+0000 7f6b237fe700 1 -- 172.21.15.86:0/1734872501 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6b30013380 con 0x7f6b3c0f5410 2024-02-19T22:08:49.876 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:49.874+0000 7f6b237fe700 1 -- 172.21.15.86:0/1734872501 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6b30014e50 con 0x7f6b3c0f5410 2024-02-19T22:08:49.876 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:49.874+0000 7f6b410a4700 1 -- 172.21.15.86:0/1734872501 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b3c109bd0 con 0x7f6b3c0f5410 2024-02-19T22:08:49.877 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:49.875+0000 7f6b237fe700 1 -- 172.21.15.86:0/1734872501 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6b300095f0 con 0x7f6b3c0f5410 2024-02-19T22:08:49.877 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:49.875+0000 7f6b410a4700 1 -- 172.21.15.86:0/1734872501 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b3c045ad0 con 0x7f6b3c0f5410 2024-02-19T22:08:49.877 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:49.875+0000 7f6b237fe700 1 --2- 172.21.15.86:0/1734872501 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f6b2402ff10 0x7f6b240323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:08:49.877 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:49.875+0000 7f6b237fe700 1 -- 172.21.15.86:0/1734872501 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6b30040e90 con 0x7f6b3c0f5410 2024-02-19T22:08:49.877 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:49.875+0000 7f6b3a59c700 1 --2- 172.21.15.86:0/1734872501 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f6b2402ff10 0x7f6b240323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:08:49.877 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:49.876+0000 7f6b3a59c700 1 --2- 172.21.15.86:0/1734872501 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f6b2402ff10 0x7f6b240323d0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7f6b2c0096a0 tx=0x7f6b2c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:08:49.880 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:49.880+0000 7f6b237fe700 1 -- 172.21.15.86:0/1734872501 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6b30014420 con 0x7f6b3c0f5410 2024-02-19T22:08:50.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:08:49 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: cluster 2024-02-19T22:08:47.982316+0000 mgr.smithi053.fyhcdi (mgr.14162) 377 : cluster [DBG] pgmap v361: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:08:50.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:50.082+0000 7f6b410a4700 1 -- 172.21.15.86:0/1734872501 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6b3c0fafa0 con 0x7f6b3c0f5410 2024-02-19T22:08:50.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:50.083+0000 7f6b237fe700 1 -- 172.21.15.86:0/1734872501 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6b30010230 con 0x7f6b3c0f5410 2024-02-19T22:08:50.084 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T22:08:50.084 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"867b6b52-cf71-11ee-95bb-87774f69a715","modified":"2024-02-19T21:55:26.357744Z","created":"2024-02-19T21:55:26.357744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:08:50.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:50.084+0000 7f6b410a4700 1 -- 172.21.15.86:0/1734872501 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f6b2402ff10 msgr2=0x7f6b240323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:08:50.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:50.084+0000 7f6b410a4700 1 --2- 172.21.15.86:0/1734872501 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f6b2402ff10 0x7f6b240323d0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7f6b2c0096a0 tx=0x7f6b2c006b40).stop 2024-02-19T22:08:50.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:50.084+0000 7f6b410a4700 1 -- 172.21.15.86:0/1734872501 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6b3c0f5410 msgr2=0x7f6b3c1094a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:08:50.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:50.084+0000 7f6b410a4700 1 --2- 172.21.15.86:0/1734872501 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6b3c0f5410 0x7f6b3c1094a0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 rx=0x7f6b30005010 tx=0x7f6b3000b040).stop 2024-02-19T22:08:50.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:50.085+0000 7f6b410a4700 1 -- 172.21.15.86:0/1734872501 shutdown_connections 2024-02-19T22:08:50.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:50.085+0000 7f6b410a4700 1 --2- 172.21.15.86:0/1734872501 >> [v2:172.21.15.53:6800/1543499597,v1:172.21.15.53:6801/1543499597] conn(0x7f6b2402ff10 0x7f6b240323d0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:08:50.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:50.085+0000 7f6b410a4700 1 --2- 172.21.15.86:0/1734872501 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6b3c0f5410 0x7f6b3c1094a0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:08:50.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:50.085+0000 7f6b410a4700 1 -- 172.21.15.86:0/1734872501 >> 172.21.15.86:0/1734872501 conn(0x7f6b3c0f0b70 msgr2=0x7f6b3c0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:08:50.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:50.085+0000 7f6b410a4700 1 -- 172.21.15.86:0/1734872501 shutdown_connections 2024-02-19T22:08:50.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T22:08:50.085+0000 7f6b410a4700 1 -- 172.21.15.86:0/1734872501 wait complete. 2024-02-19T22:08:50.086 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T22:08:50.756 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_81bd20d634209c7cb82c18be12b4b5a05643ebf1/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds 2024-02-19T22:08:50.757 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-02-19T22:08:50.758 DEBUG:teuthology.orchestra.run.smithi053:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-02-19T22:08:50.784 DEBUG:teuthology.orchestra.run.smithi086:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-02-19T22:08:50.803 INFO:tasks.cephadm:Stopping all daemons... 2024-02-19T22:08:50.804 INFO:tasks.cephadm.mon.smithi053:Stopping mon.smithi053... 2024-02-19T22:08:50.804 DEBUG:teuthology.orchestra.run.smithi053:> sudo systemctl stop ceph-867b6b52-cf71-11ee-95bb-87774f69a715@mon.smithi053 2024-02-19T22:08:50.856 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:08:50 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: audit 2024-02-19T22:08:50.084287+0000 mon.smithi053 (mon.0) 342 : audit [DBG] from='client.? 172.21.15.86:0/1734872501' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:08:51.294 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:08:50 smithi053 systemd[1]: Stopping Ceph mon.smithi053 for 867b6b52-cf71-11ee-95bb-87774f69a715... 2024-02-19T22:08:51.294 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:08:50 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T22:08:50.976+0000 7efead4c2700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi053 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-02-19T22:08:51.294 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:08:50 smithi053 ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053[38150]: debug 2024-02-19T22:08:50.976+0000 7efead4c2700 -1 mon.smithi053@0(leader) e1 *** Got Signal Terminated *** 2024-02-19T22:08:51.794 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Feb 19 22:08:51 smithi053 podman[44255]: ceph-867b6b52-cf71-11ee-95bb-87774f69a715-mon.smithi053 2024-02-19T22:08:51.811 DEBUG:teuthology.orchestra.run.smithi053:> sudo pkill -f 'journalctl -f -n 0 -u ceph-867b6b52-cf71-11ee-95bb-87774f69a715@mon.smithi053.service' 2024-02-19T22:08:51.907 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-19T22:08:51.907 INFO:tasks.cephadm.mon.smithi053:Stopped mon.smithi053 2024-02-19T22:08:51.908 INFO:tasks.cephadm.mon.smithi086:Stopping mon.smithi086... 2024-02-19T22:08:51.908 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl stop ceph-867b6b52-cf71-11ee-95bb-87774f69a715@mon.smithi086 2024-02-19T22:08:51.946 INFO:teuthology.orchestra.run.smithi086.stderr:Failed to stop ceph-867b6b52-cf71-11ee-95bb-87774f69a715@mon.smithi086.service: Unit ceph-867b6b52-cf71-11ee-95bb-87774f69a715@mon.smithi086.service not loaded. 2024-02-19T22:08:51.948 DEBUG:teuthology.orchestra.run:got remote process result: 5 2024-02-19T22:08:51.949 ERROR:tasks.cephadm:Failed to stop "ceph.mon.smithi086" Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_81bd20d634209c7cb82c18be12b4b5a05643ebf1/qa/tasks/cephadm.py", line 512, in ceph_bootstrap yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_81bd20d634209c7cb82c18be12b4b5a05643ebf1/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_81bd20d634209c7cb82c18be12b4b5a05643ebf1/qa/tasks/cephadm.py", line 531, in ceph_bootstrap ctx.daemons.get_daemon(type_, id_, cluster).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/daemon/cephadmunit.py", line 154, in stop self.remote.sh(self.stop_cmd) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 97, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi086 with status 5: 'sudo systemctl stop ceph-867b6b52-cf71-11ee-95bb-87774f69a715@mon.smithi086' 2024-02-19T22:08:51.949 INFO:tasks.cephadm:Archiving crash dumps... 2024-02-19T22:08:51.951 DEBUG:teuthology.misc:Transferring archived files from smithi053:/var/lib/ceph/867b6b52-cf71-11ee-95bb-87774f69a715/crash to /home/teuthworker/archive/yuriw-2024-02-19_19:27:49-fs-pacific-release-distro-default-smithi/7566790/remote/ubuntu@smithi053.front.sepia.ceph.com/crash 2024-02-19T22:08:51.952 DEBUG:teuthology.orchestra.run.smithi053:> sudo tar c -f - -C /var/lib/ceph/867b6b52-cf71-11ee-95bb-87774f69a715/crash -- . 2024-02-19T22:08:51.991 DEBUG:teuthology.misc:Transferring archived files from smithi086:/var/lib/ceph/867b6b52-cf71-11ee-95bb-87774f69a715/crash to /home/teuthworker/archive/yuriw-2024-02-19_19:27:49-fs-pacific-release-distro-default-smithi/7566790/remote/ubuntu@smithi086.front.sepia.ceph.com/crash 2024-02-19T22:08:51.992 DEBUG:teuthology.orchestra.run.smithi086:> sudo tar c -f - -C /var/lib/ceph/867b6b52-cf71-11ee-95bb-87774f69a715/crash -- . 2024-02-19T22:08:52.028 INFO:teuthology.orchestra.run.smithi086.stderr:tar: /var/lib/ceph/867b6b52-cf71-11ee-95bb-87774f69a715/crash: Cannot open: No such file or directory 2024-02-19T22:08:52.029 INFO:teuthology.orchestra.run.smithi086.stderr:tar: Error is not recoverable: exiting now 2024-02-19T22:08:52.029 INFO:tasks.cephadm:Checking cluster log for badness... 2024-02-19T22:08:52.030 DEBUG:teuthology.orchestra.run.smithi053:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/867b6b52-cf71-11ee-95bb-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v '\(PG_DEGRADED\)' | egrep -v 'Degraded data redundancy' | egrep -v '\(PG_' | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v 'deprecated feature inline_data' | egrep -v 'compat changed unexpectedly' | egrep -v 'overall HEALTH_' | egrep -v OSD_DOWN | egrep -v OSD_ | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2024-02-19T22:08:52.070 INFO:tasks.cephadm:Compressing logs... 2024-02-19T22:08:52.071 DEBUG:teuthology.orchestra.run.smithi053:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-02-19T22:08:52.113 DEBUG:teuthology.orchestra.run.smithi086:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-02-19T22:08:52.140 INFO:teuthology.orchestra.run.smithi053.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-02-19T22:08:52.141 INFO:teuthology.orchestra.run.smithi053.stderr:: No such file or directory 2024-02-19T22:08:52.141 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/867b6b52-cf71-11ee-95bb-87774f69a715/ceph-mon.smithi053.log 2024-02-19T22:08:52.141 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/867b6b52-cf71-11ee-95bb-87774f69a715/ceph-volume.log 2024-02-19T22:08:52.141 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/867b6b52-cf71-11ee-95bb-87774f69a715/ceph.audit.log 2024-02-19T22:08:52.142 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/867b6b52-cf71-11ee-95bb-87774f69a715/ceph-mon.smithi053.log: /var/log/ceph/867b6b52-cf71-11ee-95bb-87774f69a715/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/867b6b52-cf71-11ee-95bb-87774f69a715/ceph.cephadm.log 2024-02-19T22:08:52.142 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/867b6b52-cf71-11ee-95bb-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/867b6b52-cf71-11ee-95bb-87774f69a715/ceph-mgr.smithi053.fyhcdi.log 2024-02-19T22:08:52.142 INFO:teuthology.orchestra.run.smithi086.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-02-19T22:08:52.142 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-02-19T22:08:52.143 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/867b6b52-cf71-11ee-95bb-87774f69a715/ceph-volume.log 2024-02-19T22:08:52.143 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/cephadm.log: /var/log/ceph/867b6b52-cf71-11ee-95bb-87774f69a715/ceph-volume.log: 88.4% -- replaced with /var/log/ceph/867b6b52-cf71-11ee-95bb-87774f69a715/ceph-volume.log.gz 2024-02-19T22:08:52.143 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/867b6b52-cf71-11ee-95bb-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/867b6b52-cf71-11ee-95bb-87774f69a715/ceph.log 2024-02-19T22:08:52.144 INFO:teuthology.orchestra.run.smithi053.stderr: 90.2% -- replaced with /var/log/ceph/867b6b52-cf71-11ee-95bb-87774f69a715/ceph-volume.log.gz 2024-02-19T22:08:52.144 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/867b6b52-cf71-11ee-95bb-87774f69a715/ceph-mgr.smithi053.fyhcdi.log: 90.3% -- replaced with /var/log/ceph/867b6b52-cf71-11ee-95bb-87774f69a715/ceph.cephadm.log.gz 2024-02-19T22:08:52.144 INFO:teuthology.orchestra.run.smithi053.stderr: 88.9% -- replaced with /var/log/ceph/867b6b52-cf71-11ee-95bb-87774f69a715/ceph.audit.log.gz 2024-02-19T22:08:52.144 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/867b6b52-cf71-11ee-95bb-87774f69a715/ceph.log: 88.7% -- replaced with /var/log/ceph/867b6b52-cf71-11ee-95bb-87774f69a715/ceph.log.gz 2024-02-19T22:08:52.144 INFO:teuthology.orchestra.run.smithi086.stderr: 95.4% -- replaced with /var/log/ceph/cephadm.log.gz 2024-02-19T22:08:52.146 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-02-19T22:08:52.146 INFO:teuthology.orchestra.run.smithi086.stderr:real 0m0.014s 2024-02-19T22:08:52.146 INFO:teuthology.orchestra.run.smithi086.stderr:user 0m0.015s 2024-02-19T22:08:52.146 INFO:teuthology.orchestra.run.smithi086.stderr:sys 0m0.009s 2024-02-19T22:08:52.147 INFO:teuthology.orchestra.run.smithi053.stderr: 92.2% -- replaced with /var/log/ceph/cephadm.log.gz 2024-02-19T22:08:52.243 INFO:teuthology.orchestra.run.smithi053.stderr: 92.7% -- replaced with /var/log/ceph/867b6b52-cf71-11ee-95bb-87774f69a715/ceph-mgr.smithi053.fyhcdi.log.gz 2024-02-19T22:08:52.321 INFO:teuthology.orchestra.run.smithi053.stderr: 90.6% -- replaced with /var/log/ceph/867b6b52-cf71-11ee-95bb-87774f69a715/ceph-mon.smithi053.log.gz 2024-02-19T22:08:52.323 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-02-19T22:08:52.323 INFO:teuthology.orchestra.run.smithi053.stderr:real 0m0.194s 2024-02-19T22:08:52.323 INFO:teuthology.orchestra.run.smithi053.stderr:user 0m0.286s 2024-02-19T22:08:52.323 INFO:teuthology.orchestra.run.smithi053.stderr:sys 0m0.030s 2024-02-19T22:08:52.323 INFO:tasks.cephadm:Archiving logs... 2024-02-19T22:08:52.323 DEBUG:teuthology.misc:Transferring archived files from smithi053:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-02-19_19:27:49-fs-pacific-release-distro-default-smithi/7566790/remote/ubuntu@smithi053.front.sepia.ceph.com/log 2024-02-19T22:08:52.324 DEBUG:teuthology.orchestra.run.smithi053:> sudo tar c -f - -C /var/log/ceph -- . 2024-02-19T22:08:52.442 DEBUG:teuthology.misc:Transferring archived files from smithi086:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-02-19_19:27:49-fs-pacific-release-distro-default-smithi/7566790/remote/ubuntu@smithi086.front.sepia.ceph.com/log 2024-02-19T22:08:52.443 DEBUG:teuthology.orchestra.run.smithi086:> sudo tar c -f - -C /var/log/ceph -- . 2024-02-19T22:08:52.475 INFO:tasks.cephadm:Removing cluster... 2024-02-19T22:08:52.475 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 --force 2024-02-19T22:08:55.698 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 867b6b52-cf71-11ee-95bb-87774f69a715 --force 2024-02-19T22:11:40.706 INFO:tasks.cephadm:Removing cephadm ... 2024-02-19T22:11:40.707 DEBUG:teuthology.orchestra.run.smithi053:> rm -rf /home/ubuntu/cephtest/cephadm 2024-02-19T22:11:40.732 DEBUG:teuthology.orchestra.run.smithi086:> rm -rf /home/ubuntu/cephtest/cephadm 2024-02-19T22:11:40.750 INFO:tasks.cephadm:Teardown complete 2024-02-19T22:11:40.750 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_81bd20d634209c7cb82c18be12b4b5a05643ebf1/qa/tasks/cephadm.py", line 512, in ceph_bootstrap yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_81bd20d634209c7cb82c18be12b4b5a05643ebf1/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_81bd20d634209c7cb82c18be12b4b5a05643ebf1/qa/tasks/cephadm.py", line 1463, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_81bd20d634209c7cb82c18be12b4b5a05643ebf1/qa/tasks/cephadm.py", line 1376, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_81bd20d634209c7cb82c18be12b4b5a05643ebf1/qa/tasks/cephadm.py", line 531, in ceph_bootstrap ctx.daemons.get_daemon(type_, id_, cluster).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/daemon/cephadmunit.py", line 154, in stop self.remote.sh(self.stop_cmd) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 97, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi086 with status 5: 'sudo systemctl stop ceph-867b6b52-cf71-11ee-95bb-87774f69a715@mon.smithi086' 2024-02-19T22:11:40.962 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=6b0c71a257b143a28242a1d03fd9d733 Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_81bd20d634209c7cb82c18be12b4b5a05643ebf1/qa/tasks/cephadm.py", line 512, in ceph_bootstrap yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_81bd20d634209c7cb82c18be12b4b5a05643ebf1/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_81bd20d634209c7cb82c18be12b4b5a05643ebf1/qa/tasks/cephadm.py", line 1463, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_81bd20d634209c7cb82c18be12b4b5a05643ebf1/qa/tasks/cephadm.py", line 1376, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_81bd20d634209c7cb82c18be12b4b5a05643ebf1/qa/tasks/cephadm.py", line 531, in ceph_bootstrap ctx.daemons.get_daemon(type_, id_, cluster).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/daemon/cephadmunit.py", line 154, in stop self.remote.sh(self.stop_cmd) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 97, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi086 with status 5: 'sudo systemctl stop ceph-867b6b52-cf71-11ee-95bb-87774f69a715@mon.smithi086' 2024-02-19T22:11:40.966 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-02-19T22:11:40.975 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-02-19T22:11:40.983 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-02-19T22:11:40.983 DEBUG:teuthology.orchestra.run.smithi053:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-02-19T22:11:41.013 DEBUG:teuthology.orchestra.run.smithi086:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-02-19T22:11:41.115 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-02-19T22:11:41.123 INFO:teuthology.task.clock:Checking final clock skew... 2024-02-19T22:11:41.123 DEBUG:teuthology.orchestra.run.smithi053:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-19T22:11:41.126 DEBUG:teuthology.orchestra.run.smithi086:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-19T22:11:41.144 INFO:teuthology.orchestra.run.smithi086.stderr:bash: ntpq: command not found 2024-02-19T22:11:41.144 INFO:teuthology.orchestra.run.smithi053.stderr:bash: ntpq: command not found 2024-02-19T22:11:41.148 INFO:teuthology.orchestra.run.smithi086.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-19T22:11:41.148 INFO:teuthology.orchestra.run.smithi086.stdout:=============================================================================== 2024-02-19T22:11:41.149 INFO:teuthology.orchestra.run.smithi086.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 58 +1447us[+1414us] +/- 91ms 2024-02-19T22:11:41.149 INFO:teuthology.orchestra.run.smithi086.stdout:^* hv02.front.sepia.ceph.com 3 6 377 58 -1426us[-1459us] +/- 59ms 2024-02-19T22:11:41.149 INFO:teuthology.orchestra.run.smithi086.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 57 +736us[ +736us] +/- 77ms 2024-02-19T22:11:41.149 INFO:teuthology.orchestra.run.smithi086.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-02-19T22:11:41.150 INFO:teuthology.orchestra.run.smithi053.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-19T22:11:41.150 INFO:teuthology.orchestra.run.smithi053.stdout:=============================================================================== 2024-02-19T22:11:41.150 INFO:teuthology.orchestra.run.smithi053.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 57 +1424us[+1424us] +/- 91ms 2024-02-19T22:11:41.150 INFO:teuthology.orchestra.run.smithi053.stdout:^* hv02.front.sepia.ceph.com 3 6 377 59 -1458us[-1484us] +/- 59ms 2024-02-19T22:11:41.150 INFO:teuthology.orchestra.run.smithi053.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 57 +717us[ +717us] +/- 77ms 2024-02-19T22:11:41.151 INFO:teuthology.orchestra.run.smithi053.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-02-19T22:11:41.151 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-02-19T22:11:41.159 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-02-19T22:11:41.160 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-02-19T22:11:41.169 DEBUG:teuthology.orchestra.run.smithi053:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-02-19T22:11:41.259 DEBUG:teuthology.orchestra.run.smithi086:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-02-19T22:11:41.325 DEBUG:teuthology.orchestra.run.smithi053:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-19T22:11:41.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-19T22:11:41.358 DEBUG:teuthology.orchestra.run.smithi086:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-19T22:11:41.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-19T22:11:41.387 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-02-19T22:11:41.397 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-02-19T22:11:41.405 INFO:teuthology.task.internal:Duration was 1586.522227 seconds 2024-02-19T22:11:41.405 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-02-19T22:11:41.414 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-02-19T22:11:41.414 DEBUG:teuthology.orchestra.run.smithi053:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-02-19T22:11:41.417 DEBUG:teuthology.orchestra.run.smithi086:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-02-19T22:11:41.465 INFO:teuthology.orchestra.run.smithi053.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-19T22:11:41.472 INFO:teuthology.orchestra.run.smithi086.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-19T22:11:41.808 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-02-19T22:11:41.809 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi053.front.sepia.ceph.com 2024-02-19T22:11:41.809 DEBUG:teuthology.orchestra.run.smithi053:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-02-19T22:11:41.833 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi086.front.sepia.ceph.com 2024-02-19T22:11:41.834 DEBUG:teuthology.orchestra.run.smithi086:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-02-19T22:11:41.856 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-02-19T22:11:41.857 DEBUG:teuthology.orchestra.run.smithi053:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-02-19T22:11:41.877 DEBUG:teuthology.orchestra.run.smithi086:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-02-19T22:11:41.947 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-02-19T22:11:41.947 DEBUG:teuthology.orchestra.run.smithi053:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-02-19T22:11:41.950 DEBUG:teuthology.orchestra.run.smithi086:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-02-19T22:11:42.061 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-02-19T22:11:42.071 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-02-19T22:11:42.071 DEBUG:teuthology.orchestra.run.smithi053:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-02-19T22:11:42.099 DEBUG:teuthology.orchestra.run.smithi086:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-02-19T22:11:42.126 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-02-19T22:11:42.136 DEBUG:teuthology.orchestra.run.smithi053:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-02-19T22:11:42.142 DEBUG:teuthology.orchestra.run.smithi086:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-02-19T22:11:42.166 INFO:teuthology.orchestra.run.smithi053.stdout:kernel.core_pattern = core 2024-02-19T22:11:42.192 INFO:teuthology.orchestra.run.smithi086.stdout:kernel.core_pattern = core 2024-02-19T22:11:42.238 DEBUG:teuthology.orchestra.run.smithi053:> test -e /home/ubuntu/cephtest/archive/coredump 2024-02-19T22:11:42.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-19T22:11:42.255 DEBUG:teuthology.orchestra.run.smithi086:> test -e /home/ubuntu/cephtest/archive/coredump 2024-02-19T22:11:42.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-19T22:11:42.295 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-02-19T22:11:42.305 INFO:teuthology.task.internal:Transferring archived files... 2024-02-19T22:11:42.306 DEBUG:teuthology.misc:Transferring archived files from smithi053:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-02-19_19:27:49-fs-pacific-release-distro-default-smithi/7566790/remote/smithi053 2024-02-19T22:11:42.307 DEBUG:teuthology.orchestra.run.smithi053:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-02-19T22:11:42.348 DEBUG:teuthology.misc:Transferring archived files from smithi086:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-02-19_19:27:49-fs-pacific-release-distro-default-smithi/7566790/remote/smithi086 2024-02-19T22:11:42.348 DEBUG:teuthology.orchestra.run.smithi086:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-02-19T22:11:42.388 INFO:teuthology.task.internal:Removing archive directory... 2024-02-19T22:11:42.389 DEBUG:teuthology.orchestra.run.smithi053:> rm -rf -- /home/ubuntu/cephtest/archive 2024-02-19T22:11:42.391 DEBUG:teuthology.orchestra.run.smithi086:> rm -rf -- /home/ubuntu/cephtest/archive 2024-02-19T22:11:42.439 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-02-19T22:11:42.451 INFO:teuthology.task.internal:Not uploading archives. 2024-02-19T22:11:42.451 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-02-19T22:11:42.461 INFO:teuthology.task.internal:Tidying up after the test... 2024-02-19T22:11:42.461 DEBUG:teuthology.orchestra.run.smithi053:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-02-19T22:11:42.463 DEBUG:teuthology.orchestra.run.smithi086:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-02-19T22:11:42.478 INFO:teuthology.orchestra.run.smithi053.stdout: 525357 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 19 22:11 /home/ubuntu/cephtest 2024-02-19T22:11:42.497 INFO:teuthology.orchestra.run.smithi086.stdout: 526076 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 19 22:11 /home/ubuntu/cephtest 2024-02-19T22:11:42.498 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-02-19T22:11:42.508 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-02-19T22:11:42.527 INFO:teuthology.nuke:Checking targets against current locks 2024-02-19T22:11:42.561 DEBUG:teuthology.nuke:shortname: smithi053 2024-02-19T22:11:42.562 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-19T22:11:42.608 DEBUG:teuthology.nuke:shortname: smithi086 2024-02-19T22:11:42.609 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-19T22:11:42.620 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi053.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-19_19:27:49-fs-pacific-release-distro-default-smithi/7566790', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-19 21:36:58.616045', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOZmqTcx8AszQ0DRj1FdK+b1wOzQvjOnoRF1Cyd0hy4AkDcvdcGOVx59SKOTgOfBKymcBUFepHbZAEQBEnwCto8='} 2024-02-19T22:11:42.664 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi086.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-19_19:27:49-fs-pacific-release-distro-default-smithi/7566790', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-19 21:36:58.617595', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLZ0e+5X3hmPqCyniFmfDHi38amlVmWCZs9h7B6MrbloeJ3ONBlr9vwa9fKYCNgwke229OMcwn/jF/1/u5+91Bc='} 2024-02-19T22:11:42.715 INFO:teuthology.orchestra.console.smithi053:Power off 2024-02-19T22:11:42.715 DEBUG:teuthology.orchestra.console.smithi053:pexpect command: ipmitool -H smithi053.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-19T22:11:42.750 INFO:teuthology.orchestra.console.smithi086:Power off 2024-02-19T22:11:42.750 DEBUG:teuthology.orchestra.console.smithi086:pexpect command: ipmitool -H smithi086.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-19T22:11:42.764 DEBUG:teuthology.orchestra.console.smithi053:power off output: Chassis Power Control: Down/Off 2024-02-19T22:11:42.764 DEBUG:teuthology.orchestra.console.smithi053:pexpect command: ipmitool -H smithi053.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-19T22:11:42.796 DEBUG:teuthology.orchestra.console.smithi053:check power output: Chassis Power is on 2024-02-19T22:11:42.799 DEBUG:teuthology.orchestra.console.smithi086:power off output: Chassis Power Control: Down/Off 2024-02-19T22:11:42.800 DEBUG:teuthology.orchestra.console.smithi086:pexpect command: ipmitool -H smithi086.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-19T22:11:42.847 DEBUG:teuthology.orchestra.console.smithi086:check power output: Chassis Power is on 2024-02-19T22:11:46.797 DEBUG:teuthology.orchestra.console.smithi053:pexpect command: ipmitool -H smithi053.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-19T22:11:46.848 DEBUG:teuthology.orchestra.console.smithi086:pexpect command: ipmitool -H smithi086.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-19T22:11:46.907 DEBUG:teuthology.orchestra.console.smithi053:check power output: Chassis Power is on 2024-02-19T22:11:46.958 DEBUG:teuthology.orchestra.console.smithi086:check power output: Chassis Power is on 2024-02-19T22:11:50.912 DEBUG:teuthology.orchestra.console.smithi053:pexpect command: ipmitool -H smithi053.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-19T22:11:50.958 DEBUG:teuthology.orchestra.console.smithi086:pexpect command: ipmitool -H smithi086.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-19T22:11:51.022 DEBUG:teuthology.orchestra.console.smithi053:check power output: Chassis Power is off 2024-02-19T22:11:51.068 DEBUG:teuthology.orchestra.console.smithi086:check power output: Chassis Power is on 2024-02-19T22:11:51.123 INFO:teuthology.orchestra.console.smithi053:Power off completed 2024-02-19T22:11:55.072 DEBUG:teuthology.orchestra.console.smithi086:pexpect command: ipmitool -H smithi086.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-19T22:11:55.182 DEBUG:teuthology.orchestra.console.smithi086:check power output: Chassis Power is off 2024-02-19T22:11:55.283 INFO:teuthology.orchestra.console.smithi086:Power off completed 2024-02-19T22:11:55.388 INFO:teuthology.run:Summary data: description: fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} duration: 1586.522227048874 failure_reason: 'Command failed on smithi086 with status 5: ''sudo systemctl stop ceph-867b6b52-cf71-11ee-95bb-87774f69a715@mon.smithi086''' flavor: default owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=6b0c71a257b143a28242a1d03fd9d733 status: fail success: false 2024-02-19T22:11:55.388 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-02-19T22:11:55.474 INFO:teuthology.run:FAIL